Measuring the performance of The Cochrane Library

Gustoimages/Science Photo Library

How well does The Cochrane Library achieve its objective—to provide accessible and credible evidence to guide decision making in medicine and public health? And how should we measure success or failure? Regular users of The Cochrane Library will have a view of its quality, and we hope that you think that it's good and improving. One of our tasks as the Cochrane Library Oversight Committee (CLOC) is to report to the Steering Group of The Cochrane Collaboration on the performance of the Library and its Editor in Chief. In general we are satisfied with both. But we thought it important to try to introduce some objectivity into the evaluation of The Cochrane Library by devising a set of metrics. Together with the Cochrane Editorial Unit we have done so, and they are shown in Table 1 along with their values for the past three years. We welcome your feedback on the metrics we have devised. To be useful, metrics must provide information on progress in relation to the aims of the Library. In addition, it must be possible to measure them precisely and relatively easily. They must also change over time at a speed that is useful—not as fast as hourly but not as slowly as five yearly.

1. Metrics for assessing the performance of the Cochrane Database of Systematic Reviews and The Cochrane Library.

Metric	2012 (to June)	2012 (extrapolated)	Increase 2012–2011	2011	Increase 2011–2010	2010	Increase 2010–2009	2009
Number of new reviews published	210	420	1.0%	416	−7.3%	449	11.7%	402
Number of updated reviews published	241	482	3.0%	468	−10.7%	524	9.4%	479
Total number of active reviews	4,905	5097	8.1%	4713	8.9%	4329	9.4%	3958
Percentage of active reviews that are up to date (within two years)	35.8			36.5	0.6%	36.3	−3.5%	39.8
Number of registered authors	22,557	22,557	11.4%	20,246	−4.2%	21,123	17.0%	18,057
Number of registered authors from LMICs	4943	4943	13.7%	4346	11.8%	3886	1.3%	3838
Percentage of authors from LMICs	21.9%	21.9%	0.4%	21.5%	3.1%	18.4%	−2.9%	21.3%
Mean time (months) from protocol registration to publication						30.1	3.4%	29.1
Median time (months) from protocol registration to publication						23	0.0%	23
Full text access	1,980,715	5,942,145	37.0%	4,337,045	9.6%	3,957,567	13.9%	3,473,141
Amount of substantive feedback	34	68	−11.7%	77	−8.3%	84	−22.2%	108
Impact factor				5.912	−4.4%	6.186	9.4%	5.653
Number of citations						27,366	18.5%	23,102
Media mentions	1715	3430	−35.0%	5277	33.3%	3958	15.3%	3434

The number of new reviews is clearly an important measure and has ranged between 402 and 449 per year for the previous four years with no clear time trend. The extent to which the Collaboration achieves its aim to update all reviews regularly also seems an important measure as reviews must be updated to be useful. The number of reviews updated has been fairly constant, ranging between 479 and 524 per year for the previous four years with no clear time trend. As the Library is cumulative the total number of reviews is increasing, but the proportion that are up to date (updated within the past two years) is steadily declining—from 39.8% in 2009 to 35.8% in June 2012. It's worrying both that the figure is low and declining, given that Cochrane Reviews are living documents, readers expect Cochrane Reviews to be up to date, and review authors commit to maintaining their reviews every two years in order that the findings remain accurate. As we can see, it does not seem possible to achieve the original aspiration of the Cochrane pioneers. In April 2012 the Editor in Chief proposed that there should no longer be an expectation that every review would be updated every two years. Some reviews cover questions that are of historic interest only—with interventions that are no longer used, or covering comparisons that are considered to have been adequately evaluated. This implies that all reviews will need to be labelled appropriately, but they may not require updating. This will leave a smaller but better prioritised pool of reviews that need regular maintenance, perhaps more frequently than two years in some cases. We will modify the metric for next year to be the proportion of reviews that need to be updated that have been updated.

One of the most important measures of Cochrane content is its methodological quality and adherence to best current standards as described in the Cochrane Handbook for Systematic Reviews of Interventions. At the moment we do not have a measure for this, but the Collaboration has recently developed minimum standards for the conduct and reporting of reviews, and these will be incorporated into the metrics in the next year.

Producing systematic reviews is demanding work, and the more authors that there are to contribute to the Cochrane Database of Systematic Reviews the better. So our metrics include the total number of registered authors, and because the Collaboration wants to increase the geographical and linguistic diversity of its contributors we have included the proportion of authors from low‐ and middle‐income countries (LMICs). The latter is an example of a metric that has to be calculated manually, in this case by Claire Allen and colleagues in the Cochrane Operations Unit. This metric is a case where we might set a target. Should we perhaps aim to increase the proportion of authors from LMICs from 22% to 30% by 2015? We would like your views.

It seemed to be important to include a measure of timeliness and efficiency of review production in the metrics, and we selected the time from publication of a protocol to publication of the review. The average is a slow 30 months, but this figure includes a number of older reviews with very delayed publication, so the median is a better measure—and stands at 23 months. Is two years a reasonable time to take to publish a review? This metric is, of course, a measure of both the time taken by the review authors to undertake the research and also the work done by editors and peer reviewers. We should perhaps include a metric of the time from submission to publication of the completed review, a stage that is more under the control of Cochrane Review Group editorial teams.

The main point of The Cochrane Library is that its reviews should be accessed and used to improve the quality of health care and policy. Usage of the full text of Cochrane Reviews is thus a very important metric, and it is pleasing that it is high and increasing rapidly. We opted for usage of full text rather than hits on the website because it better reflects serious usage. The amount of feedback received might have been included under the heading of content, but we chose to include it here because people must access and use the website in order to send feedback. As feedback represents a form of post‐publication review and as the Library is interested to increase debate, the amount of feedback is disappointingly low, and we are implementing ways to try and increase it, partly by simplifying an over‐elaborate and slow process. In future we should also measure responsiveness of review authors to published feedback.

The Collaboration wants the Cochrane Library to have as much impact as possible, and so we have included measures of impact—with the impact factor in pride of place. Most readers are probably familiar with the impact factor, which has become very important to academics in many countries in the past two decades, but for those who do not know it is the number of citations of articles in a publication divided by the number of citable articles. To be more precise and to use the example of the impact factor for 2011 it is the number of citations in 2011 of articles published in 2009 and 2010 divided by the number of citable articles published in 2009 and 2010. Despite its importance to some academics, the impact factor has many problems and increasingly is gamed by editors. The impact factor matters to a publication as one measure of its use but also because it influences whether academics will submit work. The Cochrane Database of Systematic Reviews' impact factor of about 6 puts it among the highest ranked publications and is adequate for most academics to be happy to publish there. Because it is possible to have a high impact factor and publish very little (indeed, reducing the amount they publish is one of the strategies used by editors to increase their journal's impact factor) we thought it important as well to include a metric of the total number of citations. This is impressively high. As another measure of impact the Collaboration is seeking ways to measure the use of Cochrane Reviews in clinical practice guidelines.

Our final metric is the number of times that Cochrane Reviews are mentioned in the media (measured using a consistent methodology over the previous five years, and including Google News). The Collaboration is interested in influencing the broader world, including patients and politicians, so this is a useful measure although it has fluctuated considerably over recent years and can be highly sensitive to small numbers of extreme outliers. This was exemplified by the 2011 review of zinc and its role in preventing and treating the common cold, which attracted a level of media interest that was unprecedented and out of proportion to its clinical importance.

We hope that you will agree that the metrics we have selected are meaningful and useful, but we will be very grateful for feedback on these metrics and suggestions for others that we might consider and the reasons why.

Feedback on this editorial and proposals for future editorials are welcome.

End Notes

Declarations of interest

none declared