Response to Glanville et al.: how to identify randomized controlled trials in MEDLINE: ten years on

A recent study by Glanville et al. [1] assessed whether the 1994 Cochrane Highly Sensitive Search Strategy (HSSS) for detecting randomized controlled trials (RCTs) in MEDLINE could be improved. In this paper, they also compared the sensitivity and precision of the six strategies they developed with seven published strategies (twelve total sets of strategies because one strategy was the same in both groups). One of the strategies compared in their evaluation was the sensitive therapy filter available on the Clinical Queries screen in PubMed <http://www.ncbi.nlm.nih.gov/entrez/query/static/clinical.shtml>, which was developed by our research group at McMaster University [2]. We welcome such an analysis of our work and encourage researchers to continue to investigate improved ways of searching for desired article types in MEDLINE.

Glanville et al. indicated that one of the advantages of the method they used to develop the search strategies reported in the paper was the objectivity in deriving the most efficient search terms to find desired types of records. Their derivation used the techniques of word frequency analysis and discriminate analysis. This method was contrasted to the subjectivity used to derive the list of search terms when the 1994 Cochrane HSSS was developed and when we derived and tested our therapy filter in 2002. Comparing the performance of our most sensitive therapy filter (the McMaster strategy or filter) with the performance of the search strategies reported in the paper by Glanville et al. (Glanville strategies or filters) is important. This letter reports the testing of the Glanville filters in the McMaster Clinical Hedges Database and the testing of the McMaster filters in the Glanville database reported in their paper. This two-way, two-database method provides the most useful comparison of the strategies to determine their relative merits as highly sensitive search strategies for detecting RCTs in MEDLINE.

The most sensitive McMaster search strategy for detecting therapy articles, available through the Clinical Queries screen of PubMed, was tested by Glanville et al., and the data are provided in Table 1. Their testing set of documents (gold standard) included 425 records, of which 392 were indexed trials and 32 were unindexed trials. These records came from hand-searching 4,681 records published in 2003 that were retrieved by searching MEDLINE using the exploded Medical Subject Headings terms, “Otitis Media,” “Migraine,” “Cataract Extraction,” and “Asthma.” Testing the McMaster search strategy in the Glanville database produced a sensitivity of 96.46% and a precision of 25.89%. In the McMaster test database, our performance results with the same strategy were 99.3% and 9.9%, respectively.

Table 1 McMaster strategy in McMaster database: most sensitive search strategy for detecting therapy articles in MEDLINE, shown in Ovid syntax but also available through the Clinical Queries Screen of PubMed

To complete the cycle of comparing the McMaster and Glanville strategies, the Glanville strategies should be run in the McMaster test database. The McMaster Clinical Hedges Database has 49,028 articles, of which 6,568 articles were classified in a hand-search as original studies evaluating a treatment. Of the latter, 1,507 (3.2%) met our methodological criteria for being a high-quality RCT (i.e., RCT with at least 80% follow-up of those randomized and analysis consistent with the study design). The Glanville test database was smaller (4,681 articles), was proportionally richer in RCTs (425 RCTs or 9.1%), and included RCTs without any methodological criteria requirement.

The results of testing the Glanville search strategies in the McMaster Clinical Hedges Database are shown in Table 2. We find that the sensitivity and precision of these strategies are somewhat different when tested in the McMaster database than their reported characteristics when run in the Glanville database. In some cases, the sensitivity has increased, and, in some cases, it has decreased. As described below, because the proportion of RCTs is lower in the McMaster database, the precision has decreased in all cases.

Table 2 Glanville strategies in McMaster database: search strategies reported in Glanville et al.

paper to identify randomized controlled trials in MEDLINE, performance in the Clinical Hedges database

The data presented in Tables 1 and 2 show that the McMaster sensitive therapy search strategy available through PubMed is in fact highly sensitive, when compared with the Glanville strategies, and has fewer terms. Thus, it appears that the subjective method of deriving the list of search terms that we used to develop the sensitive therapy filter available through PubMed has favorable performance and likely does not introduce bias.

An additional aspect of the comparison between the McMaster search filter and the Glanville strategies requires further comment. Precision depends on the concentration of target articles (in this case, therapy articles) in the entire database. Both research groups tested the search strategies in a subset of MEDLINE records. The precision figures reported by the Glanville and McMaster groups for search strategies in both databases are, therefore, only an illustration of search strategy performance. When searching in the entire MEDLINE database, precision will likely be lower.

Much important work has been done by many groups in developing search strategies for quickly and efficiently selecting articles based on research methodologies. Studies of comparisons are important for all who are interested in the area of information retrieval and knowledge synthesis in health care. Clarifying differences in methods between such comparisons is important for understanding the implications of the searching techniques. To accommodate further comparisons, we have the capability of testing strategies in the McMaster database and are willing to do so. Please contact us for further information.