Table 8.
Characteristics of stage II evaluation of methods studies
| First Author Year Title |
Primary objective | Existence of a protocol | Study design | Health field the filter designed for | # of filters evaluated (# filters developed by the author) | Database (interfaces) | Technique to identify and/or create a gold standard | Sample size of the gold standard set or validation set (n) | Validation on a data set distinct from the derivation data | Performance measures used | Search dates for the gold standard or validation set | Name of filters evaluated (number of filters) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Boluyt 2008 [13] Usefulness of systematic review search strategies in finding child health systematic reviews in MEDLINE. |
Assess search filters for child health SRs in PubMed | NR | Comparative search filter evaluation | Child health | 9 | PubMed | Handsearching, Developed based on database searches | 387 | Yes | Sensitivity/recall, precision | Handsearch 1994, 1997, 2000, 2002, and 2004; DARE up to 2004, and year 2006 | PubMed filter 2006 Shojania 2001 Boynton 1998 White 2001 (two) Montori 2005 (four) |
|
Boynton 1998 [15] Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. |
Evaluate propose a range of search strategies to identify SRs in MEDLINE | NR | Search filter evaluation, Comparative search filter evaluation | Medicine (general and internal) | 15 (11) | MEDLINE (Ovid) | Handsearching | 288 | No | Sensitivity/recall, precision | 1992 and 1995 | Boynton 1998 (eleven) Hunt 1997 (two) CRD - Oxman 1994 (two) |
|
Eady 2008 [19] PsycINFO search strategies identified methodologically sound therapy studies and review articles for use by clinicians and researchers |
Evaluate search strategies for finding SRs in PsycINFO | NR | Search filter evaluation | Psych. | N/A | PsycINFO | Handsearching | 58 | No | Sensitivity/recall, precision, specificity, accuracy | 2000 | Eady 2008 |
|
Golder 2006 [20] Identifying systematic reviews of the adverse effects of health care interventions. |
Identify SRs of adverse effects in two major databases | NR | Search filter evaluation | Adverse effects | N/A | DARE (CDSR and CRD) | Developed based on database searches | 270 | No | Sensitivity/recall, precision | 1994 to 2005 | Golder 2006 |
|
Lee 2012 [16] An optimal search filter for retrieving systematic reviews and meta-analyses. |
Develop and validate the health-evidence.ca SR filter and compare its performance to other filters | NR | Search filter evaluation, Comparative search filter evaluation | Public health | 31 (3) | MEDLINE, EMBASE, and CINAHL | Handsearching, Developed based on database searches | 219 | Yes | Sensitivity/recall, precision, specificity NNR | 2004/2005 | health-evidence.ca SR filter - Lee 2012 (three) Montori [2005- four] Hunt 1997 (two) Shojania 2001 Boynton 1998 (two) BMJ Clin Evidence n.d. CRD Ciliska 2007 (four) SIGN n.d. Wilczynski 2007 (four) McKibbon 1998 |
|
Montori 2005 [17] Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. |
Develop optimal search strategies in Medline for retrieving SRs | NR | Search filter evaluation, Comparative search filter evaluation | Medicinefamily practice, nursing, mental health |
10 (4) | MEDLINE | Handsearching | 753 | Yes | Sensitivity/recall, specificity, precision | 2000 | Montori 2005 (four) White 2001 (three) Hunt 1997 (two) Shojania 2001 |
|
Rathbone 2016 [12] A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. |
Evaluate seven databases to determine their coverage of SRs of hypertension | NR | Comparative database evaluation | Hypertension | N/A | Cochrane, DARE, EMBASE, Epistemonikos, MEDLINE, PubMed, and TRIP | Developed based on database searches | 440 | N/A | Sensitivity/recall, precision | 2003–2015 | SR filters incorporated into the databases; MEDLINE used Montori 2005 |
|
Shojania 2001 [21] Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy. |
Evaluate a search strategy for identifying SRs | NR | Search filter evaluation | Treatment diagnosis, prognosis, causation, quality improvement, or economics | N/A | MEDLINE (PubMed) | Handsearching, Developed based on database searches | 104 | No | Sensitivity/recall, precision | 1999–2000 | PubMed n.d. |
|
White 2001 [18] A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. |
Improve methods to derive a more objective search strategy to identify SRs in MEDLINE | NR | Search filter evaluation, Comparative search filter evaluation | Treatment diagnosis, prognosis, causation | 7 (5) | MEDLINE (Ovid) | Handsearching journals | 110 | No | Sensitivity/recall, precision | 1995 and 1997 | White 2001 (five] Boynton 1998 CRD - Wolf 1996 |
|
Wilczynski 2007 [22] EMBASE search strategies achieved high sensitivity and specificity for retrieving methodologically sound systematic reviews. |
Develop search strategies that optimize the retrieval of SRs from EMBASE. | NR | Search filter evaluation | Internal medicinegeneral practice, mental health, nursing practice | N/A | MEDLINE | Handsearching journals | 220 | No | Sensitivity/recall, specificity, precision, accuracy | 2000 | Wilczynski 2007 |
|
Wilczynski 2009 [23] Consistency and accuracy of indexing systematic review articles and meta-analyses in medline. |
Determine the consistency and accuracy of indexing SRs and meta-analyses in MEDLINE | NR | Search filter evaluation | Medicine | N/A | MEDLINE | Developed based on database searches | NA | No | Sensitivity/recall, specificity, precision, accuracy | 2000 | Wilczynski 2009 |
|
Wilczynski 2011 [24] Sensitive Clinical Queries retrieved relevant systematic reviews as well as primary studies: an analytic survey. |
Determine how well the previously validated broad and narrow Clinical Queries retrieve SRs | NR | Search filter evaluation | Therapy, diagnosis prognosis, etiology | N/A | MEDLINE, EMBASE, CINAHL, and PsycINFO | Developed based on database searches | NA | No | Sensitivity/recall, specificity precision | 2000 | Wilczynski 2011 |
|
Wong 2006 [14] Comparison of top-performing search strategies for detecting clinically sound treatment studies and systematic reviews in MEDLINE and EMBASE. |
Compare sensitivity and specificity of search strategies for detecting reviews in MEDLINE and EMBASE | NR | Comparative search filter evaluation | Medicine | 7 | MEDLINE, EMBASE | Handsearching journals | 753 in MEDLINE, 220 in EMBASE | N/A | Sensitivity/recall, specificity precision | 2000 | Montori 2005 (three) Wilczynski 2007 (four] |
|
Wong 2006 [26] Optimal CINAHL search strategies for identifying therapy studies and review articles. |
Design optimal search strategies for locating review articles in CINAHL | NR | Search filter evaluation | Nursing and allied health | N/A | CINAHL | Handsearching journals | 127 | No | Sensitivity/recall, specificity precision, accuracy | 2000 | Wong 2006 |
|
Zacks 1998 [25] Developing search strategies for detecting high quality reviews in a hypertext test collection. |
Determine whether sensitive and specific search strategies exist to select SRs | NR | Search filter evaluation | Etiology, prognosis, therapy diagnosis | N/A | SWISH v.1.1.1 | Developed based on database searches | 209 | No | Sensitivity/recall, specificity | Not reported | Zacks 1998 |
Sensitivity/recall is defined as the proportion of relevant reports correctly retrieved by the filter; Precision is the number of relevant reports retrieved divided by the total number of records retrieved by the filter; NNR is the inverse of the precision; Specificity is the proportion of irrelevant reports correctly not retrieved by the filter, Accuracy is the proportion of all reports that are correctly classified
CDSR Cochrane Database of Systematic Reviews, CRD Centre for Review and Dissemination, n.d. no date, NNR number needed to read, N/A not applicable, NR not reported, PH public health, SRs systematic reviews, SWISH Simple Web Indexing System for Humans