Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery

Abstract

Objective:

Systematic reviews require comprehensive literature search strategies to avoid publication bias. This study aimed to assess and evaluate the reporting quality of search strategies within systematic reviews published in the field of stereotactic radiosurgery (SRS).

Methods:

Three electronic databases (Ovid MEDLINE^®, Ovid EMBASE^® and the Cochrane Library) were searched to identify systematic reviews addressing SRS interventions, with the last search performed in October 2014. Manual searches of the reference lists of included systematic reviews were conducted. The search strategies of the included systematic reviews were assessed using a standardized nine-question form based on the Cochrane Collaboration guidelines and Assessment of Multiple Systematic Reviews checklist. Multiple linear regression analyses were performed to identify the important predictors of search quality.

Results:

A total of 85 systematic reviews were included. The median quality score of search strategies was 2 (interquartile range = 2). Whilst 89% of systematic reviews reported the use of search terms, only 14% of systematic reviews reported searching the grey literature. Multiple linear regression analyses identified publication year (continuous variable), meta-analysis performance and journal impact factor (continuous variable) as predictors of higher mean quality scores.

Conclusion:

This study identified the urgent need to improve the quality of search strategies within systematic reviews published in the field of SRS.

Advances in knowledge:

This study is the first to address how authors performed searches to select clinical studies for inclusion in their systematic reviews. Comprehensive and well-implemented search strategies are pivotal to reduce the chance of publication bias and consequently generate more reliable systematic review findings.

INTRODUCTION

Systematic reviews of literature generate important information for clinical decision-making. Furthermore, systematic reviews can map the evidence and document knowledge gaps in the literature. These reviews synthesize all potentially relevant information to answer a research question descriptively or with a meta-analytic estimate.¹ To answer a research question adequately, a systematic review should also contain comprehensive information to avoid providing biased results. Evaluation of only a part of the relevant literature will bias treatment effect estimates, inflating or reducing them, or even changing the direction of results.² As a consequence, this information, that is normally used to develop clinical guidelines, may be biased. Therefore, clinicians may make clinical decisions based on biased information. Thus, systematic reviews should be performed with sensitive search strategies to avoid publication bias.³

Whether the search strategies reported in systematic reviews in the field of stereotactic radiosurgery (SRS) employ evidence-based recommendations that facilitate the improvement of search quality is unknown.⁴ Thus, the main objective of the present work was to evaluate the quality of search strategies reported in systematic reviews in the field of SRS using criteria from the Cochrane handbook for systematic reviews³ and the Assessment of Multiple Systematic Reviews (AMSTAR) checklist.⁵

METHODS AND MATERIALS

Eligibility criteria

Included studies were systematic reviews with or without meta-analyses in which the systematic review technique was examined as an intervention or comparator and in combination with other radiotherapy techniques or treatment strategies (i.e. general surgery, conservative management). For the purposes of this study, an article was considered to be a systematic review or meta-analysis if it contained these terms or if the main text indicated clearly that the authors had performed a systematic review and included either search terms or medical databases or both. Studies that included phantom or animal models or that were written in languages other than English were excluded.

Literature search

One author (JW) performed a structured literature search of the Ovid MEDLINE, Ovid EMBASE and Cochrane Library databases through October 2014 using medical subject headings and free-text terms related to “stereotactic radiosurgery” AND “systematic review” OR “meta-analysis”. The predefined search strategy was designed for maximal retrieval. The thesaurus vocabulary of each database was used to adapt the search terms. In addition to these automated searches, the reference lists of selected systematic reviews and related journal articles and the contents of six journals (International Journal of Radiation Oncology; Biology and Physics; Radiotherapy and Oncology; Radiation Oncology; Journal of Medical Imaging and Radiation Oncology; Seminars in Radiation Oncology; and Practical Radiation Oncology) were hand searched.

Study selection

Two reviewers (CMF and JW) first scanned all abstracts retrieved in the initial search to exclude irrelevant studies and then screened the remaining titles and abstracts to identify those that met the inclusion criteria. Full-text articles were then retrieved by one author (JW) and reviewed independently by two authors (JW and CMF) to evaluate fulfilment of the inclusion criteria. All differences of opinion were resolved by discussion among the authors.

Data extraction

We extracted data using nine questions based on the recommendations detailed in the Cochrane handbook³ and a validated checklist for the methodological assessment of systematic reviews and meta-analyses, the Measurement Tool to Assess Systematic Reviews (AMSTAR).⁵ The questions are presented in Table 1.

Table 1.

Standardized questions used to evaluate search strategies reported in stereotactic radiosurgery systematic reviews

Item	Description
1	Were at least two electronic databases searched?
2	What was the number of electronic databases searched?
3	Was the search strategy described? (a) Only keywords reported? (b) Keywords + Boolean operators that make the search reproducible?
4	Was grey literature searched (namely, information that is not published in easily accessible journals or databases, such as conference proceedings, that include the abstracts of research presented at conferences or unpublished theses (http://www.cochrane.org/glossary)?
5	Was “hand-searching” performed (namely, information that includes the searching of reviews, textbooks, reviewing the references of the selected studies and contacting specialized registers or experts in the particular field of study?
6	Were the literature searches conducted in duplicate?
7	Was the literature search performed without language restriction?
8	Were the authors of primary studies or manufacturing companies contacted by authors of systematic review to obtain further relevant information?
9	Were the interface(s) to search in the electronic databases reported by authors of systematic review?

One author (JW) entered responses directly into a standardized form created with Microsoft Excel^® (Microsoft, Redmond, WA). A second author (CMF) pilot tested the standardized form with five systematic reviews. The data were cross-checked for precision by one author (CMF). All questions were answered with YES (authors adequately addressed the question) or NO (authors inadequately addressed the question or information was insufficient to answer the question). Other relevant data such as the type and number of databases searched were reported descriptively. Any disagreement in the data-extraction process was resolved by discussion between the two authors until consensus was achieved. We also analyzed the following factors.

Impact factors

Journal impact factors (IFs), in which the systematic review was published, were retrieved from the Journal Citation Reports^® of the ISI Web of Knowledge. The current IF, rather than 5-year IF, was used. The IFs were categorized as ≤2, 2–4 and >4. Some evidence suggest that a higher IF indicates a higher paper quality.⁶

Radiation oncology journals

We checked journal names in PubMed to identify systematic reviews published in radiation oncology or oncology-based journals. Any journal whose name or single citation list included the word “radiation”, “radiotherapy” or “oncology” (including derivatives) was considered to be a discipline-specific journal.

Content scope

We classified systematic reviews according to intervention using the following categories: SRS or stereotactic radiotherapy, stereotactic body radiation therapy or stereotactic ablative radiotherapy and other treatment strategies (i.e. one or more aspects of systematic reviews combined with or separate from general surgery or conservative management).

Statistical analysis

The results of literature searches covering different periods (before 2005, 2006–10 and 2011–15) were compared to evaluate the pattern of quality over time. The frequencies of questions answered with YES in different periods were analyzed using Fisher's exact test. Linear regression was used to determine the extent to which journal characteristics and study demographic variables were univariate and multivariate predictors of search strategy quality. The quality score for each study was obtained by summing the number of items answered. Variables with p-values <0.05 in univariate analyses and those judged to be important were entered into multivariate regression models to identify those associated independently with the outcome. Results of regression analyses are reported as regression coefficients with 95% confidence intervals and R² statistic to indicate the amount of variation in the outcome explained by the predictor variables. Two-sided p-values <0.05 were considered to indicate statistical significance. All analyses were performed with SAS^® v. 9.3 (SAS Institute Inc., Cary, NC) software.

RESULTS

Characteristics of included systematic reviews

The initial search yielded 702 studies. The initial application of inclusion and exclusion criteria resulted in the exclusion of 596 irrelevant studies. Full-text screening of the remaining 106 articles led to the exclusion of 21 reviews that did not meet the criteria for a systematic review, leaving a total of 85 systematic reviews to be included (Supplementary Appendix). The majority of the systematic reviews in the sample were published in neurological and/or neurosurgery journals, with >50% published in North America and South America. Almost 50% of systematic reviews had five or more authors and >50% were published in 2012 or thereafter. Only one-third of systematic reviews presented meta-analytic estimates. Table 2 shows the overall systematic review characteristics, journal demographics and quality of reporting measures of the included systematic reviews. The list of included and excluded articles is reported in the Supplementary material.

Table 2.

Characteristic and quality mean score of the 85 systematic reviews included in the study

Characteristic	Category	n	%	Score (mean)
Journals	Neurology, neurosurgery	28	32.9	1.61
	Ear, nose and throat, including head/neck	6	7.1	1.33
	Endocrinology, dermatology	2	2.4	2.50
	Cochrane Library	5	5.9	2.80
	General medical	5	5.9	2.40
	Radiation therapy/oncology	39	45.9	2.41
Continents	Europe	21	24.7	2.14
	Americas	45	52.9	1.96
	Oceania	17	20.0	2.47
	Middle East	2	2.4	1.50
Authors	<4	17	20.0	1.88
	4–5	26	30.6	2.23
	>5	42	49.4	2.10
Years	Before 2005	12	14.1	1.67
	2006–10	25	29.4	2.04
	2011–15	48	56.5	2.23
Meta-analysis	No	53	62.4	1.81
	Yes	32	37.7	2.56
Impact factor	<2	25	29.4	1.68
	2–4	31	36.5	2.00
	>4	29	34.1	2.55

Quality of search strategies

More than half (56%) of the systematic reviews involved searches of at least two electronic databases for primary studies (median = 3). Database interfaces were reported in 59 (69%) systematic reviews. Searches of PubMed or the Cochrane Library accounted for many cases in which the platform was known. 45 (53%) systematic reviews reported only keywords used for searches, whereas 40 (47%) systematic reviews reported on the use of Boolean-based search strategies, which enabled reproducibility. The authors of 12 (14%) systematic reviews searched the grey literature. The authors of seven (8%) systematic reviews conducted hand searches, and searches were performed in duplicate in two (2%) systematic reviews. In seven (8%) systematic reviews, searches were performed with no language restriction. The authors of three (4%) systematic reviews contacted the authors of primary studies in attempts to retrieve further additional studies on the topic. No significant difference in the search strategy quality was observed among the three evaluated publication periods (Table 3).

Table 3.

Number of studies (percentage) reporting on the selected items to assess the search strategies in different periods

Items	Before 2005 (n = 7)	2006–10 (n = 18)	2011–15 (n = 60)	Total
At least two databases searched	3 (43)	7 (39)	38 (63)	48 (56)
Search terms described	3 (43)	10 (56)	32 (53)	45 (53)
Search strategy described	4 (57)	8 (44)	28 (47)	40 (47)
Grey literature searched	0 (0)	1 (6)	11 (18)	12 (14)
Hand search performed	0 (0)	1 (6)	6 (10)	7 (8)
Search performed in duplicate	0 (0)	1 (6)	1 (2)	2 (2)
Language restriction	0 (0)	0 (0)	7 (12)	7 (8)
Contact with authors	0 (0)	1 (6)	2 (3)	3 (4)
Interface reported	2 (29)	14 (78)	43 (72)	59 (69)

Data are presented as number (percentage) unless otherwise stated.

Predictors of quality

Relationships between systematic review characteristics and mean quality scores are shown in Table 4. Univariate analyses demonstrated that a meta-analysis performance, publication in a journal with a high (≥4) IF and publications in discipline-specific journals (e.g. head/neck, endocrinology and dermatology specialities) were predictors of higher mean quality scores. In the multivariate linear regression model with the variables of journal category, continents, number of authors, publication year, with/without meta-analysis estimates and IF, only meta-analysis performance remained a predictor of higher mean quality scores. After selection of prediction models, year of publication, IF and systematic reviews including a meta-analysis were significantly associated with mean quality score of search strategies (Table 5).

Table 4.

Uniariate and multivariate linear regression with mean score of quality as a dependent variable for the included systematic reviews (n = 85)

Predictor variables	Category or unit	Univariate analysis		Multivariate analysis
		β	95% confidence interval	β	95% confidence interval
Journals	Neurology/neurosurgery	Baseline (reference)		Baseline (reference)
	ENT, head/neck	−0.80	−1.34, −0.27a	−0.54	−1.18, 0.09
	Endocrine, skin structures	−1.08	−2.03, −0.13b	−0.80	−2.00, 0.40
	Cochrane Library	0.09	−1.48, 1.66	−0.30	−1.97, 1.37
	General medicine—lower gastrointestinal	0.39	−0.64, 1.42	0.25	−0.80, 1.30
	Radiation therapy/oncology	−0.01	−1.04, 1.02	0.08	−0.96, 1.13
Continents	Europe	Baseline (reference)		Baseline (reference)
	America	−0.19	−0.79, 0.42	−0.43	−1.00, 0.28
	Oceania	0.33	−0.42, 1.07	−0.11	−0.78, 0.77
	Middle East	−0.64	−2.34, 1.05	−0.14	−1.75, 1.65
Authors	<4	Baseline (reference)		Baseline (reference)
	4–5	0.35	−0.37, 1.07	0.28	−0.44, 1.00
	>5	0.21	−0.45, 0.88	0.26	−0.49, 0.94
Years	2000–08	Baseline (reference)		Baseline (reference)
	2009–11	0.37	−0.43, 1.18	−0.09	−0.95, 0.81
	2012–15	0.56	−0.17, 1.30	0.07	−0.65, 0.97
Meta-analysis	No	Baseline (reference)		Baseline (reference)
	Yes	0.75	0.26, 1.24a	0.73	0.15, 1.22b
Impact factor	<2	Baseline (reference)		Baseline (reference)
	2–4	0.32	−0.27, 0.91	0.10	−0.55, 0.75
	≥4	0.87	0.27, 1.47a	0.38	−0.37, 1.13

ENT, ear, nose and throat.

^ap < 0.01.

^bp < 0.05.

Table 5.

Multivariate linear regression with mean score of quality as a dependent variable for the included systematic reviews (SR) (n = 85)

Variable	Estimate	95% confidence interval
Year	0.09a	0.02	0.16
Meta-analysis vs SR	0.57a	0.09	1.06
Impact factor	0.08a	0.01	0.15

Year and impact factor included in the linear regression are continuous variables.

^ap < 0.05.

DISCUSSION

To our knowledge, this study is the first to evaluate the reporting quality of search strategies within systematic reviews published in the field of SRS. Our findings suggest that search strategies within these systematic reviews need to be more comprehensive, with the inclusion of multiple sources of evidence. We documented great variability in the percentage of questions answered adequately (3–89%, median = 46%), which showed that there was room for improvement. Furthermore, the results of this study identified year of publication, IF and meta-analysis performance as predictors of search strategy quality.

Nearly half of the systematic reviews in our sample involved searches of only one database. Searches of limited numbers of databases may be insufficiently comprehensive for the retrieval of the information required to answer a clinical question; although some overlapping of published material across databases occurs, some information may be published only in specific databases.³ Interestingly, some evidence from a methodological study examining systematic reviews of therapeutic interventions suggest that searching of multiple databases (i.e. sources other than PubMed) has only a small benefit.⁷ However, this study included only randomized controlled trials,⁷ which did not reflect the various study designs found within our included SR. Thus, until new and robust evidence on the minimum number of databases that should be searched is available, searching of several databases would be sensible.

About 50% of systematic reviews in our sample reported only keywords used for literature searches. This percentage was smaller than that reported in another study⁸ evaluating search strategies used in systematic reviews published in dentistry (70%). Search strategies should be reported in full, with the description of keywords and Boolean operators exactly as used in database searches. This approach would allow any interested reader to replicate the search output. In the present sample, slightly <50% of systematic reviews described the full search strategy. Although the preferred reporting items for systematic reviews and meta-analysis (PRISMA) guidelines recommend the publication of a reproducible search strategy, many of the systematic reviews failed to describe a strategy, much less a reproducible one. The criteria for search reproducibility used in this study were not stringent, as only complete Boolean logic and search statements were considered. Another important issue in facilitating the replication of a literature search is the reporting of interfaces used in electronic database searches. In the present sample, about two-thirds of systematic reviews reported these interfaces.

A grey literature search is an important component of a search strategy. The grey literature comprises all information that is not readily available, including unpublished studies.⁹ Evidence suggests that studies with positive results are twice as likely to be published as those with non-significant results.¹⁰ Thus, authors of systematic reviews examining interventions must make all possible efforts to retrieve unpublished trial data, to provide reliable treatment effect estimates. In the present sample, 14% of systematic reviews described some approach to searching grey literature, which was smaller than that reported for a study published in dentistry (34%).⁸

Other measures to reduce or minimize publication bias in systematic reviews examining interventions are hand searching and the performance of searches with no language restriction. Hand searching, which involves mainly the scrutiny of journal issues and reference lists of studies, is an important approach to increase the sensitivity of searches, especially when bibliographic database searches are not comprehensive. Although some evidence suggest that searching for documents published in English alone does not affect the size of treatment effect estimates,^11,12 this assumption may not be representative of all medical fields. Systematic review authors should thus search for publications in languages other than English. Another issue is the possibility of location bias,¹³ created when authors seek publication of their research findings based on the hierarchy of journal importance. The majority of highly ranked medical journals are published in English. Thus, findings may differ between articles published in highly ranked journals and those published in languages other than English.

Regression analyses showed that systematic reviews with meta-analyses published recently in journals with high IF were more likely to have employed comprehensive search strategies. These results could not be compared with those of other studies published in other radiation-based specialities; therefore, to our knowledge, our study was the first to evaluate predictors of the quality of search strategies used in systematic reviews on SRS.

This study has several limitations. As the sample included only systematic reviews related to SRS interventions, the results could not be extrapolated directly to all systematic reviews in the field of radiation oncology. In addition, our search was restricted to SR in English only. Although the sample included only systematic reviews published in English, most of these articles were published in radiation oncology-based journals with high IF, which typically have rigid criteria for publication. Journal IFs have been suggested to be a proxy for quality.⁶ However, the association between quality and IF should be interpreted cautiously, because the IF system may be susceptible to several forms of manipulation, resulting in deviation from true values.¹⁴ Thus, the focus should be on the manuscript quality, rather than journal IF. Finally, the present study focused hand search on a limited number of scientific journals. Nevertheless, searches in three major electronic databases and in the references of the retrieved systematic reviews may have compensated this limitation.

When the present findings are compared with a methodological study including 327 interventional systematic reviews published in 118 journals involving all clinical medicine and public health specialities, the number of systematic reviews with comprehensive search strategies seems to be lower in the present study (56% against 64.83%). Fleming et al¹⁵ used only the AMSTAR criteria for evaluating the systematic reviews.

Recommendations for future systematic reviews

We recommend the following measures to improve the methodological quality of search strategies in systematic reviews within the field of SRS:

• Systematic review authors should consider using electronic databases that follow evidence-based guidelines.⁴ We propose that authors adhere to the standards for the conduction of different steps of a systematic review, including search strategy development and implementation, as suggested by the Cochrane Collaboration.¹⁵

• Systematic review authors should consider the inclusion of an information specialist, such as a librarian, to collaborate in the development and adjustment of search strategies for different databases. Some data suggest that the participation of librarians and information specialists in systematic review projects is correlated with higher quality search strategies and reporting thereof.¹⁶

• Systematic review authors should conduct hand searches and investigate the grey literature to retrieve information that is not available in electronic databases. No clear guidelines for grey literature search strategies have been established, but a good start would be the utilization of several specialized databases that make evidence from the grey literature available. Not including grey literature in the meta-analysis might lead to inflated treatment effect estimates.¹⁷

• Systematic review authors should apply no language restriction to searches. The research team should consider the inclusion of “searchers” with proficiency in several languages to optimize literature selection. Another measure is to obtain translations of documents that may be considered for inclusion, although this approach would increase the cost and logistical complexity of the project considerably.

• Search strategies should be reported completely and transparently to enable the replication of findings.¹⁸ Journal editors could request complete “raw” data on searches for publication as Supplementary material.

CONCLUSION

The present study evaluated the search strategies of 85 systematic reviews in the field of SRS. Our results showed no significant changes to the search strategy structure since 2000. One may consider a high chance of having biased estimates since this year. Therefore, these findings suggest an improvement in the development of search strategy techniques, as the reliability of results found in a systematic review would be dependent on a comprehensive and well-implemented search strategy. Future systematic reviews published in the field of SRS should strictly observe high standards for the development and implementation of search strategies.

FUNDING

The authors Yun-Chun Wu and Yu-Kang Tu were partly supported by a grant from the Ministry of Science and Technology in Taiwan (grant number: MOST 103-2314-B-002-032-MY3).