Abstract
Setting:
Although an important part of the evidence base in health, systematic reviews are not always easy to find. Difficulties are compounded when interventions under review are “social and environmental” (that is, targeting wider determinants of health). The authors explored searches from a descriptive map containing thirty-two systematic reviews evaluating the effectiveness of social and environmental interventions for childhood obesity.
Questions:
Which sources give the highest yield of relevant reviews per 100 records? What is the value of searching databases that index literature beyond the “health” arena when looking for data on the effectiveness of social and environmental interventions?
Methods:
The authors analyzed search results from nineteen databases and calculated the precision and the relative and unique contribution of each source.
Results:
Searches of specialist systematic review databases—Database of Abstracts of Reviews of Effects (DARE), Database of Promoting Health Effectiveness Reviews (DoPHER), and Health Technology Assessment (HTA)—had the highest precision, although MEDLINE, CINAHL, and PsycINFO located many additional reviews. The Cochrane Database of Systematic Reviews should be searched for health-related reviews. Searches of education, transportation, social policy, and social sciences databases did not identify additional reviews. Searching websites and bibliographies was important.
Conclusions:
Searches for review-level evidence could profitably start with the specialist review databases. Searches of the major health-related databases are essential, but database searching beyond them may not identify much additional evidence. Internet and hand-search remain important sources of reviews not found elsewhere. Comparison of the results with previous research suggests that appropriate sources for locating primary and secondary evidence may be different.
Highlights.
Specialist review registers including the Database of Promoting Health Effectiveness Reviews (DoPHER), Database of Abstracts of Reviews of Effects (DARE), and Cochrane Database of Systematic Reviews are likely to be time efficient and good places to start searching for reviews on the social and environmental determinants of health.
Databases beyond the health arena did not yield many extra, relevant studies. This may inform the search choices for rapid evidence assessments or other resource-limited searches.
Implications.
The traditional health databases (PubMed, PsycINFO, CINAHL, and Health Technology Assessment [HTA]) should be searched for completeness.
Supplementary searches (reference checking and Internet searches) remain important as they identified a substantial number of data not found by any other source.
The choice of search sources should be informed by the type of evidence that is sought. Appropriate sources for locating primary and secondary evidence may be different, even when topic areas are similar.
INTRODUCTION
Systematic reviews are an important part of the evidence base in public health. By locating and using all of the best available evidence, systematic reviews can overcome the potential for bias that is found in individual studies and provide reliable and accessible answers to questions about policy and practice. Systematic reviews also inform future research activity by identifying gaps and preventing duplication of effort. However, it can be difficult to know where to look for systematic reviews given the multiple research databases that index them and the fact that many are, at least initially, published as reports that may not be indexed in bibliographic databases.
There is little research investigating the best sources for locating systematic reviews. Research that has been done in the area focuses on the development of effective search terms and strategies (but not sources) for locating reviews, either in medical databases [1–3] or across a broader range of databases [4]. During scoping searches, the authors only found one previous study evaluating search sources for locating reviews. The study described searches for reviews in four databases (MEDLINE, EMBASE, Cochrane Database of Systematic Reviews, and “Best Evidence”) and concluded that results from EMBASE and MEDLINE were equally profitable for locating reviews [5].
The choice about which sources to search is difficult when the topic is multidisciplinary in nature, as is often the case in public health research [6]. For example, researchers or policy makers who wish to identify systematic reviews that investigate the effects of social and environmental interventions (that is, targeting the wider determinants of health) will be interested in interventions that lie beyond the traditional health arena. Consequently, research is likely to be located in publications (and databases) that span a range of disciplines and topic areas. A methodological study by Ogilvie and colleagues investigated the productivity of different search sources across a range of disciplines, when searching for primary research on one type of social and environmental intervention. Ogilvie and colleagues' study found that most relevant research about interventions promoting a population shift from car use to active transport, with the aim of increasing physical activity, was not located in the familiar health or science databases [7]. Instead, it was found in topic-specific databases. As far as we know, there has been no similar evaluation of search sources for systematic reviews.
The analysis of search sources presented here is based on a systematic descriptive map that aimed to describe the range, focus, and quality of existing systematic reviews investigating the effectiveness of social and environmental interventions to reduce obesity and overweight among children (referred to as “the map” in this paper) [8]. Social and environmental interventions included those that aimed to alter the physical environment, social norms, technology, or the economy to impact eating behaviors, activity levels, weight status, or relevant attitudes. The map aimed to locate evidence from outside the traditional health arena and to bring together a wide range of research from health, transportation, physical activity, and food policy.
OBJECTIVES
The research sought to:
evaluate the precision of each search (identify the yield of systematic reviews that we included in our map per 100 records generated by each database) in order to measure how resource intensive it was to find and extract each relevant review from all the records generated
assess the relative and distinct contribution of each database source to the findings of our descriptive map in order to assess how necessary it was to conduct searches of databases that indexed literature beyond the health arena
METHODS
Searches
In November and December 2007, we conducted systematic searches for a descriptive map of reviews on the effectiveness of social and environmental interventions to reduce obesity in children. The searches covered nineteen electronic databases, websites, and relevant bibliographies that were from the fields of health, social science, physical activity, and transportation and included specialist registers (Table 1) [8]. We developed searches iteratively for each database and used a complex set of terms for the following concepts:
Table 1.
Seven source categories
(physical activity OR sedentary behaviour OR eating OR obesity) AND social and environmental AND review AND date (published since 1986))
Our searches were designed to be sensitive and wide-ranging, and our definition of “review” was very broad in order to locate evidence that might not be described as a “systematic review” or that did not use our “social and environmental” terminology. For example, our search terms for physical activity included those referring to activity, exercise, and sports generally; to active transport; to non-sedentary leisure activities such as gardening; and to children's nondirected “play.” We used a combination of index terms and free text for each concept to ensure that our results were not biased by how well individual databases were indexed. Full details of the strategies for each database are available from the Evidence for Policy and Practice Information and Co-ordinating (EPPI)-Centre website [9].
We did not search the Cochrane Database of Systematic Reviews directly as this source is searched quarterly and uploaded to the EPPI-Centre's own specialist review database, the Database of Promoting Health Effectiveness Reviews (DoPHER), which was searched. The search strategy used to locate public health and health promotion reviews on the Cochrane Database of Systematic Reviews has been developed over many years and is highly sensitive [10]. It includes exploded Medical Subject Heading (MeSH) terms for health education, health promotion, public health, preventive health services, preventive medicine, primary health, and primary prevention along with free text terms. We anticipated that DoPHER would include all relevant material listed in the Cochrane Database of Systematic Reviews.
With the aim of making our future searches more efficient, once the descriptive map was published, we retrospectively analyzed the usefulness of the search sources, and these analyses are presented here.
Inclusion criteria for systematic reviews
We included systematic reviews that investigated an obesity-relevant topic (broadly defined, see full report [8]); reviewed the effectiveness of interventions (broadly defined, see full report [8]); included the four-to-eighteen age group; and focused on “social and environmental” interventions (i.e., interventions targeting the wider determinants of obesity). Although we searched for reviews published since 1986, to restrict the dataset to manageable proportions, we only included reviews published in or after 1996. We applied the 1996 cut off at the first screening stage (title and abstract). At this stage, forty-five records were excluded on the basis of date alone (i.e., the abstract suggested that the review appeared to meet all our other inclusion criteria or had insufficient detail to tell whether all criteria where met). We found that there was often limited detail in abstracts about the methods used and interventions reviewed, and many of the full texts that we retrieved did not in fact meet our inclusion criteria relating to systematic review, age of target population, or social and environmental nature of the interventions. It is therefore likely that many of the forty-five records published before 1996 would also have been excluded, based on these criteria, had we had detail available in full text.
We defined reviews as “systematic” if they clearly stated their aims, search strategy, and inclusion criteria. We did not further appraise the quality of reviews. The records were divided between 3 reviewers who independently “screened” records on title, abstract, and, where records were potentially relevant, full text. “Screening” is the process of systematically checking all the bibliographic records captured electronically against predefined inclusion criteria to identify relevant studies. For quality assurance, the first 5% of all titles and abstracts were screened by all 3 reviewers, who then compared results and recalibrated their screening technique to improve consistency. All 3 reviewers were on hand throughout screening to discuss and resolve any uncertainties.
Analysis
We grouped the search sources into seven broad categories, some based on topic, others based on the type of research indexed, such as the registers of reviews (Table 1). The grouping of the sources into categories was suggestive rather than definitive. We analyzed the different sources in three ways (Table 2). All analyses were carried out using EPPI-Reviewer data-management software [11].
Table 2.
Analyses carried out to assess the performance of each search source
When conducting the searches grouped in the “other” category, the reviewers used careful judgment to identify potentially relevant records. They were, therefore, partially screened for relevancy before being uploaded. In addition, researchers did not upload records that they knew had already been found by other sources. As a result, the “other” searches cannot be compared on equal terms with the database searches in terms of precision and sensitivity and are not included in Table 3. However, the “other” searches are comparable in terms of their contribution of unique reviews and are included in Table 4.
Table 3.
Sensitivity and precision of each search source, grouped by broad category
Table 4.
Cumulative search of “health” databases, “specialist review registers” databases, and “other” searches
RESULTS
The original searches from databases generated 6,197 records, of which 697 were duplicates. We excluded 5,167 records on abstract and title and retrieved 333 full reports. We included a total of 32 relevant systematic reviews from all sources. For full details, see our map [8].
Which sources supported the most precise searches?
The source supporting the most precise search was the specialist review database, DoPHER. This source contained 13 of the systematic reviews that we included in our map and generated a total of 381 search results, making 3.4% of the records relevant (Table 3). The search of the Health Technology Assessment database (HTA) was the second most precise, as we included 2.3% of records generated by this search in our map.
Although the PubMed search located the second highest absolute number of included systematic reviews (n = 7), it was a labor-intensive source to screen. The 7 relevant reviews only accounted for 0.2% of the 3,167 records generated by the PubMed search. In other words, to get to every relevant systematic review in the PubMed results, we had to sift through about 500 nonrelevant records. In contrast, when screening the source supporting the most precise search, DoPHER, we only had to sift through about 30 nonrelevant records for every relevant systematic review (Table 3). Given that our search dates were wider than our inclusion dates, it is possible that our analyses underestimate the precision of all databases and especially of the older, larger databases such as PubMed.
Five sources supported moderately precise sources and gave a yield of between 1 and 2 relevant systematic reviews per 100 records generated. Two of these were results from databases focusing on health research: CINAHL and PsycINFO. Three came from sources focusing on social science and physical activity: Social Sciences Citation Index/Science Citation Index, Arts & Humanities Citation Index, and SPORTDiscus, respectively. One was a specialist review register: DARE.
Nine of the 19 databases did not contain any records that we included in our map, although they generated a total of 884 records, all of which had to be screened. Many of these 9 databases focused on social policy, education, or social science research (Table 3).
How important was it to search beyond the traditional health databases?
Table 3 shows that, when combined, the search results from the 4 health databases contained almost half (n = 15) of the total relevant systematic reviews that we located. The specialist review registers located 17 relevant systematic reviews, more than half of all that we found. While these 2 sources made a large contribution, many sources did not uniquely locate any systematic reviews and so did not contribute anything to our findings.
Seventeen of the 32 included systematic reviews (53%) were unique (located by only 1 search source). Table 3 shows that 70% of these unique included reviews were found by the health databases and specialist review databases (29% and 41%, respectively). An additional 4 systematic reviews were unique to the results from our searching websites, library catalogs, and bibliographies of relevant research (classified as “other” in Table 1) (Table 4).
Apart from the specialist review registers, sources that had a focus other than health contributed little to our map. Table 4 shows that only one included systematic review would have been missed if we had limited our search to the health databases (which located fifteen unique reviews), the specialist review registers (which located twelve unique reviews), and website, library catalogs, and bibliographies of relevant research (which located four unique reviews). The one additional included systematic review that was not from these search sources was located in the results from Physical Education Index (Table 3).
DISCUSSION
Principal findings
Given their precision, the specialist review registers are likely to be most time efficient for researchers and decision makers looking for evidence in this area. The least time-efficient sources were the nine sources from social science, education, and social policy, which did not locate any of the systematic reviews included in our map but together generated 884 records (before de-duplication).
Although resource intensive to search (given the high numbers of records generated), the “health’ searches contained almost half (n = 15) of the 32 relevant systematic reviews. The results from searching the specialist review registers databases were both precise and made a high relative and unique contribution to our map. Four (13%) of the reviews we included were found only by searching websites, library catalogs, and relevant bibliographies (classified as “other”). Limiting our searching to these 3 groups of sources (health, specialist review, and “other”) would have located almost all (n = 31) of the 32 relevant reviews we found. Searches of 11 databases that focused on research outside the health arena contributed only 1 extra review to our map.
Strengths and weaknesses of this analysis
Our searches were designed to be extremely sensitive, and our inclusion criteria were broad. Therefore, it was likely that we found a large proportion of all relevant data that existed and that our results might be of use to many of those interested in the social and environmental determinants of health, regardless of their specific research question. Our quality assurance measures when screening (reviewer calibration through “double screening” and constant discussion) were designed to minimize reviewer bias and to promote consistency between reviewers when selecting relevant reviews.
However, our results were likely to be affected by the fact that some search sources would be a better “match” than others to our specific research question or topic. As might be expected, the source supporting most precise searches in our analysis was DoPHER, which was matched to our research question both in terms of the study type (reviews of effectiveness) and subject focus of the indexed reviews (social and environmental determinants of health). It also included sensitive searches of other highly relevant sources, as DoPHER is populated through regular searches of specialist databases of reviews of effectiveness, including the Cochrane Database of Systematic Reviews. Further reviews for DoPHER are identified through hand-searching and searches of relevant organizations' websites. Given the high relevance of DoPHER to our research question, it might not be such a rich search source for research questions outside the scope of effectiveness of interventions and/or public health. Development and maintenance of DoPHER is funded by the UK Department of Health to increase access to systematic and nonsystematic reviews of relevance to health promotion and public health. These results indicate that DoPHER is fulfilling this function.
Our results should be interpreted in the knowledge that the categories we used for the analyses are, in practice, not mutually exclusive. For example, DoPHER could be considered both a “specialist review register” database and a “health” database. Search sources should be chosen in light of the research question; those whose scope overlaps with the research question in more than one characteristic are likely to be most relevant.
It is also possible that the discrepancy between our search dates (1986 onward) and inclusion dates (1996 onward) may have underestimated precision in all databases and especially in the older, larger databases, such as PubMed.
For the interests of this paper and in anticipation that DoPHER would include all relevant Cochrane reviews at the time of our searches, we retrospectively tested this assumption. Two of the included reviews were Cochrane reviews [12, 13], which we found on 3 other sources: DoPHER, DARE, and PubMed. We retrospectively searched the Cochrane Database of Systematic Reviews using terms derived from our PubMed strategy, as both databases index research using the same MeSH. We analyzed the results in terms of sensitivity, precision, and unique reviews. The search yielded 523 records, 2 of which were reviews meeting all our inclusion criteria. These 2 reviews had been identified by DoPHER and other sources. The sensitivity of this search was 6.3% (comparable to our searches of HTA and PsycINFO), and precision was 0.4% (comparable to the large databases such as PubMed). We did not identify any unique reviews.
Whilst the Cochrane Database of Systematic Reviews was not searched directly for this review, we recommend that it should always be searched when trying to identify trials or systematic reviews, for a number of reasons. Whilst records of Cochrane reviews are included in sources such as PubMed, DARE, and DoPHER, there will always be a time lag between their appearance in the Cochrane Database of Systematic Reviews and in other sources. Uniquely, the Cochrane Database of Systematic Reviews also provides access to records of protocols for systematic reviews in progress. Furthermore, many countries now have free access to full-text Cochrane reviews that are not available via other sources.
Finally, like Ogilvie and colleagues, we only analyzed what we found in practice and not what could have been found had our search terms or inclusion criteria been different [7]. Testing the performance of the search strategy on each database would involve manually checking whether any of the thirty-two reviews were indexed in each of the sources at the time searches were run and, if indexed, why these reviews were not identified by that particular search. As a large number of databases was searched, a comprehensive analysis would be prohibitively time consuming and be subject to issues involving the staged timing of adding citations, abstracts, and index terms to databases and subsequently uploading abstracts and indexing them.
On balance, these results, although not a definitive answer to searching for systematic reviews, will help to inform future searches, especially those seeking to answer research questions about the effectiveness of social and environmental interventions. Further analysis of substantive reviews and reviews of reviews, such as this one, may help build a picture from which a pattern can be seen and wider conclusions drawn.
CONCLUSIONS
Our analysis suggests that information scientists, librarians, and researchers who want to access the best available reviews on the effectiveness of social and environmental interventions should start by searching the specialist review databases, including DoPHER, DARE, and the Cochrane Database of Systematic Reviews. To achieve any level of completeness, the traditional health databases—such as PubMed, CINAHL, PsycINFO, and HTA—should also be searched. Supplementary searches, such as searching carefully selected Internet sites and checking the bibliographies of relevant research, remain an important source of systematic reviews in this area.
The results of this analysis suggest that it may not be necessary to search databases outside the traditional health arena to locate systematic reviews about the effectiveness of social and environmental interventions, at least in the area of obesity. To be truly exhaustive, searches of topic-specific databases outside the health arena are still required. Careful consideration should be paid to the type of research question and the extent to which it is matched to the research contained in these databases before they are ignored.
The results of our analysis may inform tools for quality assessment of “rapid evidence assessments,” which use systematic reviews to answer questions in a short time frame. Our analysis suggests that a search of the mainstream health databases, particularly PubMed, may reasonably form the backbone of rapid search strategies in this area, particularly if they are supplemented by searches of the specialist review registers databases, the Internet, and bibliographies.
Our findings did not match those of Ogilvie and colleagues in their analysis of search sources for their review on promoting a population shift from using cars to walking and cycling [7]. Ogilvie and colleagues found that very few of their relevant studies were located by the mainstream health databases, and most were located by the topic-specific source TRANSPORT. As the scope of our systematic map covered many of the interventions reviewed by Ogilvie and colleagues (such as publicity campaigns to promote walking, financial incentives or taxes to reduce car use, and engineering measures to promote safety for cyclists and walkers), the differences between our findings suggest that sources that are very useful for locating primary research in this field are not necessarily the most useful for locating systematic reviews.
Acknowledgments
The views expressed in this paper are those of the authors and are not necessarily those of the department. We thank our advisory group and the EPPI-Centre health team on whose work and experience we drew when preparing this report.
Footnotes
The review, on which this methodological work is based, was supported by UK Department of Health (grant number: 013/0007).
Contributor Information
Jenny Woodman, Centre for Paediatric Epidemiology and Biostatistics, University College London-Institute of Child Health, 30 Guilford Street, London, WC1N 1EH, United Kingdom j.woodman@ich.ucl.ac.uk.
Angela Harden, Institute for Health and Human Development, UH250, Stratford Campus, University of East London, Romford Road, London E15 4LZ, United Kingdom a.harden@uel.ac.uk.
James Thomas, Email: J.Thomas@ioe.ac.uk.
Jeff Brunton, Email: J.Brunton@ioe.ac.uk.
Josephine Kavanagh, Email: J.Kavanagh@ioe.ac.uk.
Claire Stansfield, Evidence for Policy and Practice Information and Co-ordinating Centre, Social Science Research Unit, Institute of Education, 18 Woburn Square, University of London, London, WC1H 0NR, United Kingdom C.Stansfield@ioe.ac.uk.
REFERENCES
- 1.Shojania K.G, Bero L.A. Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy. Eff Clin Prac. 2001 Jul–Aug;4(4):157–62. [PubMed] [Google Scholar]
- 2.Montori V.M, Wilczynski N.L, Morgan D, Haynes R.B. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005 Jan;330(7482):68. doi: 10.1136/bmj.38336.804167.47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Boynton J, Glanville J, McDaid D, Lefebvre C. Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. J Info Sci. 1998;24(3):137–54. [Google Scholar]
- 4.White V.J, Glanville J.M, Lefebvre C, Sheldon T.A. A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. J Info Sci. 2001;27(6):357–70. [Google Scholar]
- 5.Hunt D.L, McKibbon K. Locating and appraising systematic reviews. Ann Intern Med. 1997 Apr 1;126(7):532–8. doi: 10.7326/0003-4819-126-7-199704010-00006. [DOI] [PubMed] [Google Scholar]
- 6.Armstrong R, Waters E, Roberts H, Anderson L.M, Oliver S, Petticrew M. Systematic reviews in public health. In: Heggenhougen K, Quah S, editors. International encyclopedia of public health. Vol 6. San Diego, CA: Academic Press; 2008. pp. 297–301. [Google Scholar]
- 7.Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. finding the evidence: how far should you go. J Epidemiol Community Health. 2005 Sep;59:804–8. doi: 10.1136/jech.2005.034181. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Woodman J, Lorenc T, Harden A, Oakley A. Social and environmental interventions to reduce childhood obesity: a systematic map of reviews. London, UK: Evidence for Policy and Practice Information and Co-ordinating Centre, Social Science Research Unit, Institute of Education, University of London; 2008. Available from: < http://www.eppi.ioe.ac.uk/cms/Default.aspx?tabid=2395>. [cited 5 Mar 2010]. [Google Scholar]
- 9.Woodman J, Lorenc T, Harden A, Oakley A. Social and environmental interventions to reduce childhood obesity: a systematic map of reviews (supplementary data: database strategies) [Internet] London, UK: Evidence for Policy and Practice Information and Co-ordinating Centre, Social Science Research Unit, Institute of Education, University of London; Apr 2008 [cited 10 Oct 2009]. < http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=2395&language=en-US>. [Google Scholar]
- 10.Brunton G, Waters E, Doyle Y, Kavanagh J, Shepherd J, Rees R, Harden A, Oliver S, Oakley A. Finding a haystack among the needles: making health promotion and public health reviews in the Cochrane Database of Systematic Reviews available to policy makers. 31 Jul–3 Aug 2002. 10th Cochrane Colloquium Abstracts; Stavanger, Norway;
- 11.Thomas J. EPPI-reviewer 3.0 (web edition) London, UK: Social Science Research Unit, Institute of Education, University of London; 2006. [Google Scholar]
- 12.Priest N, Armstrong R, Doyle J, Waters E. Interventions implemented through sporting organisations for increasing participation in sport. Cochrane Database Syst Rev. 2008 Jul;16(3) doi: 10.1002/14651858.CD004812.pub3. [DOI] [PubMed] [Google Scholar]
- 13.Priest N, Armstrong R, Doyle J, Waters E. Policy interventions implemented through sporting organisations for promoting healthy behaviour change. Cochrane Database Syst Rev. 2008 Jul;16(3) doi: 10.1002/14651858.CD004809.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Centre for Reviews and Dissemination. Systematic reviews: CRD's guidance in undertaking reviews in health care [Internet] York, UK: The Centre; 2008 [rev. 25 May 2009; cited 25 May 2009]. < http://www.york.ac.uk/inst/crd/systematic_reviews_book.htm>. [Google Scholar]
- 15.Centre for Reviews and Dissemination. CRD databases: DARE [Internet] York, UK: The Centre; 2008 [cited 20 Jun 2009]. < http://www.crd.york.ac.uk/crdweb/>. [Google Scholar]
- 16.Lorenc T, Brunton G, Oliver S, Oliver K, Oakley A. Attitudes to walking and cycling among children, young people and parents: a systematic review. J Epidemiol Community Health. 2008 Oct;62(10):582–7. doi: 10.1136/jech.2007.070250. [DOI] [PubMed] [Google Scholar]