Checking reference lists to find additional studies for systematic reviews

Abstract

Background

Checking reference lists to identify relevant studies for systematic reviews is frequently recommended by systematic review manuals and is often undertaken by review authors. To date, no systematic review has explicitly examined the effectiveness of checking reference lists as a method to supplement electronic searching.

Objectives

To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists.

Search methods

We searched the databases of The Cochrane Library (Issue 3, 2008), Library and Information Science abstracts (LISA) (1969 to July 2008) and MEDLINE (1966 to July 2008). We contacted experts in systematic review methods and examined reference lists of articles.

Selection criteria

Studies of any design which examined checking reference lists as a search method for systematic reviews in any area. The primary outcome was the additional yield of relevant studies (i.e. studies not found through any other search methodologies); other outcomes were publication types identified and data pertaining to the costs (e.g. cost‐effectiveness, cost‐efficiency) of checking reference lists.

Data collection and analysis

We summarized data descriptively.

Main results

We included 12 studies (in 13 publications) in this review, but interpretability and generalizability of these studies is difficult and the study designs used were at high risk of bias. The additional yield (calculated by dividing the additional 'unique' yield identified by checking reference lists by the total number of studies found to be eligible within the study) of relevant studies identified through checking reference lists ranged from 2.5% to 42.7%. Only two studies reported yield information by publication type (dissertations and systematic reviews). No cost data were reported although one study commented that it was impossible to isolate the time spent on reference tracking since this was done in parallel with the critical appraisal of each paper, and for that particular study costs were not specifically estimated.

Authors' conclusions

There is some evidence to support the use of checking reference lists for locating studies in systematic reviews. However, this evidence is derived from weak study designs. In situations where the identification of all relevant studies through handsearching and database searching is difficult, it would seem prudent that authors of reviews check reference lists to supplement their searching. The challenge, therefore, is for review authors to recognize those situations.

Plain language summary

Examining reference lists to find relevant studies for systematic reviews

Systematic reviews are summaries of the information that is available on one topic. The most common way to find information for a systematic review is to search electronic literature databases. To increase the chances of finding important information, researchers can also search the tables of contents of journals, and they can contact experts or organizations for more information on the topic of the review. Another way to find more information is to check through the reference lists of relevant studies to see if these references include reports of other studies that might be eligible for the review. It is important to determine whether or not checking reference lists is a good use of time and resources when conducting systematic reviews.

We found 12 studies that explored whether or not checking reference lists was useful for systematic reviews. These studies reported a range of results, from identifying only a few additional studies (2.5%: 2 of 79 included studies) to identifying many additional studies (42.7%: 111 of 260 included studies) through checking reference lists. Unfortunately, none of the studies looked at how much time or money were spent on the process of checking reference lists, and it was suggested this would be almost impossible to estimate.

Unfortunately our findings are based on weak information. The data do suggest that in situations where researchers may have difficulty locating information, checking through the reference lists may be an important way to reduce the risk of missing relevant information.

Background

Systematic reviews of interventions are considered to offer the most reliable evidence about the effects of interventions because their preparation requires not only methods to minimize various forms of bias, but also a comprehensive search for the evidence addressing the question of the review (Harbour 2001). Conducting a thorough and well‐documented search is one of the main distinctions between a systematic review and a 'traditional' narrative review, and is a vital part of ensuring that the systematic review provides a rigorous and comprehensive summary of research evidence on which appropriate clinical or policy decision‐making can be made (Sackett 1996).

In order to find all relevant studies systematic review authors may employ a number of strategies, including developing sensitive search strategies for electronic databases (e.g. MEDLINE) to increase the comprehensiveness of relevant citations (Berry 2000; Dickersin 1994; Lefebvre 2009). Other strategies to increase the yield may include handsearching and contacting individuals and other organizations (Hopewell 2007).

Another commonly applied strategy is the checking of reference lists of papers and reports already retrieved to identify additional, potentially relevant, records. Two case reports have been published examining the effectiveness of checking reference lists for the identification of relevant studies, but reported inconsistent results (Avenell 2001; Royle 2003). Moher et al examined a cohort of 300 systematic reviews published in 2004 to determine a broad range of epidemiological, descriptive and reporting characteristics (Moher 2007). Unpublished data from a subset of these reviews (129 Cochrane, 171 non‐Cochrane reviews) showed that 76.3% of reviews (92.2% Cochrane, 64.3% non‐Cochrane) reported checking reference lists as part of their search (Jennifer Tetzlaff (Cochrane Bias Methods Group, Ottawa, Canada) personal communication with Tanya Horsley (Centre for Learning in Practice) July 2008). Similarly, Royle et al examined all new Cochrane Reviews submitted to The Cochrane Library in Issue 1, 2001 and reported that of the most frequently searched sources, checking reference lists ranked second only to MEDLINE (53/66, 80.3%) (Royle 2003). Evidence‐based practice would weigh the sensitivity of such an approach against resource implications (i.e. what is the additional unique contribution gained and what is the cost to conduct) and the evidence regarding the potential to influence overall estimates or outcomes of the review.

This review investigates the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors specifically resulting from the method of checking reference lists.

Objectives

To systematically review evidence examining the practice of checking reference lists to identify potentially relevant studies for inclusion in systematic reviews. The following questions were investigated:

1. What is the additional yield of relevant studies identified through checking reference lists?

2. Is there evidence that the additional yield from checking reference lists is greater for particular publication types or study designs?

3. What is the evidence regarding marginal cost, cost‐effectiveness and cost‐efficiency associated with the methodology of checking reference lists?

Methods

Criteria for considering studies for this review

Types of studies

We considered all studies, regardless of design, to be eligible for inclusion if they examined checking reference lists as a search method (distinct from other search methods) for identifying potentially relevant studies for inclusion within a systematic review in any topic area (e.g. healthcare interventions, medical education, library and information science).

Types of data

We noted and summarized information pertaining to study characteristics, including author, year of publication, publication type, study design, inclusion/exclusion characteristics, search approaches used, search dates and limits, search strategies (e.g. peer reviewed, quality control indicators), and the protocol used for checking the references. We extracted outcome data pertaining to marginal yield (e.g. studies uniquely identified through checking reference lists) by study design (e.g. randomized controlled trial (RCT)) and publication type (e.g. abstract), when available. We summarized cost data when available.

Types of methods

Any methods which utilized checking reference lists of resources to identify relevant studies.

Types of outcome measures

Primary outcomes

1. The additional yield of checking reference lists for identifying potentially relevant studies for inclusion in systematic reviews (additional yield specifies that the record or resource was not found through any other search approach).

Secondary outcomes

1. The additional yield of checking reference lists by publication type, study design or both.

2. Any data pertaining to costs associated with checking reference lists (e.g. marginal/incremental cost, cost‐effectiveness, cost‐efficiency).

Search methods for identification of studies

We searched the following electronic databases to identify published and unpublished sources in any language: the databases of The Cochrane Library (Issue 3, 2008), Library and Information Science Abstracts (LISA) (1969 to July 2008) and MEDLINE, including MEDLINE In‐Process & Other Non‐Indexed Citations (1966 to July 2008). We reviewed major texts and technical reports advocating checking reference lists as part of a standard approach for systematic reviews for relevant studies. To ensure transparency and replication of this process a detailed list of all texts and technical reports is provided (Table 1). 

1. Sources for identifying additional studies.

Additional source searched	Date of searching/access
The Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2008)	16 July 2008
Centre for Reviews and Dissemination Handbook Centre for Reviews and Dissemination 2008	16 July 2008
The Handbook of Research and Synthesis Cooper 1994	16 July 2008
The HuGE Review Handbook, V.1.0 HuGENet 2008	18 July 2008

Search strategies

We validated and tested the following MEDLINE search strategy. We translated the search for use in other databases, adjusting the controlled vocabulary as applicable (Appendix 1; Appendix 2). We applied no limits with regard to language, study design or publication type.

MEDLINE (Ovid interface)

1. bibliography/ 
 2. ((check$ or review$ or handsearch$ or screen$ or scan$ or search$ or crosscheck$) adj (cited adj (work$ or reference$))).tw. 
 3. ((check$ or review$ or handsearch$ or screen$ or Scan$ or search$ or crosscheck$) adj (reference$ or bibliograph$)).tw. 
 4. (Identif$ adj2 (cited adj (work$ or reference$))).tw. 
 5. (Identif$ adj2 (reference$ or bibliograph$)).tw. 
 6. bibliograph$.tw. 
 7. (reference$ adj2 list$).mp. 
 8. (pearl grow$).mp. 
 9. ((work or works) adj2 cited).mp. 
 10. citation$.tw. 
 11. Information Systems/ 
 12. (snow ball$ or snowball$).mp. 
 13. footnote chas$.mp. 
 14. backward chain$.mp. 
 15. or/1‐14 
 16. 15 not "cochrane database of systematic reviews".jn. 
 17. Methods/ 
 18. Epidemiologic Methods/ 
 19. exp "Bias (Epidemiology)"/ 
 20. (recall or precision).tw. 
 21. or/17‐20 
 22. 16 and 21

Other sources

We checked the reference lists of all relevant studies. We also solicited experts in systematic review methods to identify additional unpublished or grey literature (names of the individuals we contacted are available upon request). We used the Science Citation Index and Social Sciences Citation Index in Web of Science to track relevant citations.

Data collection and analysis

Study selection

We examined the titles and abstracts obtained from searches of electronic databases and retrieved full‐text copies of potentially relevant articles. We conducted screening for relevance using an internet‐based, secured, software program designed specifically for systematic reviews. All screening (title and abstract, and full text) was conducted independently online by at least two (of any of the three) authors. We used a screening algorithm that allowed only one review author to move a study to the next level of assessment (full‐text) if it was identified as potentially relevant. The algorithm, however, required that two review authors assess the record in order for it to be excluded.

Authors were not blinded to study author or affiliation. Disagreements were resolved by discussion. We recorded studies excluded at this point or thereafter in a bibliographic database, noting the reason for exclusion, and summarized them in Figure 1 using a PRISMA flow diagram (Moher 2009).

1.

Study flow diagram.

Data extraction

All authors on the review team developed and tested a standardized data extraction form a priori. Data were extracted by one author and verified for accuracy by a second author. Any disagreements were noted and resolved by through discussion amongst all authors. We contacted study authors for missing data or when clarification was required (e.g. intervention not described in sufficient detail); we recorded all attempts.

Data analysis

We summarized all relevant data from non‐comparative studies descriptively, emphasizing the limitations of the study designs on the interpretation of findings. We noted information pertaining to the source of information, study design, relevant outcome data and any description of the actual technique used for checking reference lists. This was completed by one author and checked for consensus by one of the remaining authors.

Due to heterogeneity amongst studies and the type of data presented, we were unable to collate and present data using a 'Summary of findings' (SoF) table to provide key information concerning quality of evidence, the magnitude of effect and the sum of available data.

Ongoing studies

No ongoing studies were identified. However, when updating the review, if studies are identified that are incomplete or ongoing, we will record characteristics describing the primary author, study objectives, methodology and primary outcomes of interest, where available. We will also report a description of when the study began and the estimated time to completion.

Results

Description of studies

Following a process of de‐duplication (to remove overlapping records across databases) we identified 3989 records for initial screening by title and abstract. Two authors retrieved and assessed the full‐text reports of 155 records in duplicatefor relevance. We discarded 142 reports that did not examine the contribution of checking references lists or did not report a primary outcome. In total, 12 studies (Avenell 2001; Greenhalgh 2005; Kleijnen 1992; Lemeshow 2005; McManus 1998; McNally 2004; Murphy 2003; Ogilvie 2005; Rees 2003; Savoie 2003; Stevinson 2004; van Loo 1985) (from 13 reports) met all inclusion criteria and form the basis for our analysis (Figure 1). The report by Helmer et al (Helmer 2001) was identified as a companion publication for the study by Savoie et al (Savoie 2003). We contacted two authors for missing data within potentially relevant publications (Greenhalgh 2005; Hayward 1997), both of whom responded. Data were received from one study (Greenhalgh 2005) and not available from the other, which was not included in the review (Hayward 1997). A summary of data is included in Table 2.

2. Included studies summary data.

Study ID	Study design	N databases searched	Unique yield (trials only) yield/total eligible (%)	Sources (for checking reference lists)
McNally 2004	Case report	19	NA	References of papers identified through electronic databases and other methods
Murphy 2003	Case report	13	NA	References to selected studies
Savoie 2003	2 case reports	17 & 18	All 29/382 (7.6) Acupuncture 3/21 (14.3) Lipid 26/361 (7.2)	References to included studies
Avenell 2001	Case report	6	1/15 (6.7)	References from relevant reports, reviews, epidemiological and other trial reports
Rees 2003	Case report	11	NA	NR
Stevinson 2004	Case report	7	6/25 (24) (controlled trials)	References for review articles and already located studies
Kleijnen 1992	Retrospective observational study	2	MEDLINE homeopathy 28/107 (26.1), ascorbic acid 25/61 (40.9), ginkgo 20/45 (44.4) EMBASE homeopathy 30/107 (28), ascorbic acid 29/61 (47.5), ginkgo 16/45 (35.5)	References from review articles, included studies
Lemeshow 2005	Case report	9	NA	References from pertinent review papers and recent meta‐analyses
Ogilvie 2005	Case report	20	NA	References of all documents obtained
Greenhalgh 2005 (note these data are derived through direct correspondence with the primary author)	Case report	15	NA	References from all full‐text reports
van Loo 1985	Cross‐sectional study	15	NA	References from pertinent articles
McManus 1998	Case report	8	NA	References from identified publications

NA: not applicable 
 NR: not reported

Eleven of the 12 studies were published as full papers, the exception being one published only as an abstract (Rees 2003). Study designs included case reports (N = 9) (Avenell 2001; Greenhalgh 2005; Lemeshow 2005; McManus 1998; McNally 2004; Murphy 2003; Ogilvie 2005; Rees 2003; Stevinson 2004), retrospective observational study (N = 1) (Kleijnen 1992), cross‐sectional study (N = 1) (van Loo 1985), and one classified as ‘other’ (which included two case reports) (Savoie 2003). 

The majority of studies were published by authors residing in the United Kingdom (N = 8) (Avenell 2001; Greenhalgh 2005; McManus 1998; McNally 2004; Ogilvie 2005; Rees 2003; Stevinson 2004; van Loo 1985), followed by the United States (N = 2) (Lemeshow 2005; Murphy 2003), Canada (N = 1) (Savoie 2003) and the Netherlands (N = 1) (Kleijnen 1992). Most of these studies were published after 2002 (N = 8; range 1985 to 2005) (Greenhalgh 2005; Lemeshow 2005; McNally 2004; Murphy 2003; Ogilvie 2005; Rees 2003; Savoie 2003; Stevinson 2004)

Six included studies conducted searches prospectively (Avenell 2001; Greenhalgh 2005; McNally 2004; Murphy 2003; Savoie 2003; van Loo 1985), five were retrospective analyses (Kleijnen 1992; Lemeshow 2005; McManus 1998; Ogilvie 2005; Stevinson 2004) and for one study, published only in abstract form, we were unable to determine specific search databases due to limited reporting of information (Rees 2003).

The objectives for included studies were generally aimed at determining the comprehensiveness or yield of some type of search method and had the commonality of reporting the contribution of checking references lists as a component of the findings. The primary objectives were heterogeneous and included either identifying literature within a specific subject area (learning disabilities (McNally 2004), integrated education (Rees 2003), complementary and alternative medicine (Murphy 2003)), comprehensiveness of an electronic database (e.g. MEDLINE) (Kleijnen 1992; Stevinson 2004), comprehensiveness of a search for a systematic review (Avenell 2001; Greenhalgh 2005; Savoie 2003), searching for a specific study design (observational studies, (Lemeshow 2005)), or assessing the value of supplemental searches for locating studies (McManus 1998; Ogilvie 2005; van Loo 1985). The number of databases searched for each study was dissimilar, ranging between two and 18 databases (mean 11.7). A summary of all databases and all supplemental searches used is provided in tabular format (Characteristics of included studies).

The total number of supplemental search methods (e.g. those beyond electronic searching and including checking reference lists) ranged between one and nine methods. Half the studies reported supplementing electronic searches with either handsearching (Avenell 2001; Greenhalgh 2005; Kleijnen 1992; McManus 1998; Murphy 2003; Savoie 2003), contact with experts (Avenell 2001; Kleijnen 1992; McNally 2004; Murphy 2003; Ogilvie 2005; Savoie 2003; Stevinson 2004), contact with an organization or pharmaceutical company (Kleijnen 1992; Savoie 2003; van Loo 1985) and personal bibliographies (Ogilvie 2005; Stevinson 2004). A summary of all supplemental searches are provided within the Characteristics of included studies table. The denominator from which estimates for additional yield were taken (the number of reports identified from the original search strategies for the review) ranged between 69 and 7333.

The explicit method used for identifying potentially relevant studies beyond all other supplemental searches when checking reference lists was reported in only two studies (Avenell 2001; Savoie 2003).  Savoie et al stated that the total unique contribution was the proportion of randomized trials identified by extended search methods (versus the total contribution of databases and extended search methods) (Savoie 2003), and Avenell et al stated that once all the trials were identified, they searched electronic databases again to see whether the trials were indexed by the database but had been missed by the original search strategy (Avenell 2001). Except for three studies (Greenhalgh 2005; Kleijnen 1992; McNally 2004) that used a 'snowballing' technique, which is defined as following references from references for two or more iterations, all used a traditional approach to locating studies by examining (commonly referred to as 'screening') reference lists of literature.

The sources for checking reference lists (e.g. what sources were examined in attempts to identify new or additional studies for inclusion) were heterogeneous. Three studies reported using reference lists of relevant/included studies (Avenell 2001; McManus 1998; Murphy 2003), two reported using references for review articles and those already obtained (Kleijnen 1992; Stevinson 2004), three described reviewing references of all documents obtained (Greenhalgh 2005; McManus 1998; Ogilvie 2005), two described reviewing references from ''pertinent'' review papers/recent meta‐analyses (Lemeshow 2005) or articles (van Loo 1985), one used references of papers identified through electronic databases and other search methods (McNally 2004) as the source for identifying additional references, and one did not report the sources used to check reference lists (Rees 2003).

Only one study reported data on, or rather the impact of, search structure on the yield of additional studies. Lemeshow et al reported that the reviews and meta‐analyses identified through 'cross‐checking' reference lists were not solely identified through this method. In fact, when they examined the titles identified (that were seemingly new, additional studies) they were either originally in one or more of the databases but excluded because the title did not seem relevant initially, or they were in fact in the databases but 'missed' because of the choice of search terms or the miscoding of keywords (Lemeshow 2005).

Data for additional yield were reported for either trials (only) (Avenell 2001; Kleijnen 1992; Savoie 2003; Stevinson 2004) or for a combination of several study designs (all papers), or both (Greenhalgh 2005; Lemeshow 2005; McManus 1998; Murphy 2003; Ogilvie 2005; Rees 2003; Savoie 2003; van Loo 1985). We have summarized the data accordingly within Table 2.

Risk of bias in included studies

We did not identify any studies with a contemporaneous comparator (randomized controlled trials, quasi‐randomized controlled trials, interrupted time series or controlled before‐after studies) and were unable to estimate risk of bias formally. However, as the designs of these studies are primarily retrospective and the equivalent of case reports, findings should be interpreted within this context. Instead, we considered a critical evaluation of each study and the potential impact of design on findings. These were discussed by the authors and noted within the data extraction phase.

Although we could not assess the risk of bias within each study, we highlight key issues within the studies in this section. Of note were the lack of understanding that the contribution of checking reference lists would depend in part on the comprehensiveness of other search methods within each study; and lack of transparency in the reporting of search methods (no standard of reporting existed at the time of publication of many of these reports) with fewer than half the studies providing a transcript of the electronic search strategy as part of the report (Avenell 2001; Kleijnen 1992; Lemeshow 2005; Ogilvie 2005; Stevinson 2004). Thus, for several of the studies we cannot comment on whether or not the electronic searching was of any quality (and, therefore, this may be confounding). Further compounding the issue is the lack of detail or description provided by the original authors about the expertise or training of the individual preparing the search or executing the searching. Furthermore, most of the studies were retrospective in nature and thus not well‐controlled.  

There is also the difficulty of interpreting the data (e.g. generalizability) because of the heterogeneity of different databases used within the studies (both numbers of databases and type of database). In fact, some of the databases no longer exist and are of only historical value.

Overall, interpretability and generalizability of these studies is difficult. With extensive heterogeneity amongst studies in the approaches taken and the lack of control in most studies related to the quality and comprehensiveness of searching, it is difficult, if not impossible, to tease out the true effect modifiers within this cohort.

Effect of methods

1. What is the additional yield of relevant studies identified through checking reference lists?

We estimated and reported effects using percent yield rather than absolute values. The additional yield of relevant studies identified through checking reference lists ranged from (2/79, 2.5%) (Lemeshow 2005) to (111/260, 42.7%) (Greenhalgh 2005, updated by personal communication from Greenhalgh).

2. Is there evidence that the additional yield from checking reference lists is greater for particular publication types or study designs?

Of the 12 included studies, only two reported information regarding yield by publication type (Greenhalgh 2005; Murphy 2003). Murphy et al reported identifying nine additional studies from checking references cited within the 49 eligible studies. Of these, one was a dissertation (not identified through online searching of Digital Dissertations) (Murphy 2003). The study by Greenhalgh reported an additional yield (exclusively found through looking up references) by empirical research studies 80/213 (37.5%), non‐empirical studies 111/260 (42.7%), systematic reviews 1/22 (4.5%), and all papers 192/495 (38.7%). They reported that the greatest yield was from pursuing selected references from references (snowballing) (Greenhalgh 2005).

3. What is the evidence regarding marginal cost, cost‐effectiveness and/or cost‐efficiency associated with the method of checking reference lists?

No studies explicitly reported costs associated with the method of checking reference lists. One study did, however, report relevant studies pertaining to estimating costs (Greenhalgh 2005). The authors stated that it was impossible to isolate the time spent on reference tracking since this was done in parallel with the critical appraisal of each paper. The authors do note that electronic citation tracking of selected papers took around a day in total and uncovered many important recent resources, including five systematic reviews (three were not identified by any other method) and 12% of all empirical studies ‐ around one useful paper for every 15 minutes of searching.

Discussion

Our findings suggest there is some evidence to support the use of checking reference as a means for identifying additional evidence for systematic reviews. The majority of studies that have examined the checking of reference lists to supplement the location of relevant studies are focused on reporting data relevant to marginal yield. The majority of relevant studies describe the effects of checking reference lists as a small component of a larger study examining, and often comparing, other search methods (e.g. locating reports via contact with experts, identifying reports by chance, etc.). 

The body of evidence we did identify pertaining to marginal yield, however, provides preliminary insights into the utility of checking reference lists.

We limited our review to studies focusing on retrieval methods, rather than systematic reviews that used the technique and reported on yield, for several reasons. Aside from the enormity of the task, such case studies are highly susceptible to reporting bias. As with publication bias in randomized trials in which there is a great likelihood that positive findings are reported, we hypothesized that within these reports it is also more likely that the authors reported the yield of checking reference lists only if they found additional relevant material this way. Currently, there is no reporting requirement for the yield of individual search methods within systematic reviews, including checking reference lists (Sampson 2008a) and, thus, we would not likely have evidence for those reports which examined reference lists but found no new studies.

More robust methods are required to draw conclusions regarding the marginal yield of checking reference lists. If a systematic review finds no additional studies from checking reference lists, this might also be dependent on the quality of the electronic search strategy, and comprehensiveness of other search methods used to identify relevant studies. As an extreme example, Pagliaro et al reported that of 90 relevant articles, 25 were found only through manual searching (which included checking reference lists) (Pagliaro 2003). Although it might appear that these results provide strong evidence for the practice of checking reference lists, this is not necessarily the case. We note that the electronic search strategy was extremely limited in both scope and complexity. After revising the electronic search strategy, we were able to retrieve the entire set of 93 citations from MEDLINE. These data need to be produced through prospective design across a number of reviews and topic areas, using trained information specialists to guide the searching and data retrieval.

There may be instances in which the electronic search appears unable to identify relevant studies, yet we know that relevant work has been done. A recent report by Greenhalgh and Peacock represents such a case ‐ only 30% of sources were obtained from databases and handsearching, and the remaining were identified by 'snowballing' (defined as pursuing references from references, essentially another term for checking reference lists) (Greenhalgh 2005). Greenhalgh and Peacock concluded that systematic reviews of complex interventions cannot rely solely on electronic databases and handsearching journals. "Complex interventions are those that are built up from a number of components, which may act both independently and inter‐dependently...It is not easy precisely to define the “active ingredients” of a complex intervention" (Medical Research Council 2000). These interventions have been identified as challenging to evaluate due to difficulties developing, identifying, documenting and reproducing the intervention (May 2007) and these difficulties likely contribute to the difficulty of creating Boolean searches to identify them. Greenhalgh studied the diffusion of service‐level innovations in healthcare organizations (Greenhalgh 2005) and judged checking reference lists to be cost‐effective in that context, although they were unable to isolate the time spent on the task. This is not to say that cost estimates (e.g. time spent, dollars) can not be estimated. This would require authors to conduct a targeted study isolating checking reference lists as a search method and tracking cost of ordering, time spent checking, retrieving etc. To date no such study exists, but we intend to conduct such a study in the future.

When an area or question is difficult to search electronically (e.g. a new technology (Allison 1999), a complex intervention (Greenhalgh 2005), or a cross‐disciplinary topic (Kochtanek 1982)), when the review author decides to locate grey literature, or when an electronic search (tested with good quality control) has been negative (Stielow 1988), checking reference lists may be an effective adjunct method of information retrieval.

Identifying studies through checking reference lists that could have been found through more effective electronic searching has drawbacks. Important links to retractions, errata or comments that would be apparent from the electronic record may be missed. Continued citation of retracted papers has been demonstrated (Budd 1998; Pfeifer 1990) and it has been found that errata can contain information judged important enough to be worth obtaining (Budd 1999). Evidence of frequent errors in reference lists has been found in many journals and disciplines. Despite identification of this problem over two decades ago, citation errors continue to exist in large proportions within reference lists (Wager 2008). Inaccurate citations increase the amount of time needed for locating and retrieving potentially relevant studies.

Most systematic reviews use two or more methods to identify relevant studies. Using at least one method that is not dependent on skill in constructing search strategies may be a useful safeguard. Electronic citation searching (Pao 1989) and checking reference lists manually are examples of such independent methods. Verifying the indexing status of documents identified solely through methods other than the electronic search can partially validate the effectiveness of the electronic search. When the electronic search has missed relevant indexed articles, the review team should consider revising and rerunning the electronic search. Studying the indexing of any articles found to be indexed but not retrieved by the electronic search may identify additional terms for the electronic search strategy. Finding relevant articles nominated from references lists, but previously excluded by review authors, could in fact provide a quality check on the screening process, albeit one must factor in whether the titles or abstracts were informative.

Missing studies, if found, might make no difference to the review or, in a meta‐analysis, might change the point estimate for an effect or association in either direction. When the results of missing studies are congruent with those of the studies found, the point estimate may not alter but the confidence interval around the estimate may narrow. Missing studies might provide additional insights into harms, special populations or other aspects that might have clinical relevance. If the studies are subject to bias, finding and including them could potentially add noise to the review.

In a recent study (Beck 2010) review authors tracked eligible articles identified by checking references lists and determined which of those had also been found through database searches, but had been screened out by the review authors. Of the 134 eligible studies, 37 were identified from references lists. Eleven of those 37 had been retrieved by the electronic searches but had been excluded by the review authors as ineligible. The apparent reason for excluding the articles initially was that the titles and abstracts in the bibliographic record were uninformative (Beck ‐ personal correspondence). That is an important distinction with implications for the practice of reviewing. Checking reference lists is not sufficient to identify relevant material if the titles are uninformative. Rather, relevance becomes apparent by seeing the article cited in context in the body of the article, and the full article should be reviewed. A further implication would be that cited references may be more important for identifying evidence that pre‐dates the widespread adoption of structured abstracts (Harbourt 1995), or in fields in which fewer journals use structured abstracts.

Most of the 12 studies included in this review have weak study designs. None control for the effectiveness of the electronic search, for example. Like a complex intervention, the searches done within a systematic review are composed of a number of components and the unique contribution of any one search approach is dependent on the effectiveness of the other approaches. Disentangling the effects of any single component, such as checking reference lists, through observational studies is difficult. Prospective studies and simulation exercises (e.g. retrospective replication of searches and yields) are necessary to determine the true contribution of checking reference lists. Only then could we begin to estimate the contribution to systematic reviews and the cost‐effectiveness of the technique. 

Authors' conclusions

Implication for methodological research.

Prospective, well‐designed, protocol‐driven studies are required to study the contribution of checking reference lists for systematic reviews. Of particular interest is whether or not important differences in review findings occur when studies otherwise 'miss' relevant studies, as well as the reasons why studies are missed (e.g. quality of original search strategies or quality of screening). These were not investigated by any of the studies included in this review. It would be of interest to researchers to know whether particular types of studies or publications are routinely missed and the impact this may have on review findings.

The ideal study would prospectively determine the additional yield of studies by comparing two systematic reviews of the same subject‐matter conducted in parallel with one team checking reference lists and the other team relying solely on electronic searching. Investigators could furthermore then quantify any changes in effect estimates from the additionally identified studies. However, with strict budgets on human and other resources, this seems neither cost‐effective nor likely.

A modified approach would see authors conducting a systematic review and completing the meta‐analysis, then retrospectively checking reference lists to identify additional studies. Any new studies would then be 'added' to the meta‐analysis to determine the effect on estimates. This is a reasonable approach and one that is likely to be done by one or more of the investigators on the review.

From an information science perspective, it would seem reasonable to evaluate retrospectively a cohort of search strategies from published systematic reviews to determine initially the overall 'quality' of a given strategy (Sampson 2008b). The investigators could then obtain the full‐text report of all 'included studies' within the report and systematically review references lists to identify potentially relevant additional studies. Relevance of each record would be determined, in duplicate, against the review's inclusion and exclusion criteria as outlined within each corresponding review. Included studies could then be added to the original results (descriptively or through meta‐analysis) to determine if changes in the direction or magnitude of effect occur. Further, correlations for quality and productivity of checking reference lists could be explored.

History

Protocol first published: Issue 1, 2009
 Review first published: Issue 8, 2011

Date	Event	Description
3 August 2008	Amended	Converted to new review format.

Notes

None.

Appendices

Appendix 1. Cochrane Library search strategy

The Cochrane Library (Ovid Interface) Note: In the initial search, the prefix (CMR:) was used at the start of each of the first four tems in this search strategy. Before publication of the full review, the revised search (as shown below) confirmed that this had not led to the failure to identify any eligible studies.

1. "Other methodology ‐ information retrieval ‐ retrieval techniques".kw.

2. "Other methodology ‐ information retrieval ‐ comparisons of methods".kw.

3. "Study identification ‐ search strategies ‐ general".kw.

4. "Study identification ‐ search strategies ‐ trials".kw.

5. ((check$ or review$ or handsearch$ or screen$ or scan$ or search$ or crosscheck$) adj (cited adj (work$ or reference$))).tw.

6. ((check$ or review$ or handsearch$ or screen$ or scan$ or search$ or crosscheck$) adj (reference$ or bibliograph$)).tw.

7. (Identif$ adj2 (cited adj (work$ or reference$))).tw.

8. (Identif$ adj2 (reference$ or bibliograph$)).tw.

9. bibliograph$.tw.

10. (reference$ adj2 list$).mp.

11. (pearl grow$ or pearl‐grow$).mp.

12. ((work or works) adj2 cited).mp.

13. citation$.tw.

14. (snow ball$ or snowball$).mp.

15. footnote chas$.mp.

16. backward chain$.mp.

17. or/1‐16

Appendix 2. LISA Search Strategy

Library and Information Science Abstracts (LISA) (ProQuest Interface)

((pearl within 1 grow*) OR (reference within 1 list*) OR (snow within 1 ball*) OR (footnote within 1 chas*) OR (backward within 1 chain*)) OR ((citation* OR bibliography OR bibliographies OR reference*) AND DE=(search strategies OR searching OR cited references OR methodologies))

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Avenell 2001.

Methods	Publication type: journal article Search limitations: MEDLINE 1966 to January 2000, HealthSTAR 1975 to December 1999, CINAHL 1982 to November 1999, EMBASE 1980 to January 2000, BIOSIS 1985 to December 1999, CABNAR 1983 to December 1999 Subject area: nutrition (nutritional supplementation trials in patients with hip fractures)
Data	Supplemental searches: reference lists, contact with investigators/experts, handsearching journals Databases searched: BIOSIS, CABNAR, CINAHL, EMBASE, HealthSTAR and MEDLINE
Comparisons	Checking reference list method: reference lists examined in relevant reports, other relevant reviews, and epidemiological and other trial reports. Once all the trials were identified, electronic databases were searched again to see whether trials were indexed by the database but been missed by the search strategy
Outcomes	Total eligible: 7333 Total N (included studies): (trials only) 15 Additional yield: 1 Additional yield by publication type: study sample example trials only Cost data: NR
Notes	Database abbreviations summarized in Table 3 (Table 3)

3. Search database abbreviations.

AEI ‐ Australian Education International ASLIB Index ‐ Index to Theses ASSIA ‐ Applied Social Sciences Index and Abstracts BEI ‐ British Education Index BIDS ‐ Bath Information and Data Services BIOSIS ‐ BioSciences Information Service of Biological Abstracts CABNAR ‐ Commonwealth Agricultural Bureau Nutrition Abstracts and Reviews CCTR ‐ Cochrane Controlled Trials Register CDSR ‐ Cochrane Database of Systematic Reviews CINAHL ‐ Cumulative Index to Nursing and Allied Health Literature CRISP ‐ Computer Retrieval of Information on Scientific Projects              DHSS ‐ Department of Health and Social Security EMBASE ‐ Excerpta Medica Database EPOC ‐ Cochrane Effective Practice and Organization of Care database ERIC ‐ Education Resources Information Center ETOH ‐ Alcohol and Alcohol Problems Science Database GPLit ‐ Database of general practice literature HELMIS ‐ Health Management Information Service HMIC ‐ Health Management Information Consortium HSRProj ‐ Health Services Research Projects in Progress HSTAT ‐ Health Services/Technology Assessment Text HTA Database ‐ Health Technology Assessment Database IBSS ‐ International Bibliography of the Social Sciences ICL ‐ Index to Chiropractic Literature ISTP ‐ Index to Scientific & Technical Proceedings LILACS database ‐ Latin American and Caribbean Health Sciences Literature MANTIS ‐ Manual Alternative and Natural Therapy Index System MEDLINE ‐ Index Medicus NTIS Database ‐ National Technical Information Service Database OCLC ‐ Online Computer Library Center, Inc. PEDro ‐ Physiotherapy Evidence Database REGARD ‐ bibliographic database of ESRC (Economic and Social Research Council) SCI ‐ Science Citation Index SIGLE ‐ System on Grey Literature in Europe SSCI ‐ Social Science Citation Index

Greenhalgh 2005.

Methods	Publication type: journal article Search limitations: NR Subject area: service level innovations in healthcare organizations
Data	Supplemental searches: handsearch of 32 journals, snowballing/reference tracking, citation tracking (forward tracking), personal knowledge ‐ existing knowledge and resources, contacts and academic networks, chance Databases searched: 15 databases including MEDLINE, EPOC, ASSCI and ERIC
Comparisons	Checking reference list method: the reference lists of all full‐text papers were scanned and author's used judgment to decide whether to pursue these further Method for determining whether it was additional or not was not explicitly reported.
Outcomes	Total eligible: > 6000 Total N (included studies): 495 Additional yield: 192 (all papers) Additional yield by publication type: empirical research studies 80/213; non‐empirical studies 111/260; systematic reviews 1/22 Cost data: NR
Notes	Overall, the greatest yield was from pursuing selected references of references. These data are derived through personal correspondence with Greenhalgh directly (re‐calculated data). We were unable to identify the database ASSCI listed in the manuscript (thus have maintained the abbreviation). The full list of databases is also not available through the publication (extracted verbatim).

Kleijnen 1992.

Methods	Publication type: journal article Search limitations: year limit: 1966/1974 to 1991 Subject area: controlled clinical trials for: a) homeopathy, b) ascorbic acid for the prevention and treatment of common cold, c) ginkgo biloba for intermittent claudication and cerebral insufficiency
Data	Supplemental searches: 'Current Contents', checking reference lists, contacting researchers, contacting pharmaceutical companies, "telling people that they were looking for several studies" plus for a) conference proceedings, the contents of several journals, contacting specialized librarians, and for b) Index Medicus manual search, contacting specialized libraries Databases searched: MEDLINE, EMBASE
Comparisons	Checking reference list method: checking references extensively in (review) articles on clinical research and in textbooks. Method for determining if the additional yield was unique not explicitly reported.
Outcomes	Total eligible: NR Total N (included studies): a) homeopathy 107, b) ascorbic acid 61, c) ginkgo biloba 45 Additional yield: snowballing MEDLINE a) homeopathy 28, b) ascorbic acid 25, c) gingko 20; EMBASE a) homeopathy 30, b) ascorbic acid 29, c) gingko 16 Additional yield by publication type: NR Cost data: NR
Notes	These data are derived from the article and some data confirmation through correspondence with the primary author. It should be noted that the authors indicated that they did not systematically examine database overlap, so for example within the MEDLINE results the authors indicate that, for example, in the homeopathy data ''probably a handful may have come from EMBASE"

Lemeshow 2005.

Methods	Publication type: journal article Search limitations: a) date limit: 1950 to 9 September 2003, b) date limit: 1950 to 28 October 2003 Subject area: alcohol consumption and the risk of a) breast cancer and b) large bowel cancer
Data	Supplemental searches: checking reference lists Databases searched: BIOSIS, Dissertation Abstracts Online, EMBASE, ETOH, MEDLINE, NIH CRISP, NTIS, Pre‐MEDLINE, SCI EXPANDED‐SSCI
Comparisons	Checking reference list method: reviewed bibliographies of pertinent review papers and recent meta‐analyses, and an advisory committee expert reviewed publication references deemed potentially relevant. Method for determining whether it was additional or not was not explicitly reported.
Outcomes	Total eligible: a) 3607, b) 2128 Total N (included studies): all papers 135, a) breast cancer 79, b) bowel cancer 56 Additional yield: all papers 5, a) breast cancer 2, b) bowel cancer 3 Additional yield by publication type: NR Cost data: NR
Notes

McManus 1998.

Methods	Publication type: journal article Search limitations: 1986 to 1995 Prospective/retrospective: retrospective Study objectives: to examine the usefulness of contacting experts when searching for relevant references for a systematic review in the field where a specialist focus does not exist Subject area: studies relating to 'near patient testing' (any investigation performed in a clinical setting where the result is available without a sample being sent to a lab for analysis)
Data	Supplemental searches: survey of expert network, handsearching (abstracts, reports, references) Databases searched: BIDS Science Citation Index, MEDLINE, EMBASE, CINAHL, BIDS Index to conference proceedings, GPLit, DHSS, PsychLIT
Comparisons	Checking reference list method: limited to checking reference lists of bibliographies of publications identified. Method for determining whether it was additional or not was not explicitly reported.
Outcomes	Total eligible: 1057 Total N (included studies): 102 Additional yield: 20 Additional yield by publication type: NR Cost data: NR
Notes

McNally 2004.

Methods	Publication type: journal article Search limitations: NR Subject area: theory, barriers and improved access to health services by people with learning disabilities
Data	Supplemental searches: contacting researchers, searching personal bibliographies and using current awareness services Databases searched: MEDLINE, EMBASE, Cochrane Library, CINAHL, HMIC, SSCI, ASSIA, IBSS, CareData, AgeInfo, Sociological Abstracts, PsycINFO, BEI, ERIC, SIGLE, ASLIB Index, ISTP, online library catalogues (Royal College of Nursing, Royal National Institute for the Blind)
Comparisons	Checking reference list method: personal reference collection scanned for relevant records. (Note: not necessarily included studies); snowballing (checking citations of papers identified through other methods). Method for determining whether it was additional or not was not explicitly reported.
Outcomes	Total eligible: 2221 Total N (included studies): 2221 (note this was not a review and as such the total sample was what the yield was tested against) Additional yield: snowballing ‐ 51, personal reference collection searching ‐ 0 Additional yield by publication type: NR Cost data: NR
Notes	Snowballing is the primary technique identified for inclusion within this record. Snowballing was defined as 'checking the citations of papers identified through electronic databases and other methods'.

Murphy 2003.

Methods	Publication type: journal article Search limitations: limits for the search template included human studies, publication in all languages, and publication dates between 1966 and 2001 Subject area: Complementary and alternative medicine; spinal palpation
Data	Supplemental searches: gleaning references cited in selected studies, consulting experts in the field, contacting authors of eligible conference abstracts, and manually searching 3 specific journals Databases searched: PubMed, MANTIS, MD Consult, ISI Web of Science, EMBASE, CINAHL, BIOSIS Previews, ICL, Osteopathic Database, OCLC FirstSearch, Digital Dissertation, PEDro, CDSR
Comparisons	Checking reference list method: gleaning references cited in selected studies ('gleaning' as referred to by the authors was reviewing reference lists). Method for determining whether it was additional or not was not explicitly reported.
Outcomes	Total eligible: 1189 Total N (included studies): 49 Additional yield: 9 Additional yield by publication type: 1 dissertation Cost data: cost of additional searches: not quantified for the following reasons ‐ expert reviewers were thoroughly familiar with their collection of scientific publications, inclusion lists were sent to experts who spent unspecified time conducting additional searches, authors were contacted either by phone or email for correspondence
Notes

Ogilvie 2005.

Methods	Publication type: journal article Search limitations: NR Subject area: case study of the sources of studies for one systematic review
Data	Supplemental searches: internet, peer and expert consultation, checking reference lists, review of existing collection Databases searched: ASSIA, CINAHL, Cochrane Library (CDSR and CCTR), EMBASE, Geobase, HMIC, HELMIS, IBSS, ASLIB Index, MEDLINE and PreMEDLINE, PapersFirst, PsycINFO, REGARD, Sociological Abstracts, SPORTDiscus, Transport, Web of Science (SCI or SSCI), Dissertation Abstracts
Comparisons	Checking reference list method: searched the reference lists of all documents obtained. Method for determining whether it was additional or not was not explicitly reported
Outcomes	Total eligible: 69 Total N (included studies): 22 Additional yield: 2 Additional yield by publication type: NR Cost data: NR
Notes	The authors state that 20 databases were searched within the text and only list 19 within the table of databases

Rees 2003.

Methods	Publication type: abstract Search limitations: NR Subject area: the impact of integrated education and care on outcomes for children or their parents
Data	Supplemental searches: handsearching, citation tracking Databases searched: ERIC, BEI, AEI, ASSIA, IBSS, Sociological Abstracts, Social Services Abstracts, PsycINFO, C2‐Spectr, Childdata, Caredata (final 3 are described as specialized registers)
Comparisons	Checking reference list method: NR Method for determining whether it was additional or not was not explicitly reported
Outcomes	Total eligible: 3790 Total N (included studies): 58 Additional yield: 5 Additional yield by publication type: NR Cost data: NR
Notes

Savoie 2003.

Methods	Companion: Helmer 2001 Publication type: journal article Study objectives: to evaluate the sensitivity and precision of various extended search methods in identifying RCTs for SRs Subject area: a) acupuncture in treatment of addiction; b) lipid‐lowering therapy in prevention and treatment of coronary heart disease
Data	Supplemental searches: subject‐specific databases, library web catalogues, internet peer‐reviewed sites, internet search engines, in‐house databases, directories, handsearching journals, contacting relevant organizations and researchers, scanning reference lists of retrieved materials Databases searched: both a) and b): MEDLINE, EMBASE, HealthStar, Current Contents (considered major databases) a) Cochrane Library, HSTAT, HSRProj, Dissertation Abstracts, Article1st (OCLC), Papers1st (OCLC) ‐ conferences and paper abstracts, TRIP database, Ebsco Academic Search, Ebsco Canadian MAS, Alcohol and Alcohol Problems Science database (ETOH), ACUBASE, CRISP, ClinicalTrials.gov, National Research Register b) Cochrane Library, HSTAT, HSRProj, Dissertation Abstracts, Article1st (OCLC), Papers1st (OCLC) ‐ conferences and paper abstracts, TRIP database, NTIS Database, CRISP, HTA Database, LILACS, ClinicalTrials.gov, National Research Register
Comparisons	Checking reference list method: scanning reference lists of included studies. Savoie et al stated that the total unique contribution was the proportion of RCTs identified by extended search methods (versus the total contribution of databases and extended search methods). They stated that the total unique contribution (Precision (P) which reflected the proportion of all RCTs uncovered by any extended search method (p.170) was the proportion of RCTs identified by extended search methods (versus the total contribution of databases and extended search methods)
Outcomes	Total eligible: a) 212 (2003) and 204 (2001); b) 810 (2003) and 830 (2001) Total N (included studies): trials only ‐ all papers 382, a) acupuncture 21, b) lipid 361; all studies 1022, a) acupuncture 212, b) 810 Additional yield: trials only ‐ all papers 29, a) acupuncture 3, b) lipid 26; all studies ‐ all papers 98, a) acupuncture 36, b) lipid 62 Additional yield by publication type: NR Cost data: NR
Notes

Stevinson 2004.

Methods	Publication type: journal article Search limitations: year limit: database inception to July 2002 Subject area: systematic review of controlled trials of exercise interventions for cancer patients
Data	Supplemental searches: electronic search of British Library Journals, contact with 20 experts, existing literature files and reference lists of review articles and studies Databases searched: MEDLINE, EMBASE, Cochrane Library, Cancer Lit, Psychlit, SPORTDiscus, CINAHL
Comparisons	Checking reference list method: the reference lists of review articles on the subject and studies already located were checked for further potential trials. Method for determining whether it was additional or not was not explicitly reported.
Outcomes	Total eligible: (749 (electronic database) + 27 (non‐electronic database sources)) 776 Total N (included studies): 25 (controlled trials) Unique yield: 6 Unique yield by publication type: evaluation conducted in controlled trials only Cost data: NR
Notes

van Loo 1985.

Methods	Publication type: journal article Search limitations: material published in the UK between 1975 and 1983 Subject area: medical and psychological effects of unemployment
Data	Supplemental searches: identification and contact with relevant organizations, visits to special libraries, follow up of references cited in articles traced Databases searched: MEDLINE, Nursing bibliography, British National Library, Current Literature on Health Services, Social Service Abstracts, PsycINFO, British Reports, Translations and Theses, Popular Medical Index, Chadwyck Healey, British Humanities Index, Nursing Research Abstracts, Hospital Abstracts, ASLIB Index
Comparisons	Checking reference list method: references from pertinent articles identified which had not been located by using other search methodologies (e.g. contacting organizations). Method for determining whether it was additional or not was not explicitly reported although from the text it is assumed they cross‐referenced each yield.
Outcomes	Total eligible: NR Total N (included studies): 196 Additional yield: 31 Additional yield by publication type: NR Cost data: NR
Notes

NR: not reported 
 RCT: randomized controlled trial 
 SR: systematic review

See Table 3 for the search database abbreviations

Differences between protocol and review

None.

Contributions of authors

Tanya Horsley ‐ review supervision, developed review protocol and conceptual framework for study classification, data acquisition, extraction, analysis and interpretation of data, initial draft and critical revision of manuscript, and characteristics of studies tables.

Orvie Dingwall ‐ developed search strategies, document retrieval and management, data acquisition and extraction, initial draft searching and critical revision of protocol and manuscript.

Margaret Sampson ‐ study concept, peer reviewed search strategies, data acquisition, extraction and interpretation, initial draft discussion and critical revision of protocol and manuscript.

Sources of support

Internal sources

• No sources of support supplied

External sources

• Canadian Agency for Drugs and Technologies in Health, Canada.

• Canadian Institutes of Health Research, Canada.

Declarations of interest

None declared.

New