Handsearching versus electronic searching to identify reports of randomized trials

Abstract

Background

Systematic reviewers need to decide how best to reduce bias in identifying studies for their review. Even when journals are indexed in electronic databases, it can still be difficult to identify all relevant studies reported in these journals. Over 1700 journals have been or are being handsearched within The Cochrane Collaboration to identify reports of controlled trials in order to help address these problems.

Objectives

To review systematically empirical studies, which have compared the results of handsearching with the results of searching one or more electronic databases to identify reports of randomized trials.

Search methods

Studies were sought from The Cochrane Methodology Register (The Cochrane Library, Issue 2, 2002), MEDLINE (1966 to Week 1 July 2002), EMBASE (1980 to Week 25 2002), AMED (1985 to June 2002), BIOSIS (1985 to June 2002), CINAHL (1982 to June 2002), LISA (1969 to July 2002) and PsycINFO (1972 to May 2002). Researchers who may have carried out relevant studies were contacted.

Selection criteria

A research study was considered eligible for this review if it compared handsearching with searching one or more electronic databases to identify reports of randomized trials.

Data collection and analysis

The main outcome measure was the number of reports of randomized trials identified by handsearching as compared to electronic searching. Data were extracted on the electronic database searched, the complexity of electronic search strategy used, the characteristics of the journal reports identified, and the type of trial report identified.

Main results

Thirty‐four studies were included. Handsearching identified between 92% to 100% of the total number of reports of randomized trials found in the various comparisons in this review. Searching MEDLINE retrieved 55%, EMBASE 49% and PyscINFO 67%. The retrieval rate of the electronic database varied depending on the complexity of the search. The Cochrane Highly Sensitive Search Strategy (HSSS) identified 80% of the total number of reports of randomized trials found, searches categorised as 'complex' (including the Cochrane HSSS) found 65% and 'simple' found 42%. The retrieval rate for an electronic search was higher when the search was restricted to English language journals; 62% versus 39% for journals published in languages other than English. When the search was restricted to full reports of randomized trials, the retrieval rate for an electronic search improved: a complex search strategy (including the Cochrane HSSS) retrieved 82% of the total number of such reports of randomized trials.

Authors' conclusions

Handsearching still has a valuable role to play in identifying reports of randomized trials for inclusion in systematic reviews of health care interventions, particularly in identifying trials reported as abstracts, letters and those published in languages other than English, together with all reports published in journals not indexed in electronic databases. However, where time and resources are limited, searching an electronic database using a complex search (or the Cochrane HSSS) will identify the majority of trials published as full reports in English language journals, provided, of course, that the relevant journals have been indexed in the database.

Plain language summary

Handsearching versus electronic searching to identify reports of randomized trials

This review shows that handsearching alone will miss a small proportion of studies and, that a combination of handsearching and electronic searching is the most comprehensive approach in identifying reports of randomized trials.

Background

The validity of the results of a systematic review is highly dependent on the results of the underlying data and, as such, requires the identification of as unbiased and complete a set of relevant studies as possible. This can be a particularly time‐consuming part of the systematic review process and The Cochrane Collaboration has done a large amount of work to identify relevant studies (Lefebvre 2001; Dickersin 2002). Even though many reports of trials are included in bibliographic databases, such as MEDLINE and EMBASE, the majority of journals are not indexed in these databases. This is especially true for journals in languages other than English. For example, a study of 68 Spanish journals in general medicine found that only six were indexed in MEDLINE (Marti 1999).

Even when journals are indexed in databases such as MEDLINE, it can still be difficult to identify all relevant studies. This can be for a number of reasons, one of which is the lack of appropriate indexing terms. The MEDLINE Publication Type term for RANDOMIZED CONTROLLED TRIAL was only introduced in 1991; likewise the Publication Type term CONTROLLED CLINICAL TRIAL was not introduced until 1995. Some research has also identified an inconsistency of indexing by MEDLINE indexers (Lefebvre 1994). Lack of cover to cover indexing is also a potential problem, as some sections of a journal might not be included in MEDLINE. This may be a particular problem with supplements and conference abstracts (Langham 1999; Hopewell 2002a). Finally, authors may not have described their research methods clearly enough to allow accurate indexing of the methodology.

To address these problems, over 1700 journals have been or are being handsearched within The Cochrane Collaboration to identify reports of controlled trials, as of September 2002 (www.cochrane.org/Cochrane/hsearch.htm). Handsearching requires a trained person to check a journal from cover to cover, reading each item until they are satisfied as to whether or not it is an eligible report, irrespective of the health care intervention or condition under investigation. All full reports, short reports, editorials, correspondence sections, meeting abstracts and supplements are checked. The reports of identified trials form part of an international register of controlled trials published as The Cochrane Central Register of Controlled Trials (which is available in The Cochrane Library).

In addition, a highly sensitive search strategy (HSSS) for identifying reports of controlled trials in MEDLINE was developed in 1993 by one of the authors (CL) (Dickersin 1994). The first and second phases of this search strategy have been run against MEDLINE, the titles and abstracts have been assessed and those records judged to be definitely or possibly reports of randomized controlled trials or controlled clinical trials have been submitted to the US National Library of Medicine for re‐tagging with appropriate indexing terms (Dickersin 2002). To date (September 2002) this work has been carried out for the years 1966 to 2001. The re‐tagged records for 1966 to 2000 are now available in MEDLINE and records for 2001 should be available when MEDLINE is re‐issued in January / February 2003.

Those who carry out systematic reviews need to decide how best to identify as unbiased and complete a set of relevant studies as possible for inclusion in a review. This review aims to build on earlier work (Dickersin 1994) and systematically review research studies, which have compared the results of handsearching with the results of electronic searching in identifying reports of randomized trials.

Objectives

To review systematically empirical studies, which have compared the results of handsearching with the results of electronic searching in identifying reports of randomized trials.

To determine the additional yield of handsearching journals for reports of randomized trials as opposed to carrying out searches in electronic databases.

Methods

Criteria for considering studies for this review

Types of studies

A research study was considered eligible for inclusion in this review if it compared handsearching with searching one or more electronic databases, such as MEDLINE, EMBASE or PsycINFO, to identify reports of randomized trials.

Types of data

Information was collected on the number of reports of trials identified by each method (hand and electronic searching) and the overlap, the electronic databases searched (e.g. MEDLINE or EMBASE) and the type of electronic search strategy or strategies used (e.g. a simple or more complex search). In this review a simple search is defined as a search that uses a single type of search term (i.e. Publication Type terms only, MeSH headings only, free‐text terms only) and a complex search uses a combination of types of search terms.

Information on the date(s) of the searches, the characteristics of the journal reports identified (e.g. English or other languages, area of health care, year of publication and date of entry into MEDLINE or other databases), and the type of trial report identified (e.g. full article, abstract or letter) were also collected.

Types of methods

Handsearching of journals and electronic searching of bibliographic databases.

Types of outcome measures

The main outcome measure assessed was the number of reports of randomized trials identified by handsearching as compared to electronic searching and the overlap of retrieval between the two methods. The definition of a 'report of a randomized trial' was that which was used by the authors of the empirical studies. We did not independently assess the reports retrieved in the component studies. Where possible, information on the relative costs of different searches was recorded (East 1980).

Search methods for identification of studies

Studies were sought from The Cochrane Methodology Register (The Cochrane Library, Issue 2, 2002), MEDLINE (1966 to Week 1 July 2002), EMBASE (1980 to Week 25 2002), Allied and Complementary Medicine Database (AMED) (1985 to June 2002), Biological Abstracts (BIOSIS) (1985 to June 2002), Cumulative Index to Nursing and Allied Health Literature (CINAHL) (1982 to June 2002), Library and Information Science Abstracts (LISA) (1969 to July 2002) and Psychological Abstracts (PsycINFO) (1972 to May 2002). For the full search strategy, see appendices (Appendix 1; Appendix 2; Appendix 3; Appendix 4; Appendix 5; Appendix 6; Appendix 7).

Studies were also sought during the handsearching of selected journals, which is being carried out by the UK Cochrane Centre for all studies relevant to the methodology of systematic reviews. The abstracts presented at all Cochrane Colloquia (1993 to 2001), Systematic Reviews Symposia (1998 to 2002) and Society for Clinical Trials Meetings (1980 to 2001) (as published in Controlled Clinical Trials) have also been handsearched as part of this activity.

The titles and abstracts of records retrieved with these strategies were assessed for relevance to this review (see below, Identifying studies). Researchers who may have carried out relevant studies were also contacted. As of September 2002, searches have identified over 1100 citations (excluding any duplicate citations) and full papers were obtained for 99 records.

Only studies which compared the results of handsearching specific journals versus electronic searching of those same journals to identify reports of randomized trials were included in this review. No systematic attempts were made to identify studies that compared hand and electronic searching more generally as part of a broader study, for example in searching for reports of randomized trials for a specific systematic review.

Data collection and analysis

Identifying studies

Two reviewers (SH and CL) screened the titles and abstracts of all retrieved records to identify studies, which were potentially eligible for inclusion in the review. Any disagreements were resolved through discussion. Full copies of the reports were obtained for each of the non‐rejected records. These were then assessed independently by two reviewers (SH and MC) to determine if they met the inclusion criteria for the review. Any disagreements were resolved through discussion.

Assessment of methodological quality

The methodological quality of the included studies was assessed against the following criteria: were explicit criteria used to identify the reports of randomized trials; did two or more investigators agree regarding the identification of reports of randomized trials; was there completeness of data in the studies reviewed; did the studies control for other methodological differences, such as the independence of assessment of the handsearching compared to the electronic searching. This assessment was done by two reviewers (SH and RS) and any disagreements were resolved through discussion.

Data extraction

Data extraction was performed independently by two reviewers (SH and RS), any disagreements were resolved by a third reviewer (MC). Information was extracted on the number of reports of trials identified by each method, the electronic database searched (e.g. MEDLINE or EMBASE), the type of electronic search strategy used (e.g. a simple or more complex search), the date of the searches, the characteristics of the journal where reports of trials were identified (e.g. English or other languages, area of health care, year of publication and date of entry into MEDLINE), and the type of trial report identified (e.g. full article, abstract or letter). Information was also extracted on the methodological quality of the included studies. If any of the data for a study were insufficient or missing, attempts were made to contact the authors of the study. As of September 2002, some of these attempts are still ongoing.

Data analysis

The decision on whether or not to combine the results of the included studies was dependent on an assessment of heterogeneity. The included studies were first assessed for homogeneity of study design. Where the included studies were judged to be sufficiently homogenous in their study design, a meta‐analysis of these studies was carried out. This was done by comparing the retrieval rate of the handsearch with the retrieval rate of the electronic search. The difference in the retrieval rate between the two searches could then be compared. The retrieval rate of the handsearch was calculated by dividing the number of reports of trials identified by the handsearch by the total number of reports of trials identified. Similarly, the retrieval rate of the electronic search was calculated by dividing the number of reports of trials identified by the electronic search by the total number of reports of trials identified. Where heterogeneity in the study design was perceived (for example in the type of electronic database searched or the type of search strategy used) a subgroup analysis of these studies was carried out. This method of analysis is similar to that which is used to meta‐analyse data from screening and diagnostic tests (Deeks 2001). This type of analysis is not currently available in Review Manager so data were analysed using Microsoft Excel (Windows 2000). A copy of the analysis is available from the authors.

Subgroup analysis

Subgroup analyses were performed based on the following characteristics: 
 the type of electronic database searched (e.g. MEDLINE or EMBASE); 
 the type of electronic search strategy used (e.g. a simple or more complex search); 
 the characteristics of the journal reports identified (e.g. English or non‐English language); 
 the type of report identified (e.g. full article, abstract or letter).

Insufficient data were available to perform the following proposed subgroups:

• the characteristics of the journal reports identified (e.g. area of health care, year of publication and date of entry into MEDLINE);

• the date when the searches were done.

Results

Description of studies

As of September 2002, 34 studies have been identified which met the inclusion criteria for this review. Twenty‐seven of the included studies were published as full papers (Adams 1994; Bender 1997; Bernstein 1988; Croft 1999; Cullum 1997; Dickersin 1994; Duggan 1997; Gluud 1998; Hay 1996; Hedger 1999; Hopewell 2002a; Jadad 1993a; Jadad 1993b; Jadad 1996; Jefferson 1996; Kirpalani 1999; Langham 1999; Marson 1996; Marti 1999; Neal 1996; Nwosu 1998; Poynard 1985; Silagy 1993; Solomon 1994; Suarez‐Almazor 2000; Watson 1999a; Watson 1999b). One of these, which was published in Spanish, needed to be translated into English (Marti 1999). Of the remaining seven included studies, six were published as abstracts (Bara 1998; Fernandes 2000; Galandi 2000; Kennedy 2000; McDonald 1997; Reynolds 1997) and additional information for one of these was obtained from the authors (McDonald 1997). The remaining study was an unpublished report from a Cochrane Collaborative Review Group (Tavender 1999).

The majority of the included studies described how journals had been searched from 'cover‐to‐cover' to identify reports of randomized trials. This meant that each journal article, review article, letter and meeting abstract, included in the journal, had been checked. The number of people handsearching the journals varied across the studies. Two of the included studies (Bara 1998; Bender 1997) did not describe the method of handsearching. Seven studies reported some mechanism for assessing the quality of the handsearching. In four studies (Adams 1994; Hay 1996; Hopewell 2002a; Jadad 1996) a 10% sample of journal years were searched by a second person, in two studies (Croft 1999; Hedger 1999) the most fruitful journal years were searched twice, and in one study (Langham 1999) one year per journal was searched twice.

The method of identifying reports of randomized trials from the records retrieved by the electronic search(es) varied across the included studies. In some studies the titles, abstracts and keywords of reports found by the electronic search were read and full papers were obtained for those records thought to be reports of randomized trials. These reports were then compared to those reports found by handsearching. In some studies only the title and abstracts of reports found by the electronic search were read and then these were compared to those reports found by handsearching. However, in some studies the way in which the results of the hand and electronic search(es) were compared was unclear. The majority of the 34 included studies searched for reports of randomized trials that had been published in specialized health care journals. Seven studies searched for reports in general health care journals. More information is available in the Table of Included Studies.

Sixteen studies, which were initially thought to be suitable for inclusion in this review, have been excluded. The reasons for exclusion are given in the Table of Excluded Studies. One of the most common reasons was because the reports found by handsearching were then individually searched for in an electronic database such as MEDLINE. The electronic search was therefore not carried out independently of the handsearching and involved highly specific searching for individual reports, rather than for all reports of trials in the specific journals.

As of September 2002 nine studies, which may be suitable for inclusion in this review, are still awaiting assessment.

Risk of bias in included studies

We tried to assess the methodological quality of the included studies using the criteria highlighted above. In 17 of the 34 included studies, the handsearching of the journals was judged to have been carried out appropriately. This was unclear for the remaining 17 included studies (four were abstracts) because of the limited amount of information reported. In 29 of the 34 included studies, the electronic search(es) were judged to have been designed and carried out appropriately. This was unclear for the remaining five studies (four were abstracts) because of the limited amount of information reported.

In 11 of the 34 included studies, the eligibility of reports of randomized trials found by handsearching was agreed by two or more investigators. This was unclear for the other 23 studies (four were abstracts). In eight of the 34 studies, two or more investigators agreed on the eligibility of reports found by the electronic search(es), and this was unclear for 24 studies (five were abstracts). In 28 of the 34 included studies, the methods for judging eligibility of reports found by handsearching versus electronic searching were judged to be sufficiently comparable with each other to produce similar results. For six of the studies, further information is needed for clarification (two were abstracts).

Effect of methods

Information on the number of reports of randomized trials identified, by handsearching and electronic searching, is given in Table 1.

1. Number of reports of randomized trials.

Study ID	Database	Type of search	Found by HS	Found only by HS	Found by ES	Found only by ES	Found by HS & ES	Total found
Adams 1994	MEDLINE	Simple & Complex	698		Simple: 133 Complex: 388			743
Bara 1998	MEDLINE & EMBASE	Cochrane HSSS	1101	0	1186	85		1186
Bassler 2000
Bender 1997	MEDLINE	Simple & Complex	333	Simple: 157 Complex: 119	Simple: 176 Complex: 221	Simple: 0 Complex: 7	Simple: 176 Complex: 214	Simple: 333 Complex: 340
Berstein 1988	MEDLINE	Complex	194	100	95	1	94	195
Brand 1998
Croft 1999	MEDLINE	Cochrane HSSS	10	7	3	0	3	10
Cullum 1997	MEDLINE	Complex	134	44	90	0	90	134
Dickersin 1994	MEDLINE	Cochrane HSSS	61	7	54	0	54	61
Duggan 1997	MEDLINE & PsycLIT	Cochrane HSSS	37	MEDLINE: 13 PsycLIT: 11	MEDLINE: 24 PsycLIT: 26	0	MEDLINE: 24 PsycLIT: 26	37
Fernandes 2000	Lilacs & CCTR	Cochrane HSSS	9	8	1	0	1	9
Galandi 2000	MEDLINE & EMBASE	Simple	1690	MEDLINE: 1132 EMBASE: 980	MEDLINE: 558 EMBASE: 710	0	MEDLINE: 558 EMBASE: 710	1690
Gluud 1998	MEDLINE	Cochrane HSSS	169	32	139	2	137	171
Hay 1996	EMBASE & PsycLIT	Complex	40	EMBASE: 5 PsycLIT: 13	EMBASE: 35 PsycLIT: 27	0	EMBASE: 35 PsycLIT: 27	40
Hedger 1999	MEDLINE	Cochrane HSSS	5	3	2	0	2	5
Hopewell 2002	MEDLINE	Simple	682	369	345	32	313	714
Jadad 1993	MEDLINE	Complex	313	177	138	2	136	315
Jadad 1993 (1)	MEDLINE	Complex	202	23	179	0	179	202
Jadad 1996	MEDLINE	Complex	4336		2523			4702
Jefferson 1996	MEDLINE	Complex	231	171	60	0	60	231
Kennedy 2000	MEDLINE	Complex	172	17	155	0	155	172
Kirpalani 1989	MEDLINE	Simple	53	21	32	0	32	53
Langham 1999	MEDLINE	Cochrane HSSS	592	227	483	118	365	710
Marson 1996	MEDLINE	Simple & Complex	118	Simple: 42 Complex: 16	Simple: 76 Complex: 102	0	Simple: 76 Complex: 102	118
Marti 1999	MEDLINE	Cochrane HSSS	215	69	146	0	146	215
McDonald 1997	MEDLINE	Simple & Cochrane HSSS	358	Simple: 150 Cochrane HSSS: 65	Simple: 208 Cochrane HSSS: 293	0	Simple: 208 Cochrane HSSS: 293	358
Neal 1996	MEDLINE	Cochrane HSSS	143	54	89	0	89	143
Nwosu 1998	MEDLINE	Simple	389	111	292	14	278	403
Poynard 1985	MEDLINE	Complex	244	90	189	35	154	279
Reynolds 1997	MEDLINE	Cochrane HSSS	395		380			470
Silagy 1993	MEDLINE	Complex	198		135			204
Solomon 1994	MEDLINE	Complex	37	20	17	0	17	37
Suarez‐Almazor 2000	MEDLINE & EMBASE	Cochrane HSSS	259	MEDLINE: 71 EMBASE: 39	MEDLINE: 188 EMBASE: 220	0	MEDLINE: 188 EMBASE: 220	259
Tavender 1999	MEDLINE	Cochrane HSSS	522	176	346	0	346	522
Watson 1999	MEDLINE & PsycINFO	Complex	22		MEDLINE: 16 PsycINFO: 14			23
Watson 1999 (1)	MEDLINE, EMBASE & PsycINFO	Simple & Complex	34		MEDLINE (simple):31 MEDLINE (complex): 36 EMBASE (simple): 25 EMBASE: (complex): 28 PsycINFO (simple): 14 PsycINFO (complex): 24			37

Type of electronic database searched

(Table 2) 
 Thirty‐one studies (Adams 1994; Bender 1997; Bernstein 1988; Croft 1999; Cullum 1997; Dickersin 1994; Duggan 1997; Galandi 2000; Gluud 1998; Hedger 1999; Hopewell 2002a; Jadad 1993a; Jadad 1993b; Jadad 1996; Jefferson 1996; Kennedy 2000; Kirpalani 1999; Langham 1999; Marson 1996; Marti 1999; McDonald 1997; Neal 1996; Nwosu 1998; Poynard 1985; Reynolds 1997; Silagy 1993; Solomon 1994; Suarez‐Almazor 2000; Tavender 1999; Watson 1999a; Watson 1999b) compared handsearching with searching of MEDLINE using either a complex or a simple search strategy. The median number of reports of randomized trials identified per study was 209 (interquartile range (IQR) 75‐392); range 5 to 4702. The retrieval rate of the handsearching was 0.95 (95% CI 0.94‐0.95) and the retrieval rate of the MEDLINE search was 0.55 (95% CI 0.54‐0.56). Overall, the difference in the retrieval rates between the two searches was 0.40 (40%). One study (Bara 1998) was excluded because it compared handsearching to searching both MEDLINE and EMBASE, and the number of reports identified in each database was not available separately. In this study, handseaching retrieved 93% and electronic searching 100% of the total number of reports of randomized trials found.

2. Type of electronic database searched.

No. studies	Comparison	Type of search	Median no. trials	HS retrieval	ES retrieval	% Difference
31 studies	HS versus MEDLINE	Complex/HSSS or Simple	209 (IQR 75‐392)	0.95 (95% CI 0.94‐0.95)	0.55 (95% CI 0.54‐0.56)	40%
7 studies	HS versus databases other than MEDLINE	Complex/HSSS or Simple	37 (IQR 23‐259)	0.99 (95% CI 0.99‐1.00)	0.50 (95% CI 0.48‐0.52)	49%
4 studies	HS versus EMBASE	Complex/HSSS or Simple	149 (IQR 38‐1332)	0.99 (95% CI 0.99‐1.00)	0.49 (95% CI 0.47‐0.52)	51%
4 studies	HS versus PsycINFO	Complex/HSSS or Simple	37 (IQR 26‐39)	0.99 (95% CI 0.99‐1.00)	0.67 (95% CI 0.58‐0.76)	32%
1 study	HS versus Lilacs & CENTRAL	Complex/HSSS or Simple	9	1.00	0.11 (95% CI 0.01‐0.15)	89%

Seven studies (Duggan 1997; Fernandes 2000; Galandi 2000; Hay 1996; Suarez‐Almazor 2000; Watson 1999a; Watson 1999b) compared handsearching with searching of electronic databases, other than MEDLINE, using either a complex or simple search strategy. These databases were EMBASE, PsycINFO/PsycLIT, LILACS and the Cochrane Central Register of Controlled Trials (CENTRAL). The median number of reports of randomized trials identified per study was 37 (IQR 23‐259); range 9 to 1690. The retrieval rate of the handsearching was 0.99 (95% CI 0.99‐1.00) and the retrieval rate for the electronic databases (excluding MEDLINE) was 0.50 (95% CI 0.48‐0.52). Overall, the difference in the retrieval rate between the two searches was 0.49 (49%).

Four of the above seven studies (Galandi 2000; Hay 1996; Suarez‐Almazor 2000; Watson 1999b) compared handsearching with searching of EMBASE using either a complex or simple search strategy. The median number of reports of randomized trials identified per study was 149 (IQR 38‐1332); range 37 to 1690. The retrieval rate of the handsearching was 0.99 (95% CI 0.99‐1.00) and the retrieval rate of the EMBASE search was 0.49 (95% CI 0.47‐0.52). Overall, the difference in the retrieval rate between the two searches was 0.51 (51%).

Four studies (Duggan 1997; Hay 1996; Watson 1999a; Watson 1999b) compared handsearching with searching PsycINFO/PsycLIT using either a complex or simple search strategy. The median number of reports of randomized trials identified per study was 37 (IQR 26‐39); range 23 to 40. The retrieval rate of the handsearching was 0.99 (95% CI 0.97‐1.00) and the retrieval rate of the PsycINFO/PsycLIT search was 0.67 (95% CI 0.58‐0.76). Overall, the difference in the retrieval rate between the two searches was 0.32 (32%).

One study (Fernandes 2000) compared handsearching with searching LILACS and CENTRAL. Nine reports of randomized trials were found by handsearching and only one of these was found by the electronic search.

Type of electronic search strategy used

(Table 3) 
 Searches were categorized as either a simple search, which is defined as using a single type of search term (i.e. Publication Type terms only, MeSH headings only, free‐text terms only); a complex search, which is defined as using a combination of types of search terms; or the Cochrane Highly Sensitive Search Strategy (HSSS), which was designed to identify reports of randomized trials in MEDLINE with a relatively high degree of sensitivity (Dickersin 1994).

3. Type of electronic search strategy used.

No. studies	Comparison	Type of search	Median no. trials	HS retrieval	ES retrieval	% Difference
9 studies	HS versus ES	Simple	385 (IQR 86‐729)	0.98 (95% CI 0.97‐0.98)	0.42 (95% CI 0.41‐0.43)	56%
30 studies	HS versus ES	Complex or Cochrane HSSS	198 (IQR 39‐344)	0.94 (95% CI 0.93‐0.95)	0.65 (95% CI 0.64‐0.66)	29%
14 studies	HS versus ES	Cochrane HSSS	193 (IQR 30‐483)	0.93 (95% CI 0.93‐0.94)	0.80 (95% CI 0.79‐0.82)	13%
12 studies	HS versus MEDLINE	Cochrane HSSS	193 (IQR 43‐442)	0.93 (95% CI 0.93‐0.94)	0.73 (95% CI 0.71‐0.74)	21%

Nine studies (Adams 1994; Bender 1997; Galandi 2000; Hopewell 2002a; Kirpalani 1999; Marson 1996; McDonald 1997; Nwosu 1998; Watson 1999b) compared handsearching with searching an electronic database using a simple search strategy. The median number of reports of randomized trials identified per study was 358 (IQR 86‐729); range 37 to 1690. The retrieval rate of the handsearching was 0.98 (95% CI 0.97‐0.98) and the retrieval rate for the simple search was 0.42 (95% CI 0.41‐0.43). Overall, the difference in the retrieval rate between the two searches was 0.56 (56%).

Thirty studies (Adams 1994; Bara 1998; Bender 1997; Bernstein 1988; Croft 1999; Cullum 1997; Dickersin 1994; Duggan 1997; Fernandes 2000; Gluud 1998; Hay 1996; Hedger 1999; Jadad 1993a; Jadad 1993b; Jadad 1996; Jefferson 1996; Kennedy 2000; Langham 1999; Marson 1996; Marti 1999; McDonald 1997; Neal 1996; Poynard 1985; Reynolds 1997; Silagy 1993; Solomon 1994; Suarez‐Almazor 2000; Tavender 1999; Watson 1999a; Watson 1999b) compared handsearching with searching an electronic database using either a complex search strategy or the Cochrane HSSS. The median number of reports of randomized trials identified per study was 198 (IQR 39‐344); range 5 to 4702. The retrieval rate of the handsearching was 0.94 (95% CI 0.93‐0.94) and the retrieval rate for the complex search (including the Cochrane HSSS) was 0.65 (95% CI 0.64‐0.66). Overall, the difference in the retrieval rate between the two searches was 0.29 (29%).

Fourteen studies (Bara 1998; Croft 1999; Dickersin 1994; Duggan 1997; Fernandes 2000; Gluud 1998; Hedger 1999; Langham 1999; Marti 1999; McDonald 1997; Neal 1996; Reynolds 1997; Suarez‐Almazor 2000; Tavender 1999) compared handsearching with searching an electronic database using the Cochrane HSSS, or a version of the Cochrane HSSS modified for searching databases other than MEDLINE. The databases searched were MEDLINE, EMBASE, PyscINFO/PsycLIT and LILACS. The median number of reports of randomized trials identified per study was 193 (IQR 30‐483); range 5 to 1186. The retrieval rate of the handsearching was 0.93 (95% CI 0.93‐0.94) and the retrieval rate for the Cochrane HSSS was 0.80 (95% CI 0.79‐0.82). Overall, the difference in the retrieval rate between the two searches was 0.13 (13%).

The Cochrane HSSS was specifically designed to identify reports of randomized trials in MEDLINE, not EMBASE, PsycINFO or LILACS. Therefore, we have analysed separately those studies which compared handsearching with searching MEDLINE. Twelve studies (Croft 1999; Dickersin 1994; Duggan 1997; Gluud 1998; Hedger 1999; Langham 1999; Marti 1999; McDonald 1997; Neal 1996; Reynolds 1997; Suarez‐Almazor 2000; Tavender 1999) compared handsearching with searching MEDLINE using the Cochrane HSSS. The median number of reports of randomized trials identified per study was 193 (IQR 43‐442); range 5 to 710. The retrieval rate of the handsearching was 0.93 (95% CI 0.93‐0.94) and the retrieval rate for the MEDLINE search using the Cochrane HSSS was 0.73 (95% CI 0.71‐0.74). Overall, the difference in the retrieval rate between the two searches was 0.21 (21%).

Language of publication of the journals searched

(Table 4) 
 In twenty‐nine of the studies (Adams 1994; Bara 1998; Bender 1997; Bernstein 1988; Croft 1999; Cullum 1997; Duggan 1997; Gluud 1998; Hay 1996; Hedger 1999; Hopewell 2002a; Jadad 1993a; Jadad 1993b; Jadad 1996; Jefferson 1996; Kennedy 2000; Kirpalani 1999; Langham 1999; Marson 1996; McDonald 1997; Neal 1996; Nwosu 1998; Reynolds 1997; Silagy 1993; Solomon 1994; Suarez‐Almazor 2000; Tavender 1999; Watson 1999a; Watson 1999b) all of the journals searched were published in English. The median number of reports of randomized trials identified per study was 203 (IQR 50‐420); range 5 to 4702. The retrieval rate of the handsearching was 0.94 (95% CI 0.94‐0.95) and the retrieval rate for the electronic search was 0.62 (95% CI 0.61‐0.62). Overall, the difference in the retrieval rate between the two searches was 0.33 (33%).

4. Language of publication of the journals searched.

No. studies	Comparison	Type of search	Language	Median no. trials	HS retrieval	ES retrieval	% Difference
29 studies	HS versus ES	Complex/HSSS or Simple	English only	203 (IQR 50‐420)	0.94% (95% CI 0.94‐0.95)	0.62 (95% CI 0.61‐0.62)	33%
5 studies	HS versus ES	Complex/HSSS or Simple	English & other languages	215 (IQR 35‐984)	0.98 (95% CI 0.98‐0.99)	0.42 (95% CI 0.40‐0.44)	56%
3 studies	HS versus ES	Complex/HSSS or Simple	Languages other than English	215 (IQR 9‐1690)	1.00	0.39 (95% CI 0.38‐0.41)	61%

Five studies (Dickersin 1994; Fernandes 2000; Galandi 2000; Marti 1999; Poynard 1985) included a mixture of journals published in English and other languages. The median number of reports of randomized trials identified per study was 215 (IQR 35‐984); range 9 to 1690. The retrieval rate of the handsearching was 0.98 (95% CI 0.98‐0.99) and the retrieval rate for the electronic search was 0.42 (95% CI 0.40‐0.44). Overall, the difference in the retrieval rate between the two searches was 0.56 (56%).

Three studies (Fernandes 2000; Galandi 2000; Marti 1999) only included journals published in languages other than English. The median number of reports of randomized trials identified per study was 215; range 9 to 1690. The retrieval rate of the handsearching was 1.00 (95% CI 1.00‐1.00) and the retrieval rate for the electronic search was 0.39 (95% CI 0.38‐0.41). Overall, the difference in the retrieval rate between the two searches was 0.61 (61%)

Type of report identified

(Table 5) 
 Studies which compared handsearching with electronic searching to identify only full reports of randomized trials (i.e. those which excluded conference abstracts, letters, editorials and journal supplements) were analysed separately. Information on the number of full reports of randomized trials identified, by handsearching and electronic searching, is given in Table 6.

5. Type of report identified.

No. studies	Comparison	Type of search	Type of report	Median no. trials	HS retrieval	ES retrieval	% Difference
14 studies	HS versus ES	Complex/HSSS or Simple	Full	195 (IQR 23‐403)	0.93 (95% CI 0.92‐0.94)	0.77 (95% CI 0.76‐0.79)	16%
5 studies	HS versus ES	Simple	Full	333 (IQR 77‐442)	0.96 (95% CI 0.95‐0.97)	0.68 (95% CI 0.66‐0.68)	28%
12 studies	HS versus ES	Complex /HSSS	Full	131 (IQR 12‐324)	0.92 (95% CI 0.91‐0.93)	0.82 (95% CI 0.80‐0.83)	10%

6. Number of FULL reports of randomized trials.

Study ID	Database	Type of search	Found by HS	Found only by HS	Found by ES	Found only by ES	Found by HS & ES	Total found
Bara 1988	MEDLINE & EMBASE	Cochrane HSSS	1101	0	1186	85		1186
Bender 1997	MEDLINE	Simple & Complex	333	Simple: 157 Complex: 119	Simple: 176 Complex: 221	Simple: 0 Complex: 7	Simple: 176 Complex: 214	Simple: 333 Complex: 340
Berstein 1988	MEDLINE	Complex	194	100	95	1	94	195
Croft 1999	MEDLINE	Cochrane HSSS	8	5	3	0	3	8
Fernandes 2000	Lilacs & CCTR	Cochrane HSSS	9	8	1	0	1	9
Hedger 1999	MEDLINE	Cochrane HSSS	5	3	2	0	2	5
Hopewell 2002	MEDLINE	Simple	450	81	345	32	313	482
Jadad 1993	MEDLINE	Complex	142		125			144
Langham 1999	MEDLINE	Cochrane HSSS	592	227	483	118	365	710
Marson 1996	MEDLINE	Simple & Complex	118	Simple: 42 Complex: 16	Simple: 76 Complex: 102	0	Simple: 76 Complex: 102	118
Nwosu 1998	MEDLINE	Simple	389	111	292	14	278	403
Poynard 1985	MEDLINE	Complex	244	90	189	35	154	279
Watson 1999	MEDLINE & PsycINFO	Complex	22		MEDLINE: 16 PsycINFO: 14			23
Watson 1999 (1)	MEDLINE, EMBASE & PsycINFO	Simple & Complex	34		MEDLINE (simple):31 MEDLINE (complex): 36 EMBASE (simple): 25 EMBASE: (complex): 28 PsycINFO (simple): 14 PsycINFO (complex): 24			37

Fourteen studies (Bara 1998; Bender 1997; Bernstein 1988; Croft 1999; Fernandes 2000; Hedger 1999; Hopewell 2002a; Jadad 1993a; Langham 1999; Marson 1996; Nwosu 1998; Poynard 1985; Watson 1999a; Watson 1999b) compared handsearching with searching an electronic database for full reports of randomized trials. The median number of reports of randomized trials identified per study was 195 (IQR 23‐403); range 5 to 1186. The retrieval rate of the handsearching was 0.93 (95% CI 0.92‐0.94) and the retrieval rate for the electronic search was 0.77 (95% CI 0.76‐0.79). Overall the difference in the retrieval rate between the two searches was 0.16 (16%).

Five studies (Bender 1997; Hopewell 2002a; Marson 1996; Nwosu 1998; Watson 1999b) compared handsearching versus searching an electronic database, using a simple search strategy, for full reports of randomized trials. The median number of reports of randomized trials identified per study was 333 (IQR 77‐442); range 37 to 482. The retrieval rate of the handsearching was 0.96 (95% CI 0.95‐0.97) and the retrieval rate for the electronic search, using a simple search strategy was 0.68 (95% CI 0.66‐0.70). Overall, the difference in the retrieval rate between the two searches was 0.28 (28%).

Twelve studies (Bara 1998; Bender 1997; Bernstein 1988; Croft 1999; Fernandes 2000; Hedger 1999; Jadad 1993a; Langham 1999; Marson 1996; Poynard 1985; Watson 1999a; Watson 1999b) compared handsearching with searching an electronic database, using a complex search strategy (including the Cochrane HSSS), for full reports of randomized trials. The median number of reports of randomized trials identified per study was 131 (IQR 12‐324); range 5 to 1186. The retrieval rate of the handsearching was 0.92 (95% CI 0.91‐0.93) and the retrieval rate for the electronic search, using a complex search strategy was 0.82 (95% CI 0.80‐0.83). Overall, the difference in the retrieval rate between the two searches was 0.10 (10%).

Time and costs involved

Only four of the 34 included studies reported the time taken to perform the searches. In the Adams 1994 study, handsearching took 200 hours (mean time per report identified 17.20 minutes) and the electronic search took eight hours (mean time per report identified 1.24 minutes). In the Croft 1999 study, handsearching took 34 hours (mean time per report 3.4 hours), the time taken for the electronic search was not given. In the Hedger 1999 study, the total time taken to carry out both the hand and electronic search was 37 hours (mean time per report 7.4 hours). Finally, in the Jadad 1996 study, handsearching took 623 hours (mean time per report 8.62 minutes) and the time taken for the electronic search was not given. No studies provided information on the cost of handsearching or electronic searching.

Discussion

This review shows that handsearching identifies a greater number of reports of randomized trials in comparison to electronic searching. When compared with simple electronic searching, the additional yield of handsearching is, of course, greater than when compared with complex electronic searching. Across the subgroups analysed, handsearching identified between 92% to 100% of the total number of reports of randomized trials found. This suggests that while handsearching is very effective, it might still miss up to 8% of the total number of reports of randomized trials. This proportion will obviously vary depending on the quality of the handsearching.

In this review, MEDLINE was the most common electronic database searched (31 out of 34 included studies). Searching MEDLINE retrieved 55% of the total number of reports of randomized trials found in the relevant studies. The retrieval rate for EMBASE was slightly lower (49%) than that for searching MEDLINE, and was slightly higher for PyscINFO/PsycLIT (67%), however, the number of trials identified in these studies was relatively small. One explanation for the difference in retrieval rates between the electronic searches might be related to what was being searched. For example searches of PsycINFO, which had the highest retrieval rate, were for a small number of very specific journals. In contrast, studies comparing handsearching to searching MEDLINE included a greater number and more general journals.

The retrieval rate of the electronic search varied greatly depending on the type of electronic search used. For example, searching electronic databases using either the Cochrane HSSS designed for MEDLINE or an adaptation of this, identified 80% of the total number of reports of randomized trials found. Here the difference in retrieval between hand and electronic searching was only 13%. Searching an electronic database using any type of complex search (including the Cochrane HSSS) identified 65% of the total number of reports of randomized trials found and, as might be expected, the retrieval rate using a simple search was less (42%). The retrieval rate of the electronic search was higher in studies restricted to English language journals. However, if journals published in languages other than English are omitted from a search, bias may be introduced into a review as evidence suggests these trials may have different results to those published in English language journals (Juni 2002).

There were a number of reasons why the electronic searches failed to identify reports of the randomized trials. Three studies (Hopewell 2002a; Jadad 1996; Suarez‐Almazor 2000) found that an important reason was a lack of the relevant MEDLINE indexing terms. For example, the Publication Type term RANDOMIZED CONTROLLED TRIAL was only introduced into MEDLINE in 1991 and so could not be applied by indexers prior to that date. These three studies showed that the sensitivity and precision of searching MEDLINE for reports of randomized trials had improved over time, especially in recent years. Two studies (Jadad 1996; Kirpalani 1999) also reported that there was inconsistency by MEDLINE indexers and that trials had not been adequately indexed with the appropriate indexing terms. In addition, seven studies (Adams 1994; Gluud 1998; Hopewell 2002a; Jadad 1996; McDonald 1997; Neal 1996; Suarez‐Almazor 2000) found that a major reason for failure to identify reports of randomized trials by electronic searching was because the reports were published as abstracts and/or included in supplements, which are not routinely indexed by electronic databases such as MEDLINE.

When studies that only assessed full reports of randomized trials (i.e. those which excluded reports of conference abstracts, letters, editorials and journal supplements) were analysed separately, the retrieval rate for the electronic search improved considerably. Searching an electronic database using a complex search strategy (including the Cochrane HSSS) produced a retrieval rate as high as 82% with only a 10% difference in the retrieval rate between handsearching and electronic searching.

Authors' conclusions

Implication for methodological research.

In this review, no attempts were made to assess the importance of the trials missed by either handsearching or electronic searching. There is evidence to suggest that only around half of all trials reported as abstracts are subsequently published in full (Scherer 2002) and that published trials may show a larger treatment effect than 'grey' trials (for example those published as conference abstracts) (Hopewell 2002b). Further research is needed to assess the importance of those trials missed by either method of searching.

Research is also needed to determine the relative importance and number of reports of randomized trials found in journals not indexed by electronic databases such as MEDLINE.

What's new

Date	Event	Description
27 December 2007	Amended	Converted to new review format.

History

Protocol first published: Issue 2, 2001
 Review first published: Issue 2, 2003

Date	Event	Description
20 February 2007	New citation required and conclusions have changed	Substantive amendment

Appendices

Appendix 1. Cochrane Methodology Register search strategy

The Cochrane Methodology Register was searched (The Cochrane Library, Issue 2, 2002) using the following indexing terms: 
 #1 IDENTIFICATION 
 #2 (SEARCH:KY next STRATEGIES:KY) 
 #3 GENERAL:KY 
 #4 ((#1:KY and #2:KY) and #3:KY) 
 #5 TRIALS:KY 
 #6 ((#1:KY and #2:KY) and #5:KY) 
 #7 (INFORMATION:KY next RETRIEVAL:KY) 
 #8 (#7:KY and #3:KY) 
 #9 (RETRIEVAL:KY next TECHNIQUES:KY) 
 #10 (#7 and #9) 
 #11 (COMPARISONS:KY next METHODS:KY) 
 #12 (#7:KY and #11:KY) 
 #13 ((((#4:KY or #6:KY) or #8:KY) or #10:KY) or #12:KY) 
 #14 HANDSEARCH* 
 #15 (HAND near SEARCH*) 
 #16 HAND‐SEARCH* 
 #17 (MANUAL* near SEARCH*) 
 #18 (JOURNAL* near SEARCH*) 
 #19 (FULLTEXT near SEARCH*) 
 #20 ((FULL near TEXT) near SEARCH*) 
 #21 (FULL‐TEXT near SEARCH*) 
 #22 SENSITIVITY 
 #23 DATABASES 
 #24 COMPUTER* 
 #25 ONLINE 
 #26 MEDLINE 
 #27 PUBMED 
 #28 MEDLARS 
 #29 EMBASE 
 #30 (EXCERPTA next MEDICA) 
 #31 PSYCLIT 
 #32 PSYCINFO 
 #33 PSYCHLIT 
 #34 PSYCHINFO 
 #35 ((COCHRANE next CONTROLLED) next TRIALS) 
 #36 CCTR 
 #37 CENTRAL 
 #38 AMED 
 #39 ((ALLIED next COMPLEMENTARY) next MEDICINE) 
 #40 BIOSIS 
 #41 (BIOLOGICAL next ABSTRACTS) 
 #42 (CAB next HEALTH) 
 #43 CINAHL 
 #44 ((CUMULATIVE next INDEX) next NURSING) 
 #45 LILACS 
 #46 ((SCIENCE next CITATION) next INDEX) 
 #47 SCISEARCH 
 #48 (CURRENT next CONTENTS) 
 #49 (WEB next SCIENCE) 
 #50 TRIALS 
 #51 RCTS 
 #52 ((((((((#14 or #15) or #16) or #17) or #18) or #19) or #20) or #21) or #22) 
 #53 (((((((#23 or #24) or #25) or #26) or #27) or #28) or #29) or #30) 
 #54 (((((((((#31 or #32) or #33) or #34) or #35) or #36) or #37) or #38) or #39) or #40) 
 #55 ((((((((#41 or #42) or #43) or #44) or #45) or #46) or #47) or #48) or #49) 
 #56 ((#53 or #54) or #55) 
 #57 (#50 or #51) 
 #58 (#13 or ((#52 and #56) and #57))

Appendix 2. MEDLINE and EMBASE search strategy

MEDLINE (1966 to Week 1 July 2002) and EMBASE (1980 to Week 25 2002) were searched using the following search strategy (the two databases were searched concurrently using the Ovid Web interface for ease of de‐duplication): 
 1 medline/ 
 2 "information storage and retrieval"/ 
 3 medlars/ 
 4 review literature/ 
 5 "abstracting and indexing"/ 
 6 information retrieval/ 
 7 1 or 2 or 3 or 4 or 5 or 6 
 8 handsearch$.mp. 
 9 (hand adj5 search$).mp. 
 10 hand‐search$.mp. 
 11 (manual$ adj5 search$).mp. 
 12 (journal$ adj5 search$).mp. 
 13 (fulltext adj5 search$).mp. 
 14 (full adj5 text adj5 search$).mp. 
 15 (full‐text adj5 search$).mp. 
 16 sensitivity.ti,ab. 
 17 8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 or 16 
 18 medline.mp. 
 19 pubmed.mp. 
 20 medlars.mp. 
 21 embase.mp. 
 22 (excerpta adj5 medica).mp. 
 23 psyclit.mp. 
 24 psycinfo.mp. 
 25 psychlit.mp. 
 26 psychinfo.mp. 
 27 (cochrane adj5 controlled adj5 trials).mp. 
 28 cctr.mp. 
 29 amed.mp. 
 30 (allied adj5 complementary adj5 medicine).mp. 
 31 biosis.mp. 
 32 biological abstracts.mp. 
 33 (cab adj5 health).mp. 
 34 cinahl.mp. 
 35 (cumulative adj5 index adj5 nursing).mp. 
 36 lilacs.mp. 
 37 (science adj5 citation adj5 index).mp. 
 38 (current adj5 contents).mp. 
 39 (web adj5 science).mp. 
 40 or/18‐39 
 41 trials.mp. 
 42 rcts.mp. 
 43 41 or 42 
 44 17 and (7 or 40) and 43

Appendix 3. PsycINFO search strategy

Psychological Abstracts (PsycINFO) was searched using SilverPlatter (WINSPIRS) version 4.0 (1972 to May 2002) using the following search strategy: 
 #1 handsearch* 
 #2 hand‐search* 
 #3 hand near5 search* 
 #4 manual* near5 search* 
 #5 journal* near5 search* 
 #6 fulltext near5 search* 
 #7 full‐text near5 search* 
 #8 full near5 text near5 search* 
 #9 sensitivity 
 #10 databases 
 #11 computer near1 searching 
 #12 (database* near5 search*) or (computer* near5 search*) or (online near5 search*) 
 #13 medline 
 #14 pubmed 
 #15 medlars 
 #16 embase 
 #17 (excerpta near1 medica) in ti,ab 
 #18 psyclit 
 #19 psycinfo in ti 
 #20 psychlit 
 #21 psychinfo 
 #22 cochrane near5 controlled near5 trials 
 #23 cctr 
 #24 amed 
 #25 allied near5 complementary near5 medicine 
 #26 biosis 
 #27 biological near5 abstracts 
 #28 cab near5 health 
 #29 cinahl 
 #30 cumulative near5 index near5 nursing 
 #31 lilacs 
 #32 science near5 citation near5 index 
 #33 current near5 contents 
 #34 web near5 science 
 #35 trials 
 #36 rcts 
 #37 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 
 #38 #10 or #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #27 or #28 or #29 or #30 or #31 or #32 or #33 or #34 
 #39 #35 or #36 
 #40 #37 and #38 and #39

Appendix 4. LISA search strategy

The Library and Information Science Abstracts database (LISA) was searched using SilverPlatter (WINSPIRS) version 4.0 (1969 to July 2002) using the following search strategy: 
 #1 handsearch* 
 #2 hand‐search* 
 #3 hand near5 search* 
 #4 manual* near5 search* 
 #5 journal* near5 search* 
 #6 fulltext near5 search* 
 #7 full‐text near5 search* 
 #8 full near5 text near5 search* 
 #9 sensitivity 
 #10 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 
 #11 database* or (information near1 retrieval) or online or (computer* near5 search*) 
 #12 medline 
 #13 pubmed 
 #14 medlars 
 #15 embase or (excerpta near1 medica) in ti,ab 
 #16 psyclit 
 #17 psycinfo 
 #18 psychlit 
 #19 psychinfo 
 #20 cochrane near5 controlled near5 trials 
 #21 cctr 
 #22 amed 
 #23 allied near5 complementary near5 medicine 
 #24 biosis 
 #25 biological near5 abstracts 
 #26 cab near5 health 
 #27 cinahl in ti, ab 
 #28 cumulative near5 index near5 nursing 
 #29 lilacs 
 #30 science near5 citation near5 index 
 #31 current near5 contents 
 #32 web near5 science 
 #33 #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #27 or #28 or #29 or #30 or #31 or #32 
 #34 trials 
 #35 rcts 
 #36 #34 or #35 
 #37 #10 and #33 and #36

Appendix 5. BIOSIS search strategy

Biological Abstracts (BIOSIS) was searched using SilverPlatter (WINSPIRS) version 4.0 (1985 to June 2002) using the following search strategy: 
 #1 handsearch* 
 #2 hand‐search* 
 #3 hand 
 #4 search* 
 #5 hand near5 search* 
 #6 manual* 
 #7 search* 
 #8 manual* near5 search* 
 #9 journal* 
 #10 search* 
 #11 journal* near5 search* 
 #12 fulltext near5 search* 
 #13 full‐text near5 search* 
 #14 full near text near5 search* 
 #15 sensitivity 
 #16 #1 or #2 or #5 or #8 or #11 or #12 or #13 or #14 or #15 
 #17 (online near5 search*) or (database* near5 search*) or (computer* near5 search*) 
 #18 medline 
 #19 pubmed 
 #20 medlars 
 #21 embase 
 #22 excerpta near1 medica 
 #23 psyclit 
 #24 psycinfo 
 #25 psychlit 
 #26 psychinfo 
 #27 cochrane near5 controlled near5 trials 
 #28 cctr 
 #29 amed 
 #30 allied near1 complementary near1 medicine 
 #31 biosis 
 #32 biological near1 abstracts 
 #33 cab near1 health 
 #34 cinahl 
 #35 cumulative near1 index near1 nursing 
 #36 lilacs 
 #37 science near1 citation near1 index 
 #38 current near1 contents 
 #39 web near1 science 
 #40 #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #27 or #28 or #29 or #30 or #31 or #32 or #33 or #34 or #35 or #36 or #37 or #38 or #39 
 #41 trials 
 #42 rcts 
 #43 #41 or #42 
 #44 #16 and #40 and #43

Appendix 6. AMED search strategy

The Allied and Complementary Medicine Database (AMED) was searched using SilverPlatter (WINSPIRS) version 4.0 (1985 to June 2002) using the following search strategy: 
 #1 handsearch* 
 #2 hand‐search* 
 #3 hand near5 search* 
 #4 manual* near5 search* 
 #5 journal* near5 search* 
 #6 fulltext near5 search* 
 #7 full‐text near5 search* 
 #8 full near5 text near5 search* 
 #9 sensitivity 
 #10 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 
 #11 online or database* or (computer* near5 search*) 
 #12 medline 
 #13 pubmed 
 #14 medlars 
 #15 embase or (excerpta near1 medica) in ti,ab 
 #16 psyclit 
 #17 psycinfo 
 #18 psychlit 
 #19 psychinfo 
 #20 cochrane near5 controlled near5 trials 
 #21 cctr 
 #22 amed 
 #23 allied near5 complementary near5 medicine 
 #24 biosis 
 #25 biological near5 abstracts 
 #26 cab near5 health 
 #27 (cumulative near5 index near5 nursing)or cinahl 
 #28 lilacs 
 #29 science near5 citation near5 index 
 #30 current near5 contents 
 #31 web near5 science 
 #32 #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #27 or #28 or #29 or #30 or #31 
 #33 rcts 
 #34 trials 
 #35 #33 or #34 
 #36 #10 and #32 and #35

Appendix 7. CINAHL search strategy

Cumulative Index to Nursing and Allied Health Literature (CINAHL) was searched using SilverPlatter (WINSPIRS) version 4.0 (1982 to August 2002) using the following search strategy: 
 #1 handsearch* 
 #2 hand‐search* 
 #3 hand near5 search* 
 #4 manual* near5 search* 
 #5 journal* near5 search* 
 #6 fulltext near5 search* 
 #7 full‐text near5 search* 
 #8 full near5 text near5 search* 
 #9 sensitivity 
 #10 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 
 #11 (online near5 search*) or (database* near5 search*) or (computer* near5 search*) 
 #12 medline 
 #13 pubmed 
 #14 medlars 
 #15 embase or (excerpta near1 medica) in ti,ab 
 #16 psyclit 
 #17 psycinfo 
 #18 psychlit 
 #19 psychinfo 
 #20 cochrane near5 controlled near5 trials 
 #21 cctr 
 #22 amed 
 #23 allied near5 complementary near5 medicine 
 #24 biosis 
 #25 biological near5 abstracts 
 #26 cab near5 health 
 #27 cinahl in ti, ab 
 #28 cumulative near5 index near5 nursing 
 #29 lilacs 
 #30 science near5 citation near5 index 
 #31 current near5 contents 
 #32 web near5 science 
 #33 #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #27 or #28 or #29 or #30 or #31 or #32 
 #34 trials 
 #35 rcts 
 #36 #34 or #35 
 #37 #10 and #33 and #36 
 #38 cochrane 
 #39 library 
 #40 (cochrane near1 library) in bk 
 #41 #37 not #40 
 #42 clinical 
 #43 practice 
 #44 guideline 
 #45 (clinical near1 practice near1 guideline) in tx 
 #46 #41 not #45

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Adams 1994.

Methods	Journals were handsearched cover‐to‐cover by one person. A 10% sample of journal years was searched by a second person. MEDLINE was searched (in 1992) for the years 1971, 1976, 1981, 1986, 1991. Citations were downloaded into ProCite and compared to the handsearch.
Data	12 mental health journals (1971, 1976, 1981, 1986, 1991).
Comparisons	Handsearch vs MEDLINE using a simple search (using 'clinical trial') and a complex search using MeSH and text terms.
Outcomes	A total of 743 RCTs were identified; 698 RCTs were identified by HS, 133 RCTs were identified by the simple ES and 388 RCTs were identified by the complex ES.
Notes	Of those found only by HS, 42 RCTs were abstracts, 5 RCTs were letters and 20 RCTs were brief reports, book reviews etc. The time taken to HS was 200 hours and to ES was 8 hours.

Bara 1998.

Methods	No description of handsearching was given. MEDLINE and EMBASE were searched (date not given). The title, abstracts and keywords were checked and full papers were obtained for those records thought to be RCTs.
Data	6 respiratory journals: American Review of Respiratory Diseases, Thorax, British Journal of Diseases of the Chest, Pediatric Pulmonology, Clinical Allergy & European Respiratory Review (1970‐1995).
Comparisons	Handsearch vs MEDLINE and EMBASE using the Cochrane Airways Group search strategy.
Outcomes	A total of 1186 RCTs were identified; 1101 RCTs were identified by HS, 1186 RCTs were identified by ES.
Notes	All 1186 RCTs were full reports.

Bender 1997.

Methods	No description of handsearching was given. MEDLINE was searched (date not given) for the years 1985‐1994.
Data	10 anaesthetics and obstetrics journals (1985‐1994).
Comparisons	Handsearch vs MEDLINE using a simple search (using MeSH) and a complex search (using MeSH and free text terms).
Outcomes	A total of 333 RCTs were identified by the HS and simple ES; 333 RCTs were identified by HS, 176 RCTs were identified by ES. A total of 340 RCTs were identified by the HS and complex ES; 333 RCTs were identified by HS, 221 RCTs were identified by ES.
Notes	All 340 RCTs were full reports. Letters and abstracts were excluded from the analysis.

Bernstein 1988.

Methods	The handsearch results from an earlier study by Poynard (1985) were used. MEDLINE was searched for the years 1966‐1982. Reports found by MEDLINE were checked against the Poynard references and other reports were assessed.
Data	34 gastroenterologic, hepatic, surgical and general medical journals (1966‐1982) searched by Poynard (1985).
Comparisons	Handsearch vs MEDLINE using a revised version of the search strategy used by Poynard (1985).
Outcomes	A total of 195 RCTs were identified; 194 RCTs were identified by HS, 95 RCTs were identified by ES.
Notes	All 195 RCTs were full reports.

Croft 1999.

Methods	One journal was handsearched cover‐to‐cover. The most fruitful years were searched twice to avoid missing trials. MEDLINE was searched (date and period not given) from 1990 to 1998.
Data	Journal of the Royal Medical Corps (1948‐1998).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 10 RCTs/CCTs were identified; 10 RCTs/CCTs identified by HS, 3 RCTs/CCTs identified by ES.
Notes	8 RCTs/CCTs were full reports. 2 RCTs/CCTs were letters found only by HS. The time taken to HS was 34 hours.

Cullum 1997.

Methods	Journals were handsearched cover‐to‐cover. MEDLINE was searched (date and period not given) for the years 1987‐1994. The title, abstracts and keywords found by MEDLINE were checked and full papers were obtained for those records thought to be RCTs.
Data	11 nursing research journals, however, data were only given for 3: Research in Nursing & Health, Journal of Advanced Nursing, International Journal of Nursing Studies (first issue‐1994).
Comparisons	Handsearch vs MEDLINE using a complex search with MeSH headings.
Outcomes	A total of 134 RCTs were identified; 134 RCTs were identified by HS, 90 RCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports. The results were given for only 3 journals, it is not clear why these 3 were reported and not the others.

Dickersin 1994.

Methods	Journals were handsearched cover‐to‐cover by one author. MEDLINE was searched (date and period not given) for the year 1989. Reports of RCTs were copied and verified by two authors. For reports published in 1989, a letter was sent to the authors to confirm it was an RCT.
Data	4 journals for research on vision (1989). English and non‐English language journals.
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 61 RCTs were identified; 61 RCTs were identified by HS, 54 RCTs identified by ES.
Notes	It is not clear if all RCTs were full reports. Data for 1988 journals have been excluded from the analysis, as not all RCTs found were in journals that were handsearched.

Duggan 1997.

Methods	One journal was handsearched cover‐to‐cover by two people, uncertain cases were discussed. MEDLINE and PsycLIT were searched (date and period not given) for the years 1974‐1994.
Data	Journal of Intellectual Disability Research (1957‐1994).
Comparisons	Handsearch vs MEDLINE and PsycLIT using the Cochrane HSSS.
Outcomes	A total of 37 RCTs were identified; 37 RCTs were identified by HS, 24 RCTs were identified by MEDLINE, 26 RCTs were identified by PsycLIT.
Notes	It was not clear if all RCTs were full reports.

Fernandes 2000.

Methods	Journals were handsearched cover‐to‐cover. LILACS and the Cochrane Controlled Trials Register were searched (date and period not given).
Data	5 Brazilian specialized journals in angiology and vascular surgery: Cirurgia Vascular e Angiologia, Revista de Angiologia e Cirurgia Vascular, Revista Brasileira de Angiologia e Cirurgia Vascular, Angiopatias. Non‐English language journals.
Comparisons	Handsearch vs LILACS and the Cochrane Controlled Trials Register using the Cochrane HSSS.
Outcomes	A total of 9 RCTs/CCTs were identified; 9 RCTs/CCTs were identifed by HS, 1 RCT/CCT was identifed by ES.
Notes	All RCTs/CCTs were full reports.

Galandi 2000.

Methods	Journals were handsearched cover‐to‐cover. MEDLINE and EMBASE were searched (date and period not given).
Data	6 German general healthcare journals (1948‐1998). Non‐English language journals.
Comparisons	Handsearch vs MEDLINE using RCT in PT and EMBASE using 'controlled study'.
Outcomes	A total of 1690 RCTs were identified; 1690 RCTs were identified by HS, 558 RCTs were identified by MEDLINE, 710 RCTs were identified by EMBASE.
Notes	It is not clear if all RCTs were full reports.

Gluud 1998.

Methods	One journal was handsearched cover‐to‐cover by one person. Uncertain cases were discussed. MEDLINE was searched (date and period not given).
Data	Journal of Hepatology (1985‐1997).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 171 RCTs were identified; 169 RCTs were identified by HS, 139 RCTs were identified by ES.
Notes	141 RCTs were full reports and 30 RCTs were abstracts.

Hay 1996.

Methods	One journal was handsearched cover‐to‐cover. A 10% sample of journal years was searched by a second person. EMBASE and PsycLIT were searched (date and period not given).
Data	International Journal of Eating Disorders (Aug 1981‐Nov 1993).
Comparisons	Handsearch vs EMBASE and PsycLIT using a complex search strategy.
Outcomes	A total of 40 RCTs were identified; 40 RCTs were identified by HS, 35 RCTs were identified by EMBASE, 27 RCTs were identified by PsycLIT.
Notes	It is not clear if all RCTs were full reports.

Hedger 1999.

Methods	One journal was handsearched cover‐to‐cover. The most fruitful years were searched twice to avoid missing trials. MEDLINE was searched (date not given) for the years 1990‐1998 (1990 was the date the journal was first indexed in MEDLINE).
Data	Journal of the Royal Naval Medical Service (1948‐1998).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 5 RCTs/CCTs were identified; 5 RCTs/CCTs were identified by HS, 2 RCTs/CCTs were identified by ES.
Notes	All RCTs/CCTs were full reports. The total time taken to HS and ES was 37 hours.

Hopewell 2002a.

Methods	Journals were handsearched cover‐to‐cover. A 10% sample of journal years was searched by a second person. MEDLINE was searched (October 1999) for the years 1970‐1999. The full article was obtained for reports identified by MEDLINE and not found by handsearching.
Data	22 UK specialized healthcare journals (sample years from 1970‐1999).
Comparisons	Handsearch vs MEDLINE search using RCT and CCT in PT.
Outcomes	A total of 714 RCTs were identified; 682 RCTs were identified by HS, 345 RCTs were identified by ES.
Notes	252 RCTs had no MEDLINE record, of which 232 RCTs were abstracts and supplements. 117 RCTs had a MEDLINE record but were not indexed as RCTs/CCTs, the majority of which were published pre‐1991.

Jadad 1993a.

Methods	Journals were handsearched cover‐to‐cover. MEDLINE was searched (date not given) for the years 1966‐1992. The title, abstracts and keywords were checked and full papers were obtained for those records thought to be RCTs.
Data	9 anaesthetics and pain journals (1970, 1980, 1990).
Comparisons	Handsearch vs MEDLINE using a complex search with MeSH and truncated text.
Outcomes	A total of 315 RCTs were identified; 313 RCTs were identified by HS, 138 RCTs were identified by ES.
Notes	144 RCTs were full reports; 142 RCTs were found by HS, 125 RCTs were found by ES. 171 RCTs were abstracts and letters; 171 RCTs were found by HS, 13 RCTs were found by ES.

Jadad 1993b.

Methods	The handsearch results from an earlier study by Silagy (1993) were used. MEDLINE was searched (date and period not given) and the results compared to those identified by Silagy to identify further RCTs.
Data	7 primary healthcare journals searched by Silagy (1993).
Comparisons	Handsearch vs MEDLINE using a revised version of the search strategy used by Silagy (1993).
Outcomes	A total of 202 RCTs were identified; 202 RCTs were identified by HS, 179 RCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports.

Jadad 1996.

Methods	Journals were handsearched cover‐to‐cover. All reports identified were then further assessed and a random sample of all handsearching was also checked. MEDLINE was searched (date not given) for the years 1966‐1990. The title, abstracts and keywords were checked and full papers were obtained for those records thought to be RCTs.
Data	36 journals on pain research (1950‐1990).
Comparisons	Handsearch vs MEDLINE using a refined high yield search strategy.
Outcomes	A total of 4702 RCTs were identified; 4336 RCTs were identified by HS, 2523 RCTs were identified by ES.
Notes	Of those reports found only by HS (2179), 1125 RCTs were abstracts, 959 RCTs were inadequately indexed and the failure to find 95 RCTs by ES could not be explained. The time taken to HS was 623 hours.

Jefferson 1996.

Methods	One journal was handsearched cover‐to‐cover. Uncertain cases of randomization were classed as CCT. MEDLINE was searched (date and period not given).
Data	Vaccine (1983‐1994).
Comparisons	Handsearch vs MEDLINE using a complex search strategy.
Outcomes	A total of 231 RCTs/CCTs were identified; 231 RCTs/CCTs were identified by HS, 60 RCTs/CCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports.

Kennedy 2000.

Methods	One journal was handsearched cover‐to‐cover by one person, a second person searched selected issues and a third person adjudicated discrepancies. MEDLINE was searched (date and period not given).
Data	AIDS including supplements (1987‐1999).
Comparisons	Handsearch vs MEDLINE search using Randomized Controlled Trial in PT and RCT in MeSH.
Outcomes	A total of 172 RCTs were identified; 172 RCTs were identified by HS, 155 RCTs were identified by ES.
Notes	It is not clear if all RCTs/CCTs were full reports.

Kirpalani 1999.

Methods	Three people handsearched separate journals with no overlap. MEDLINE was searched (date not given) for the year 1985 .
Data	10 journals for neonatology trials (1985).
Comparisons	Handsearch vs MEDLINE using free text terms (random allocation, random, placebo).
Outcomes	A total of 53 RCTs were identified; 53 RCTs were identified by HS, 32 RCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports.

Langham 1999.

Methods	Journals were handsearched cover‐to‐cover. There was an overlap of one year per journal to assess quality of handsearching. MEDLINE was searched (date not given) for the years 1966‐1995. The full article was obtained for records identified by MEDLINE and not found by handsearching. These were assessed by one person.
Data	14 emergency medicine journals (1966‐1995).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS
Outcomes	A total of 710 RCTs were identified; 592 RCTs were identified by HS, 483 RCTs were identified by ES.
Notes	All 710 RCTs were full reports. An additional 285 RCT abstracts were found only by the HS but these were excluded from the analysis.

Marson 1996.

Methods	Journals were handsearched for full reports only. MEDLINE was searched (date not given) for the years 1966‐1993.
Data	3 epilepsy journals: Epilepsia and Acta Neurologica Scandinavica (1966‐1993), Epilepsy Research (1987‐1993).
Comparisons	Handsearch vs MEDLINE using a complex search (MeSH, PT, truncated text) and a simple search (RCT in PT, RANDOM*).
Outcomes	A total of 118 RCTs were identified; 118 RCTs were identified by HS, 76 RCTs were identified by the simple ES and 102 RCTs were identified by the complex ES.
Notes	All 118 RCTs were full reports.

Marti 1999.

Methods	Journals were handsearched cover‐to‐cover. MEDLINE was searched (date not given) for the years 1971‐1995.
Data	4 Spanish general and internal medicine journals. Medicina Clinica, Revista Clinical Espanola, Atencion Primaria, Anales de Medicina Interna. Non‐English language journals.
Comparisons	Handsearching vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 215 RCTs were identified; 215 RCTs were identified by HS, 146 RCTs were identified by ES.
Notes	It is not clear now many of the 215 RCTs were full reports, abstracts, letters etc.

McDonald 1997.

Methods	Journals were handsearched cover‐to‐cover. MEDLINE was searched (date not given) for the year 1996. All reports found by MEDLINE and indexed RCT or CCT in PT were checked.
Data	12 UK general healthcare journals (1996).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS and RCT in PT.
Outcomes	A total of 358 RCTs were identified; 358 RCTs were identified by HS, 293 RCTs were identified using the Cochrane HSSS and 208 RCTs were identified using RCT in PT.
Notes	It is not clear how many of the 358 RCTs were full reports, abstracts of letters.

Neal 1996.

Methods	One journal was handsearched cover‐to‐cover by one person. MEDLINE was searched (date not given) for the years 1966‐1995.
Data	New Zealand Medical Journal (1943‐1995).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 143 RCTs were identified; 143 RCTs were identified by HS, 89 RCTs were identified by ES.
Notes	115 RCTs were full reports, 37 RCTs were abstracts.

Nwosu 1998.

Methods	Journals were handsearched cover‐to‐cover by one person, all articles identified were checked by a second person. One person also performed a sample check as quality control. MEDLINE was searched (date not given) for the years 1966‐1996.
Data	4 obstetrics and gynaecology journals: Acta Obstertricia et Gynecologica, American Journal of Obstetrics & Gynaecology, British Journal of Obstetrics & Gynaecology, Obstetrics & Gynaecology (1975, 1980, 1985, 1990, 1995).
Comparisons	Handsearch vs MEDLINE using RCT and CCT in PT.
Outcomes	A total of 403 RCTs were identified; 389 RCTs were identified by HS, 292 RCTs were identified by ES.
Notes	All 403 RCTs were full reports.

Poynard 1985.

Methods	One person searched each issue including references to reports found in addition to full reports. Abstracts were excluded from the analysis. MEDLINE was searched (March 1983) for the years 1966‐1982. All publications reviewed and letters sent to authors if randomization was unclear.
Data	34 gastroenterologic, hepatic, surgical and general medical journals (1966‐1982). English and non‐English language journals.
Comparisons	Handsearch vs MEDLINE using MeSH, including subheadings and free text.
Outcomes	A total of 279 RCTs were identified; 244 RCTs were identified by HS, 189 RCTs were identified by ES.
Notes	218 RCTs were full reports, 61 RCTs were references to full articles.

Reynolds 1997.

Methods	Trained MEDLINE searchers handsearched each article of the journals and differences in retrieval were discussed. MEDLINE was searched (date not given) for the years 1970, 1981, 1992. The seachers reviewed each article found by the MEDLINE search.
Data	11 US general medical journals (1970, 1981, 1992).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 470 RCTs/CCTs were identified, 395 RCTs were identified by HS, 380 RCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports.

Silagy 1993.

Methods	Journals were handsearched cover‐to‐cover by one author. MEDLINE was searched (date not given) for the years 1987‐1991. Full papers were obtained if it was unclear it is was an RCT.
Data	7 primary healthcare journals (1987‐1991).
Comparisons	Handsearch vs MEDLINE using a complex search.
Outcomes	A total of 204 RCTs were identified; 198 RCTs were identified by HS, 135 RCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports.

Solomon 1994.

Methods	Journals were handsearched cover‐to‐cover by one person. MEDLINE was searched (date not given) for the year 1990. The title, abstracts and keywords retrieved by MEDLINE were checked and full papers were obtained for those records thought to be RCTs.
Data	3 general surgery journals: British Journal of Surgery, Surgery, Diseases of the Colon & Rectum (1990).
Comparisons	Handsearch vs MEDLINE using a simple search (Mesh, PT) and a complex search (Mesh, PT, free text).
Outcomes	A total of 37 RCTs were identified; 37 RCTs were identified by HS, 17 RCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports.

Suarez‐Almazor 2000.

Methods	Journals were handsearched cover‐to‐cover. MEDLINE and EMBASE were searched (1996 and updated in 1999) for the years 1988‐1994. The title and abstract of reports identified were assessed by 2 people. If it was unclear if it was a controlled trial (CT) the whole article was obtained.
Data	Specialized journals in rheumatoid arthritis, osteoporosis and low back pain (1988, 1994).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS and EMBASE using a comparable search.
Outcomes	A total of 259 CTs were identified; 259 CTs were identified by HS, 188 CTs were identified by MEDLINE and 220 CTs were identified by EMBASE.
Notes	Of those CTs found only by HS, 12 CTs were inadequately indexed, the journal issue was not indexed for 3 CTs and the relevant journal year was not indexed for 1 CT.

Tavender 1999.

Methods	Journals were handsearched cover‐to‐cover. MEDLINE was searched (date not given) for the years 1966‐1998.
Data	5 oral health journals: Journal of American Dental Association, Journal of Clinical Periodontics, Journal of Periodontal Research, Journal of Periodontology, (1984‐1993); American Journal of Orthodontic & Dentofacial Orthopedics (1989‐1998).
Comparisons	Handsearch vs MEDLINE using the Cochrane HSSS.
Outcomes	A total of 522 RCTs were identified; 522 RCTs were identified by HS, 346 RCTs were identified by ES.
Notes	It is not clear if all RCTs were full reports.

Watson 1999a.

Methods	Journals were handsearched cover‐to‐cover by one person. MEDLINE and PsycINFO were searched.
Data	10 group psychotherapy journals (1993‐1994).
Comparisons	Handsearch vs MEDLINE and PsycINFO using a simple and complex search.
Outcomes	A total of 23 RCTs were identified; 22 RCTs were identified by HS. MEDLINE identified 16 RCTs and PsycINFO identified 14 RCTs.
Notes	All 23 RCTs were full reports.

Watson 1999b.

Methods	Journals were handsearched cover‐to‐cover by one person. MEDLINE, EMBASE and PsycINFO were searched for the years 1992‐1996.
Data	5 journals: American Journal of Psychiatry, Archives of General Psychiatry, Behaviour Research & Therapy, British journal of Psychiatry, Journal of Consulting & Clinical Psychology (1992‐1996).
Comparisons	Handsearch vs MEDLINE, EMBASE and PsycINFO using a simple search (using indexing terms) and a complex search (using free text and indexing terms).
Outcomes	A total of 37 RCTs were identified; 34 RCTs were identified by HS, MEDLINE identified 36 RCTs, EMBASE identified 28 RCTs, PsycINFO identified 24 RCTs.
Notes	All 37 RCTs were full reports.

Abbreviations:

HS: handsearching. 
 ES: electronic searching. 
 'cover‐to‐cover': the full contents of each journal were handsearched (i.e., each journal article, review, letter, meeting abstract, etc were checked). 
 RCT: randomized controlled trial as described by the authors of the study. 
 CCT: controlled clinical trial as described by the authors of the study. 
 Cochrane HSSS: Cochrane highly sensitive search strategy.

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Adetugbo 2000	This study compared handsearching to MEDLINE for reports of randomized trials in the Journal of Clinical & Experimental Dermatology. This study was excluded because reports found by handsearching were then specifically searched for in MEDLINE.
Bareta 1990	This study compared handsearching to MEDLINE for articles on consultation‐liaison psychiatry, which were not reports of randomized trials.
Bereczki 2000	This study compared handsearching to MEDLINE for reports of randomized trials in the Hungarian Journal of Clinical Neuroscience. This study was excluded because reports found by handsearching were then specifically searched for in MEDLINE.
Burdett 1997	This study was excluded because there was not direct comparison between handsearching and electronic searching.
Campbell 2000	This study was excluded because there was not direct comparison between handsearching and electronic searching.
Dickersin 1985	This study compared MEDLINE to searching a Perinatal Trials Register, which included reports of randomized trials identified by handsearching, MEDLINE searching and additional adhoc methods. This study was excluded because the Register could not be separated into reports found by handsearching and by other methods.
Dumbrigue 1999	This study was excluded because there was not direct comparison between handsearching and electronic searching.
Gotzsche 1991	This study was excluded because there was not direct comparison between handsearching and electronic searching.
Johnson 1995	This study was excluded because the journal was searched full text 'electronically' and was not searched by hand.
Kleijnen 1992	This study was excluded because there was not direct comparison between handsearching and electronic searching.
Liu 1998	This study compared handsearching to MEDLINE for reports of randomized trials in Chinese medical journals. This study was excluded because reports found by handsearching were then specifically searched for in MEDLINE.
Matthews 1999	This study was excluded because there was not direct comparison between handsearching and electronic searching.
Ruther 1997	This study was excluded because reports found by handsearching were then specifically searched for in MEDLINE.
Schlomer 1999	This study was excluded as there was no direct comparison of handsearching versus MEDLINE and CINAHL.
Slim 2000	This study was excluded because there was not direct comparison between handsearching and electronic searching.
Vlassov 2000	This study compared handsearching to MEDLINE for reports of screening and diagnostic tests, which were not reports of randomized trials, in the Russian medical literature.

Contributions of authors

Sally Hopewell contributed to the identification of studies, data collection, analysis and preparation of the final manuscript. Mike Clarke and Roberta Scherer contributed to the identification of studies, data collection, analysis and commenting on the final manuscript. Carol Lefebvre contributed to the search strategies, identification of studies and commenting on the final manuscript.

Sources of support

Internal sources

• UK Cochrane Centre, NHS Research and Development Programme, UK.

• University of Maryland, Baltimore, USA.

External sources

• No sources of support supplied

Declarations of interest

Sally Hopewell, Mike Clarke and Carol Lefebvre work at the UK Cochrane Centre, Oxford and are involved in the preparation of systematic reviews and other aspects of the production of The Cochrane Library (including The Cochrane Central Register of Controlled Trials and The Cochrane Methodology Register).

Unchanged