Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews

Introduction

Identifying relevant studies is “the most fundamental challenge” when compiling a systematic review.¹ Electronic databases, such as Medline, may detect only about half of papers identified by the gold standard of hand searching journals.¹ Hand searching requires a focus, usually the specialist literature,² which may not exist for newly developed fields or those that cross boundaries with other areas. We examined the usefulness of contacting other experts when searching for relevant references for a systematic review of a field where such a specialist focus does not exist.

Methods and results

As part of a systematic review undertaken in 1996, all published literature relating to “near patient testing” (any investigation performed in a clinical setting where the result is available without a sample being sent to a laboratory for analysis) in primary care was identified for 1986-95.³ Electronic databases were searched and secondary citations were collected from identified publications (see table). The search strategy is reported elsewhere.³ Indexes of abstracts from major international primary care scientific conferences were hand searched. We sent a questionnaire to 194 academics in the United Kingdom (heads of academic departments of general practice and clinical chemistry and researchers identified from the previously mentioned abstracts) and to 152 commercial companies known to have an interest in near patient testing. The questionnaire requested key references from journals, unpublished data, and names of other workers in the field.

Overall, 156 (45%) questionnaires were returned completed (103 (53%) of those sent to academics and 53 (35%) of those sent to commercial companies). No unpublished data were offered.

Articles that did not report original data, were not relevant to primary care, or were not in English were excluded. Remaining articles were then judged by both an external and internal reviewer against standard appraisal criteria,⁴ with discrepancies being adjudicated on by a third reviewer (BCD). A verifying search did not identify any further key words or references.

The searches yielded 1286 citations comprising 1057 unique references potentially eligible for inclusion in the review. The table shows a breakdown of the results by source—102 references were relevant, of which 29 were of high quality. Of the 102 unique eligible references, 50 (49%) were identified by one of the electronic databases, 40 (39%) by people working in the field, and 31 (30%) by hand searches. Each of these sources produced a similar proportion of high quality papers.

The 1057 potential references came from 418 different publications, of which only 48 were found to contain relevant articles. Only one journal, the BMJ, contained more than 10% (11 articles) of the relevant references.

Comment

This study confirms that searching electronic databases may uncover only half of all relevant studies¹ and shows the importance of contacting other experts when compiling a systematic review. Over 300 of the references in this review were identified by experts working in the field, of which 40 were found subsequently to be eligible. Twenty four references (24% of the total number of eligible references) would have been missed entirely without the input of people working in the field.

The lack of sensitivity of electronic databases may be due to problems with indexing⁵: articles from journals that lie outside of the mainstream (which currently includes primary care) take some years to be electronically indexed; furthermore, near patient testing is an expanding field without unique medical subject headings (MeSH terms). The continuing work of the Cochrane Collaboration is improving this situation,⁵ but at the time of this study, making up the shortfall in articles by hand searching would not have been justifiable in terms of either time or money.

This study has quantified the contribution of experts to a systematic review and has found them to be an essential source for identifying literature. We suggest that appropriate experts should be consulted when performing a systematic review in a developing field that does not have a clearly defined specialist literature.

Table.

Results of various types of search

Search	No of articles identified	No of articles eligible in review	Unique articles identified	High quality articles*
Electronic databases
BIDS Science Citation Index	185	32	15	5
Medline	335	20	11	9
Embase	40	11	5	4
CINAHL	48	0	0	0
BIDS index to conference proceedings	32	0	0	0
GPLit	26	0	0	0
DHSS	18	0	0	0
PsychLit	10	0	0	0
Total No of articles	694	63	31	18
No of unique articles	581	50	31	15
Survey of expert network
Academics	297	37	20	7
Commercial organisations	43	4	3	2
Total No of articles	340	41	23	9
No of unique articles	338	40	23	9
Hand searching†
Abstracts	44	0	0	0
Reports	29	7	1	2
References	129	24	20	8
Total No of articles (all unique)	202	31	21	10
Overall results
Total citations identified	1236	135	75	37
Total unique references‡	1057	102	75	29

^*Eligible papers scoring 4 or 5 for methodology (one point was given for each of the following: an independent blind comparison to a reference standard; an appropriate spectrum of patients in the sample; reference standard performed on all patients; test methods described sufficiently to permit replication; likelihood ratios quoted). 

^†Limited hand search of published abstracts of international primary care meetings, and bibliographies of publications identified from above sources. 

^‡Excluding overlaps between the three search methods.