Grey literature in meta‐analyses of randomized trials of health care interventions

Abstract

Background

The inclusion of grey literature (i.e. literature that has not been formally published) in systematic reviews may help to overcome some of the problems of publication bias, which can arise due to the selective availability of data.

Objectives

To review systematically research studies, which have investigated the impact of grey literature in meta‐analyses of randomized trials of health care interventions.

Search methods

We searched the Cochrane Methodology Register (The Cochrane Library Issue 3, 2005), MEDLINE (1966 to 20 May 2005), the Science Citation Index (June 2005) and contacted researchers who may have carried out relevant studies.

Selection criteria

A study was considered eligible for this review if it compared the effect of the inclusion and exclusion of grey literature on the results of a cohort of meta‐analyses of randomized trials.

Data collection and analysis

Data were extracted from each report independently by two reviewers. The main outcome measure was an estimate of the impact of trials from the grey literature on the pooled effect estimates of the meta‐analyses. Information was also collected on the area of health care, the number of meta‐analyses, the number of trials, the number of trial participants, the year of publication of the trials, the language and country of publication of the trials, the number and type of grey and published literature, and methodological quality.

Main results

Five studies met the inclusion criteria. All five studies showed that published trials showed an overall greater treatment effect than grey trials. This difference was statistically significant in one of the five studies. Data could be combined for three of the five studies. This showed that, on average, published trials showed a 9% greater treatment effect than grey trials (ratio of odds ratios for grey versus published trials 1.09; 95% CI 1.03‐1.16). Overall there were more published trials included in the meta‐analyses than grey trials (median 224 (IQR 108‐365) versus 45(IQR 40‐102)). Published trials had more participants on average. The most common types of grey literature were abstracts (55%) and unpublished data (30%). There is limited evidence to show whether grey trials are of poorer methodological quality than published trials.

Authors' conclusions

This review shows that published trials tend to be larger and show an overall greater treatment effect than grey trials. This has important implications for reviewers who need to ensure they identify grey trials, in order to minimise the risk of introducing bias into their review.

Plain language summary

Grey literature in meta‐analyses of randomized trials of health care interventions

This methodology review identified five studies which investigated the effect of including trials found in the grey literature in systematic reviews of health care interventions. They showed that trials found in the published literature tend to be larger and show larger effects of a health care intervention than those trials found in the grey literature. There was limited evidence to show whether grey trials are of poorer methodological quality than published trials. This means that those carrying out systematic reviews need to search for trials in both the published and grey literature in order to help minimise the effects of publication bias in their review.

Background

The validity of a systematic review is highly dependent on the results of the underlying data. The inclusion of grey literature in systematic reviews may help to overcome some of the problems of publication bias, which can arise due to the selective availability of data. The definition of what constitutes grey literature varies and the terminology can be confusing (Auger 1998; Loo 1985; Alberani 1990; Cook 1993; McAuley 2000; Song 2000). Acceptance of the term 'grey literature' dates back to 1978 with the creation of the System for Information on Grey Literature in Europe database, which is now managed by the European Association for Grey Literature Exploitation (EAGLE). In 1997, the term grey literature was defined at the Third International Conference on Grey Literature (Auger 1998). This revised definition is widely accepted and is known as 'The Luxembourg Convention' after the city in which the conference was held. They define grey literature as "that which is produced on all levels of governmental, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers". Examples of grey literature include conference abstracts, research reports, book chapters, unpublished data, dissertations, policy documents and personal correspondence.

In social sciences research, it has been known for some time that omitting unpublished studies from a meta‐analysis can magnify the effect of the intervention. For example, in a study of 11 meta‐analyses published between 1976 and 1980, the average experimental effects found in studies published in journals was larger than the corresponding effect from theses and dissertations (Glass 1981). There is now some evidence in support of this in health research, suggesting that the exclusion of grey literature from meta‐analyses can lead to an exaggeration of the effect of treatment (McAuley 2000; Sterne 2000).

However, the identification of relevant studies in the grey literature and their inclusion in systematic reviews can be particularly time‐consuming and difficult. There is also some controversy as to whether unpublished studies should be included in meta‐analyses because they might be incomplete and their methodological quality can be difficult to assess. A survey by Cook and colleagues in the early 1990s showed that 78% of meta‐analysts and methodologists felt that unpublished material should be included in meta‐analyses compared to only 47% of journal editors (Cook 1993).

Research is needed to help assess the potential implications for reviewers of not including grey literature in systematic reviews of health care interventions. This review builds on work carried out by one of the authors (Egger 2003) and systematically reviews research studies that have investigated the impact of including grey literature in meta‐analyses of healthcare interventions.

Objectives

To review systematically research studies, which have investigated the impact of including grey literature in meta‐analyses of randomized trials of healthcare interventions.

Methods

Criteria for considering studies for this review

Types of studies

A research study was considered eligible for inclusion in this review if it compared the impact of the inclusion and exclusion of grey literature on the results of a cohort (more than one) of meta‐analyses of randomized trials. The definition of what constitutes grey literature was that used by the authors in each of the empirical studies and which also conformed to the definition of 'grey literature' described earlier in this review. A meta‐analysis is defined as being where the results of two or more trials are calculated and then statistically pooled to produce a single estimate of treatment effect.

A previous version of this review included the results of individual systematic reviews, where the impact of trials reported in the grey literature on the overall results of a meta‐analysis were assessed as a sensitivity analysis and then reported in the review. In this updated version of the review these and other single meta‐analyses have been excluded (see Characteristics of Excluded Studies table). Recent empirical evidence on selective outcome reporting bias (Chan 2004) would suggest that including these studies might introduce bias; if individual systematic reviews did not find a difference between grey and published trials, they are much less likely to report this than if they did find a difference.

Types of data

Information was collected on the area of health care, the number of meta‐analyses, the number of randomized trials, the number of participants in these trials, the year of publication of the trials, the language of publication of the trials, the number and type of grey and published literature, and the methodological quality of the empirical studies and the randomized trials they included.

Types of methods

The inclusion and the exclusion of trials from the grey literature on the pooled effect estimate of the meta‐analysis.

Types of outcome measures

The main outcome measure was an estimate of the impact of trials from the grey literature on the pooled effect estimates of the meta‐analyses. Where data were available this was done by calculating a ratio of odds ratios (ROR) between the results of trials identified in the grey literature and the results of trials identified in the published literature and estimating the percentage change. A ratio greater than 1.0 would indicate that published trials showed a greater treatment effect likewise a ratio of below 1.0 would indicate that grey trials would show a greater treatment effect. A weighted average was used to combine ratios from meta‐analyses in each of the included studies to produce an overall pooled effect estimate, which also takes into account factors such as number of trials, patients and events.

Search methods for identification of studies

Studies were sought from the Cochrane Methodology Register (The Cochrane Library Issue 3, 2005) and MEDLINE (1966 to 20 May 2005). The following search terms were used to search the Cochrane Methodology Register: Data collection ‐ unpublished data OR Study identification ‐ publication bias. The following search strategy was run against MEDLINE on OVID:

1 (Meta‐Analysis or Clinical‐Trials or Randomized‐Controlled‐Trials or Selection‐Bias or Review‐Literature).sh 
 2 (meta‐analys$ or systematic review$ or data synthes$).tw 
 3 or/1‐2 
 4 (grey literature or gray literature or unpublished).tw 
 5 exp Publication Bias 
 6 or/4‐5 
 7 3 and 6

Studies were also sought during the handsearching of selected journals, which is being carried out by the UK Cochrane Centre for all studies relevant to the methodology of systematic reviews. The abstracts presented at all Cochrane Colloquia (1993 to 2004), Systematic Reviews Symposia (1998 to 2002) and Society for Clinical Trials Meetings (1980 to 2004) (as published in Controlled Clinical Trials) and, more recently, Clinical Trials, have also been handsearched as part of this activity.

The titles and abstracts of records retrieved with these strategies were assessed for relevance to this review (see below, Identifying studies). References in relevant reports were checked to identify additional studies. The Science Citation Index (June 2005) was used to identify articles that cited relevant reports. Finally, researchers who may have carried out relevant studies were contacted.

Data collection and analysis

Identifying studies

One reviewer (SH) screened the titles and abstracts of all retrieved records to identify obvious exclusions (i.e. records that were clearly irrelevant to this review but were found by the electronic searches noted above). A second reviewer (SM) checked less obvious records, before rejection. Full copies of the reports were obtained for each of the non‐rejected records. These reports were then assessed independently by at least two reviewers (SH and SM) to determine if they met the inclusion criteria for the review. Any disagreements were resolved through discussion.

Assessment of methodological quality

The methodological quality of the included studies was assessed against the following criteria: Were explicit criteria used to categorise or define grey literature? Did two or more investigators agree regarding the inclusion of grey literature material? Was there completeness of data for the randomized trials in the meta‐analyses included in the empirical study? Any disagreements were resolved through discussion.

Data extraction

Data extraction was performed independently by two reviewers (SH and SM). Information was extracted for each included empirical study on the area of health care, the number of meta‐analyses, the number of randomized trials, the number of participants in these trials, the year of publication of the trials, the language of publication of the trials, the number and type of grey and published literature. Information was also extracted to assess the methodological quality of the included empirical studies. If any of the data for an empirical study were insufficient or missing, these were sought initially from the named contact author (or the responsible person in the case of unpublished studies).

Data analysis

The decision on whether or not to combine the results of the included studies was dependent on an assessment of heterogeneity. Where studies were judged to be sufficiently homogenous in their design a meta‐analysis of these studies was carried out. This was done by using the generic inverse variance method available in RevMan and calculating the log of the ratio of odds ratios (ROR) for grey versus published trials and its corresponding standard error (SE) for each study.

Subgroup analysis

Data for the following proposed subgroup analyses are presented descriptively: the type of grey literature; the area of health care; the number of randomized trials and the number of participants in these trials.

Results

Description of studies

Included studies

Five studies met the inclusion criteria for this review (Burdett 2003; Egger 2003; Fergusson 2000; Hopewell 2004; McAuley 2000), four were published as full papers and one was an unpublished thesis (Hopewell 2004). All five studies included a cohort of meta‐analyses comparing the inclusion and exclusion of grey literature on the pooled effect estimate of the meta‐analyses. One of these studies (Burdett 2003) assessed a cohort of individual patient data meta‐analyses.

Three studies included meta‐analyses from specific areas of health care (orthopaedic and cardiac surgery and cancer) and two studies included meta‐analyses from a number of different medical specialties. The meta‐analyses were identified from a variety of sources, including searches of MEDLINE and specialized trial registers, handsearches of journals and contacting researchers in the area of interest. Within the included studies the definition of what constituted grey literature varied slightly, however, in general it included abstracts, book chapters, unpublished data, theses, company reports and letters. More information is available in the Characteristics of Included Studies table.

Excluded studies

Four studies were excluded from this review (Bhandari 2000; Horn 2000; Jeng 1995; Man‐Son‐Hing 1998); each reported the results of an individual systematic review (meta‐analysis) where the difference between the treatment effect of grey and published trials was explored in a sensitivity analysis (see Characteristics of Excluded Studies table).

Ongoing studies

No ongoing studies have been identified.

Risk of bias in included studies

We assessed the methodological quality of the included studies using the criteria highlighted above. All of the studies used explicit criteria to categorise or define grey literature and in four of the five studies (Burdett 2003; Egger 2003; Hopewell 2004; McAuley 2000) two or more investigators agreed regarding the grey literature material included in the meta‐analyses. In four of the five studies there was judged to be completeness of data for the randomized trials in the individual meta‐analyses included in the empirical study. It was not clear if this was the case for the remaining study (Fergusson 2000).

Effect of methods

Characteristics of the meta‐analyses

(Table 1) 
 The number of meta‐analyses included in the studies ranged from between 10 and 60 individual meta‐analyses. The studies containing the greatest number of meta‐analyses were by Egger 2003 (60 meta‐analyses) and McAuley 2000 (41 meta‐analyses).

1. Characteristics of the meta‐analyses.

Study	No. meta‐analyses	No. trials	No. participants	No. grey trials	No. pub trials	Area healthcare	Year publication
Burdett 2003	11	120	18,377	45	75	Cancer
Egger 2003	60	783	167,733	153	630	Various medical specialties	1994‐1998
Fergusson 2000	10	114	11,142	6	108	Cardiac and Orthopaedic surgery
Hopewell 2004	17	264	356,544	40	224	Cancer
McAuley 2000	41	467	217,427	102	365	Gastrointestinal (24%), Cardiac (21%), Infection (12%), Reproduction (12%), Circulatory (9%), Other (21%)

Characteristics of the trials included in the meta‐analyses

(Table 2) 
 The number of trials included in the meta‐analyses ranged from 114 to 783 trials. Again the studies by Egger 2003 (783 trials) and McAuley 2000 (467 trials) contained the greatest number of trials. In all cases there were more published trials included in the meta‐analyses than grey trials (median 224 (IQR 108‐365) versus median 45 (IQR 40‐102)). Grey trials are defined as trials in the grey literature, which have not yet been published in full. The number of participants included in the trials was also larger for published trials compared to grey trials. The median number of participants per study was 10,162 (IQR 6221‐21,573) for grey trials compared to a median of 146,169 (12,156‐194,141) for published trials. Egger 2003 also assessed the statistical significance of the trial results included in the meta‐analyses. They found that published trials (30%) were more likely to have statistically significant results (p<0.05) than grey trials (19%).

2. Characteristics of the trials included in meta‐analyses.

Study	No. trials	No. participants	No. per trial	Positive trials	Negative trials
Burdett 2003	Grey 45; Published 75	Grey 6,221; Published 12,156	Grey (mean) 138; Published (mean) 162
Egger 2003	Grey 153; Published 630	Grey 21,573; Published 146,160	Grey (median) 91 (range 9‐1012); Published (median) 102 (range 8‐5042)	p<0.05: Grey 29 (19%); Published 187 (30%)	p<0.01: Grey 18 (12%); Published 100 (16%)
Fergusson 2000	Grey 6; Published 108	Grey 820; Published 10,322	Grey (median) 149 (IQR 85‐250); Published (median) 638 (IQR 240‐828)
Hopewell 2004	Grey 40; Published 224	Grey 10,162; Published 346,382	Grey (median) 167 (IQR 77‐326); Published (median) 186 (IQR 80‐349)
McAuley 2000	Grey 102; Published 365	Grey 23,286; Published 194,141	Grey (median) 84 (IQR 48‐190); Published (median) 113 (IQR 59‐228)	p<0.05: Grey 70 (68%); Published 240 (66%)	p>=0.05: Grey 32 (32%); Published 125 (35%)

Language of publication of the trials

(Table 3) 
 Information on the language of publication of the trials included in the meta‐analyses was available for three (Burdett 2003; Fergusson 2000; McAuley 2000) of the five included studies. The majority of trials were published in English. This was similar for both grey and published trials and no language restrictions appear to have been imposed by the authors of the studies.

3. Language of publication of the trials.

Study	English	French	German	Spanish	Italian	Other
Burdett 2003	Grey 44 (98%); Published 75 (100%)					Grey 1 (2%)
Fergusson 2000	Overall 100 (88%)	Overall 6 (4%)	Overall 4 (3%)	Overall 2 (2%)	Overall 3 (2%)	Overall 1 Belgian (1%)
McAuley 2000	Grey 99 (97%); Published 340 (93%)	Grey 2 (2%); Published 9 (2%)	Published 9 (2%)	Grey 1 (1%); Published 3 (1%)	Published 3 (1%)	Japanese: Published 1

Type of grey literature

(Table 4) 
 For all five included studies the most common type of grey literature were abstracts (55%). Unpublished data was the second largest type of grey literature (30%). The definition of what constituted unpublished data varied slightly since we used the definition used by the authors of the study; however, it generally included data from trial registers, file‐drawer data and data from individual trialists. Book chapters were the third largest type of grey literature (9%), with unpublished reports, pharmaceutical company data, in press publications, letters and theses making up the small remainder.

4. Type of grey literature.

Study	Abstract	Unpublished	Book chapter	Report	Drug company	In press	Letter	Thesis	Other
Burdett 2003	17 (38%)	24 (53%)	3 (7%)						1 (2%) Non‐English language
Egger 2003	69 (45%)	57 (37%)	22 (14%)					5 (3%)
Fergusson 2000	4 (67%)	1 (17%)			1 (17%)
Hopewell 2004	36 (90%)	3 (8%)					1 (2%)
McAuley 2000	63 (62%)	18 (17%)	6 (6%)	5 (5%)	3 (3%)	3 (3%)	2 (2%)	2 (2%)

Methodological quality of the randomized trials included in the meta‐analyses

(Table 5) 
 Information on the methodological quality of the trials included in the meta‐analyses were only available for two of the five included studies (Egger 2003; Hopewell 2004). The study by Egger 2003 found that the quality of reporting of allocation concealment and blinding was better for published trials than for grey trials. In this study the quality of allocation concealment and blinding assessment was only assessed in the subset of trials which were included in Cochrane reviews and it was sometimes unclear or had not been assessed by the authors of Cochrane reviews. In contrast the study by Hopewell 2004 found no difference in the quality of reporting of allocation concealment or generation of the allocation sequence between published and grey trials; this information was often unclear in both groups.

5. Methodological quality of the randomized trials included in the meta‐analyses.

Study	Concealment YES	Concealment NO	Concealment UNCLEAR	Generation YES	Generation NO	Generation UNCLEAR	Blinding YES	Blinding NO	Blinding UNCLEAR
Egger 2003	Grey 26/77 (34%); Published 138/339 (41%)	Grey 51/77 (66%); Published 201/339 (59%)		N/A	N/A	N/A	Grey 32/71 (45%); Published 227/345 (66%)	Grey 39/71 (55%); Published 118/345 (34%)
Hopewell 2004	Grey 10/35 (29%); Published 46/175 (26%)		Grey 25/35 (71%); Published 129/175 (74%)	Grey 7/33 (21%); Published 38/162 (23%)	Grey 0/33 (0%); Published 8/162 (5%)	Grey 26/33 (79%); Published 116/162 (72%)	Grey 10/19 (53%); Published 48/117 (41%)	Grey 5/19 (26%); Published 40/117 (34%)	Grey 4/19 (21%); Published 29/117 (25%)

Outcome measures for grey and published trials

(Table 6) 
 All five included studies (Burdett 2003; Egger 2003; Fergusson 2000; Hopewell 2004; McAuley 2000) found that published trials showed an overall greater treatment effect than grey trials. This difference was statistically significant in one of the five studies (McAuley 2000). Data could be combined formally for three of the five studies (Egger 2003; Hopewell 2004; McAuley 2000), these showed that on average published trials showed a 9% greater treatment effect than grey trials (ratio of odds ratios for grey versus published trials 1.09; 95% CI 1.03‐1.16). The study by Burdett 2003 included individual patient data meta‐analyses and showed that when published trials were included in the meta‐analyses they produced a 4% greater treatment effect when compared to all trials (pooled hazard ratios 1.08; 95% CI 1.03‐1.11 for published data compared to 1.04; 95% CI 1.01‐1.08 for all data).

6. Summary outcome measures for grey and published trials.

Study	Summary outcome	Conclusion
Burdett 2003	Pooled hazard ratios 1.04 (95% CI 1.01‐1.08) for all trials Pooled hazard ratios 1.08 (95% CI 1.03‐1.11) for published trials	Published trials showed a greater treatment effect in favour of the experimental intervention; this was not statistically significant.
Egger 2003	Ratio of odds ratios (grey versus published) 1.07 (95% CI 0.98‐1.15)	Published trials showed a greater treatment effect in favour of the experimental intervention; this was not statistically significant.
Fergusson 2000	Estimates of treatment effect were not combined	Published trials showed a greater treatment effect in favour of the experimental intervention; this was not statistically significant.
Hopewell 2004	Ratio of odds ratios (grey versus published) 1.05 (95% CI 0.83‐1.34)	Published trials showed a greater treatment effect in favour of the experimental intervention; this was not statistically significant.
McAuley 2000	Ratio of odds ratio (grey versus published) 1.15 (95% CI 1.04‐1.28)	Published trials showed a greater treatment effect in favour of the experimental intervention; this was statistically significant.

Only one included study (McAuley 2000) assessed the type of grey literature and its impact on the overall results of the meta‐analyses. They found that published trials showed an even greater treatment effect than grey trials if abstracts were excluded from the meta‐analyses. The ratio of odds ratios for all grey trials versus published trials was 1.15 (95% CI 1.04‐1.28) compared to 1.33 (95% CI 1.10‐1.60) for grey trials, excluding abstracts, versus published trials.

Discussion

This systematic review shows that published trials tend to be larger and show an overall greater treatment effect than trials found only in the grey literature. One study also showed that published trials are more likely to have statistically significant results compared to grey trials. Therefore excluding grey trials from a systematic review and or meta‐analysis may artificially inflate its results and conclusions. The findings of this review may have particular implications in meta‐analyses containing only a few trials where the impact of excluding trials found in the grey literature has the greatest potential to introduce bias. Interestingly, the study by Egger 2003, which contained the greatest number of meta‐analyses, specifically excluded meta‐analyses containing only a small number of trials. A recent assessment of systematic reviews published in The Cochrane Library found that the average number of trials included in a typical Cochrane review was six (Mallett 2002). If the Egger 2003 study had included meta‐analyses containing only a small number of trials, it is anticipated that the difference in treatment effect, between published and grey trials, may have been greater. There is some evidence to show that in some areas of health care the effect of excluding grey trials from a meta‐analysis can result in greater differences in treatment effect than in other areas of health care (Egger 2003). However, it is not until such trials have been identified and included in a systematic review that their effect on the overall results and therefore conclusion of the systematic review can be fully analysed.

The methodological quality of grey and published trials may also have an impact on the overall results of a systematic review. Only two of the included studies assessed the methodological quality of the trials included in the meta‐analyses. One of these two studies (Egger 2003) found that allocation concealment and blinding of outcome assessor was better in published trials than trials found in the grey literature. However, in this study the methods of allocation concealment and blinding were only assessed using data from the subset of meta‐analyses that were Cochrane reviews and sometimes the necessary information was unclear or had not been assessed by the authors of the original Cochrane review. The other study (Hopewell 2004), again assessing a subset of Cochrane reviews, found no difference in the quality of reporting of allocation concealment or generation of the allocation sequence between published and grey trials, as this information was often unclear in both groups.

A recent study by Middleton 2004 documented how the methods of allocation concealment were described in a sample of Cochrane reviews. Middleton assessed 101 reviews (including 984 studies) published in Issue 1 2003 of The Cochrane Library. She found that 39% (388/984) of studies were miscoded and, for 256 of these 388 studies (26% overall), reviewers failed to make any comment on the method of allocation concealment. Common reasons for miscoding were confusion of allocation concealment with the randomization process in general, and confusion with the generation of the schedule and blinding of the intervention after randomization. This highlights possible problems of using the findings of the original reviewers to assess quality; in particular the problems of using the allocation concealment score assigned by authors of Cochrane reviews. It may point to a need for a fuller and more independent assessment of trial quality.

The definition of grey literature varied slightly across the five included studies, however, by far the most common type of grey literature in each of the included studies were trials reported in conference abstracts. Only one of the included studies (McAuley 2000) assessed the impact of the inclusion and exclusion of different types of grey literature on the overall results of the meta‐analyses. The study by McAuley and colleagues found that the exclusion of abstracts from the meta‐analyses showed that published trials had an even greater treatment effect than grey trials.

One of the potential problems of trying to identify and include trials reported in conference abstracts in a systematic review is the limited amount of information reported about that trial (Hopewell 2005). It has been argued by some that trials reported in abstracts should not be included in meta‐analyses as the data reported is often preliminary and may not be representative of the final results of the trial (Cook 1993). However, evidence suggests that just over half (56%) of all trials reported as abstracts and presented at conferences are subsequently published in full and that it take on average three years for a trial reported in an abstract and presented at a scientific meeting to be published in full (Scherer 2007). This means that a large number of trials presented at scientific meetings will never be published in full. Therefore, where information about a trial, its methodological quality and or its results, is either incomplete or missing it is important that systematic reviewers contact the trialist for additional information. This philosophy should apply to all trials irrespective of their final publication status.

Authors' conclusions

Implication for methodological research.

It is possible that a trial or meta‐analysis may have been duplicated across the five included studies and therefore analysed more than once. An "individual trial data" meta‐analysis (analogous to an individual patient data meta‐analysis) would help overcome this problem and, more importantly, would allow greater exploration of issues such as the quality of individual trials and the impact of different types of grey literature on the overall results of the meta‐analyses.

All the studies included in this review compared binary outcome measures. As far as we are aware no study has yet assessed the impact of inclusion and exclusion of grey literature in meta‐analyses of continuous outcomes.

What's new

Date	Event	Description
27 December 2007	Amended	Converted to new review format.

History

Protocol first published: Issue 2, 2002
 Review first published: Issue 4, 2002

Date	Event	Description
20 February 2007	New citation required and conclusions have changed	Substantive amendment

Data and analyses

Comparison 1. Grey versus published trials.

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Estimates of treatment effect	3		ROR (Fixed, 95% CI)	Subtotals only
1.1 Ratio of odds ratios	3		ROR (Fixed, 95% CI)	1.09 [1.03, 1.16]

1.1. Analysis.

Comparison 1 Grey versus published trials, Outcome 1 Estimates of treatment effect.

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Burdett 2003.

Methods	Multiple individual patient data meta‐analyses. Trials were identified by electronic searches, handsearching, trial registers and by contacting trialists.
Data	11 meta‐analyses of cancer treatments, containing 120 RCTs and 18,377 participants.
Comparisons	Inclusion and exclusion of grey literature on pooled effect estimate of the meta‐analyses. Grey literature was defined as unpublished data, abstracts, book chapters and non‐English language.
Outcomes	Published trials showed a greater treatment effect than grey trials; this difference was not statistically significant. Pooled hazard ratios 1.08 (95% CI 1.03‐1.11) for published trials versus 1.04 (95% CI 1.01‐1.08) for all trials.
Notes

Egger 2003.

Methods	Multiple meta‐analyses were identified by handsearching 8 high‐impact medical journals (1994‐1998), searching CDSR (issue 1, 1998), DARE (1994‐1998) and handsearching HTA reports. Meta‐analyses were eligible if they combined binary outcomes of at least 5 trials and had comprehensive literature searches.
Data	60 meta‐analyses from a number of different medical specialties, containing 783 RCTs and 167,733 participants.
Comparisons	Inclusion and exclusion of grey literature on pooled effect estimate of the meta‐analyses. Grey literature was defined as conference abstracts, books, theses, file drawer data and material from trial registers.
Outcomes	Published trials showed a greater treatment effect; this difference was not statistically significant. Ratio of odds ratio 1.07 (95% CI 0.98‐1.15) for grey trials versus published trials.
Notes

Fergusson 2000.

Methods	Multiple meta‐analyses identified from a project undertaken by the International Study of Peri‐operative Transfusion. Trials were identified by searching MEDLINE and EMBASE (1966‐1996).
Data	10 meta‐analyses of methods to reduce the need for peri‐operative transfusion, containing 114 trials and 11,142 participants.
Comparisons	Inclusion and exclusion of grey literature on pooled effect estimate of the meta‐analyses. Grey literature was defined as abstracts, letters and conference proceedings.
Outcomes	Published trials showed a greater treatment effect than grey trials; this difference was not statistically significant.
Notes

Hopewell 2004.

Methods	Multiple meta‐analyses identified from searching CDSR (issue 3 2003).
Data	17 meta‐analyses in cancer, containing 264 RCTs and 356,544 participants.
Comparisons	Inclusion and exclusion of grey literature on pooled effect estimate of the meta‐analyses. Grey literature was defined as conference abstracts, letters, books, government and pharmaceutical reports, theses, "file drawer" data and personal correspondence.
Outcomes	Published trials showed a greater treatment effect; this difference was not statistically significant. Ratio of odds ratio 1.05 (95% CI 0.83‐1.33) for grey trials versus published trials.
Notes

McAuley 2000.

Methods	Multiple meta‐analyses were selected at random from an existing database of meta‐analyses established by searching MEDLINE (1966‐1995).
Data	41 meta‐analyses from a number of different medical specialties, containing 467 RCTs and 217,427 participants.
Comparisons	Inclusion and exclusion of grey literature on pooled effect estimate of the meta‐analyses. Grey literature was defined as abstracts, unpublished studies, conference proceedings, theses, book chapters and company reports.
Outcomes	Published trials showed a greater treatment effect; this difference was statistically significant. Ratio of odds ratio 1.15 (95 % CI 1.04‐1.28) for grey versus published trials. Ratio of odds ratio 1.02 (95% CI 0.91‐1.14) for abstracts versus published trials. Ratio of odds ratio 1.33 (95 % CI 1.10‐1.60) for grey (excluding abstracts) versus published trials.
Notes

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Bhandari 2000	This was an individual systematic review (meta‐analysis) where the difference between the treatment effect of grey and published trials was explored in a sensitivity analysis.
Horn 2000	This was individual systematic review (meta‐analysis) where the difference between the treatment effect of grey and published trials was explored in a sensitivity analysis.
Jeng 1995	This was an individual systematic review (meta‐analysis) where the difference between the treatment effect of grey and published trials was explored in a sensitivity analysis.
Man‐Son‐Hing 1998	This was an individual systematic review (meta‐analysis) where the difference between the treatment effect of grey and published trials was explored in a sensitivity analysis.

Contributions of authors

SH and SM conducted the searches, data extraction and data analysis. SH, SM, MC and ME were involved in writing and commenting on the completed review.

Sources of support

Internal sources

• UK Cochrane Centre, NHS Research & Development Programme, UK.

• Australasian Cochrane Centre, Australian Goverment Department of Health and Ageing, Australia.

• Department of Social and Preventive Medicine, University of Bern, Switzerland.

External sources

• No sources of support supplied

Declarations of interest

Sally Hopewell is the author of one of the studies included in this review and Matthias Egger is author of another of the included studies, which was published as a NHS Health Technology Assessment review.

Unchanged