Overlapping meta-analyses on the same topic: survey of published studies

Abstract

Objective To assess how common it is to have multiple overlapping meta-analyses of randomized trials published on the same topic.

Design Survey of published meta-analyses.

Data sources PubMed.

Study selection and methods Meta-analyses published in 2010 were identified, and 5% of them were randomly selected. We further selected those that included randomized trials and examined effectiveness of any medical intervention. For eligible meta-analyses, we searched for other meta-analyses on the same topic (covering the same comparisons, indications/settings, and outcomes or overlapping subsets of them) published until February 2013.

Results Of 73 eligible meta-analyses published in 2010, 49 (67%) had at least one other overlapping meta-analysis (median two meta-analyses per topic, interquartile range 1-4, maximum 13). In 17 topics at least one author was involved in at least two of the overlapping meta-analyses. No characteristics of the index meta-analyses were associated with the potential for overlapping meta-analyses. Among pairs of overlapping meta-analyses in 20 randomly selected topics, 13 of the more recent meta-analyses did not include any additional outcomes. In three of the four topics with eight or more published meta-analyses, many meta-analyses examined only a subset of the eligible interventions or indications/settings covered by the index meta-analysis. Conversely, for statins in the prevention of atrial fibrillation after cardiac surgery, 11 meta-analyses were published with similar eligibility criteria for interventions and setting: there was still variability on which studies were included, but the results were always similar or even identical across meta-analyses.

Conclusions While some independent replication of meta-analyses by different teams is possibly useful, the overall picture suggests that there is a waste of efforts with many topics covered by multiple overlapping meta-analyses.

Introduction

Systematic reviews and meta-analyses are often considered to be the highest level in the hierarchy of evidence,^{1 2} and justifiably these designs have become increasingly popular.³ Meta-analysis in particular requires some extra quantitative skills in synthesizing data with sophisticated statistical methods. The ready availability of multiple meta-analysis software that can be used even by minimally trained people, however, allows such analyses to be conducted on a massive scale. A search on 29 January 2013 with the “meta-analysis” tag for “type of publication” in PubMed showed a 17-fold acceleration in the annual number of meta-analyses published between 1991 (n=334) and 2011 (n=5861). Meta-analyses of randomized trials are a large share of this literature. It is unknown whether these meta-analyses are totally different and done on different topics; represent serial updates of the same topic done by the same team of authors who want to bring their data up to date; or are on some topics that attract attention of multiple different teams of systematic reviewers who independently perform and publish their meta-analyses. Multiple independent meta-analyses on the same topic have been identified in the past for diverse topics such as gastric ulcer prophylaxis, dosing of aminoglycosides, selective decontamination of the digestive tract, orthopedic procedures, and wound healing, among others.^{4 5 6 7 8 9 10 11 12 13 14} Usually multiple reviews on the same topic would find the same results, but discrepancies can ensue either at the level of the objective results or, more commonly, at the level of their interpretation,¹² and potential discrepancies can cause endless debates. Some meta-analyses might need updating when new evidence emerges (especially if this evidence is likely to change the conclusions),¹⁵ and independent replication by different teams is also welcome as in any field of research. It would be concerning, however, if many overlapping meta-analyses were published on the same topic by different teams. This could cause confusion and duplication in the vast data space of meta-analyses¹⁶ and would suggest potentially wasted effort. Indeed, there is some concern that such redundancy and inefficiency might be a real problem, and efforts are underway to encourage the registration of meta-analysis protocols, as in the PROSPERO initiative.¹⁷

We assessed how common it is for a published meta-analysis of randomized trials to have other published overlapping meta-analyses and investigated the characteristics of these potential redundancies.

Methods

Index meta-analyses: search and eligibility criteria

For the selection of index meta-analyses we searched PubMed for meta-analyses of randomized controlled trials published in 2010 using the search terms “randomi* AND 2010[dp] AND Meta-Analysis[ptyp]” without language restrictions. With the sample 5 command in Stata software version 11.0 (StataCorp, College Station, TX, USA), we selected a 5% random sample of the resulting items to assess for eligibility at the abstract level. Eligible for inclusion were meta-analyses of randomized trials evaluating the effectiveness of diagnostic, preventive, or therapeutic interventions for any condition. Meta-analyses including both randomized trials and observational studies were also eligible. We excluded studies pertaining to safety of interventions, prognostic associations, those with observational studies only, qualitative reviews without meta-analysis, meta-analyses pertaining to dentistry, and diagnostic accuracy meta-analyses.

Additional meta-analyses on same topic: search and eligibility criteria

Using individualized search algorithms for the topic described in each index meta-analysis we searched PubMed for overlapping meta-analyses of randomized controlled trials indexed until February 2013. Meta-analyses including both randomized trials and observational studies were also eligible. Potential redundancy was defined as overlap in terms of comparisons of interventions, type of populations, and outcomes between meta-analyses. There had to be at least one overlapping analysis (same comparison, type of population/indication, and outcome) for them to be considered as overlapping. When the index study was a Cochrane review, its updates were not considered as overlapping. When an original Cochrane review and its update were identified as overlapping, we considered only the update.

Data extraction

For each index meta-analysis, we extracted information on the intervention(s) tested, the diseases/indications assessed, the number of additional meta-analyses, and whether an author was involved in more than one meta-analysis on the same topic. For each of the index meta-analyses and the additional ones, we also noted whether it was a Cochrane review and the country of affiliation of the first author, as listed in PubMed.

We randomly selected 20 topics with overlapping meta-analyses publications and a pair of meta-analyses from each (40 articles total) and examined the extent of overlap in the outcomes and whether the more recently published meta-analysis (the one with the higher PubMed ID number) had any additional outcomes evaluated with quantitative synthesis compared with the older meta-analysis. When there were more than two overlapping meta-analyses in a topic, the pair of meta-analyses was selected randomly.

Evaluation of overlapping meta-analyses

We extracted characteristics of the index and additional meta-analyses including the publication venue (Cochrane versus other), type of condition (cardiovascular diseases, gastroenterology, neurology, hematology/oncology, psychiatry, pulmonary diseases, surgery, obstetrics and gynecology, other), type of intervention (drugs/biologics, surgical/interventional, behavioral/psychological, physical therapy, other), and countries of origin. Index meta-analyses with and without additional meta-analyses were compared by exact test on the publication venues, types of conditions, types of interventions, and countries of origin.

Topics with large number of overlapping meta-analyses

For topics for which we could identify eight or more meta-analyses, we also evaluated whether the overlapping meta-analyses differed in their eligibility criteria regarding the breadth of interventions being assessed and the eligible settings.

Use of statins for atrial fibrillation after cardiac surgery had the largest number of overlapping meta-analyses with similar eligibility criteria for type of intervention and indication/setting, and we analyzed each of the published meta-analyses in more depth. We captured information on and compared these meta-analyses regarding their publication date, date of last literature search, type of studies considered eligible (randomized trials or also observational studies), summary metric used for the effect on the main outcome (postoperative atrial fibrillation), summary treatment effect and 95% confidence interval, and which studies had been included in the calculations. For each study that had been included in the calculations of at least one meta-analysis, we evaluated whether it had been included in each meta-analysis; we also determined whether each non-inclusion was because of ineligibility of the design type, publication after the search date of the meta-analysis, or neither of these reasons. Finally, we noted whether the meta-analyses published in 2012 cited those published in previous years.

Results

Index meta-analyses

We screened 95 meta-analyses published in 2010 and excluded 22 that did not meet eligibility criteria: safety of interventions only (n=7), prognostic associations (n=5), observational studies only (n=4), qualitative reviews without meta-analysis (n=3), dentistry topics (n=2), diagnostic accuracy meta-analysis (n=1). After these exclusions, 73 index meta-analyses were eligible (see appendix). Twenty seven (37%) of those were Cochrane reviews, and the rest were meta-analyses published in peer reviewed journals. Cardiovascular diseases was the most common topic (19%), followed by gastroenterology (12%) and neurology (11%). The interventions evaluated in our sample of meta-analyses most commonly pertained to drugs or other biologics (67%). Table 1 gives details of characteristics.

Table 1.

 Characteristics of eligible index meta-analyses in study

Characteristic	No (%)
Cochrane review:
Yes	27 (37)
No	46 (63)
Type of condition:
Cardiovascular disease	14 (19)
Gastroenterology	9 (12)
Neurology	8 (11)
Hematology/oncology	7 (10)
Psychiatry	6 (8)
Pulmonary disease	5 (7)
Surgery	5 (7)
Obstetrics and gynecology	5 (7)
Other	14 (19)
Type of intervention:
Drugs/biologics	49 (67)
Surgical/interventional	12 (17)
Behavioral/psychological	5 (7)
Physical therapy	4 (5)
Other	3 (4)
Country:
United Kingdom	18 (25)
United States	11 (15)
Canada	8 (11)
China	7 (9)
Australia	5 (7)
Other	24 (33)

Topics with overlapping meta-analyses

There were 138 overlapping meta-analyses (see appendix), corresponding to 49 (67%) of the 73 index meta-analyses. Among these 49 topics, the median number of overlapping meta-analyses per topic was two (interquartile range 1-4), including the index meta-analysis.

Table 2 lists the outcomes evaluated in pairs of overlapping meta-analyses in 20 randomly selected topics. As shown, most of the outcomes were evaluated with quantitative synthesis in both articles of each pair. When we examined only the 27 outcomes that were stated to be the primary ones or the unique outcome assessed in a meta-analysis, 22 of them (81%) were evaluated by both articles of each pair. In 13 of the 20 pairs, the more recent meta-analysis did not include any additional clearly different outcomes beyond those already assessed in the older meta-analysis. In 17 of the 20 pairs, the more recent meta-analysis did not include any additional primary outcome that had not been evaluated in the older meta-analysis.

Table 2.

 Evaluated outcomes in pairs of overlapping meta-analyses in 20 randomly selected topics

Topic and PubMed ID	Outcomes evaluated
Open v endovascular aneurysm repair:
19836274	Operative mortality; mortality related to abdominal aortic aneurysm; all cause mortality
18658009	Operative mortality*
Isometric handgrip exercise:
20009767	Change in SBP; change in DBP
21896934	Change in SBP*; change in DBP*; anthropometrics; exercise tolerance; blood lipids; glucose
Dexamethasone in bacterial meningitis:
19475753	Mortality*; hearing impairment or other neurological sequelae; adverse effects related to study regimens
20138011	Mortality; severe neurological sequelae; hearing loss
Peginterferon alpha-2a v alpha-2b in chronic hepatitis C:
20187106	Sustained virologic response*; discontinuation for adverse events*; all cause mortality*
20974315	Sustained virologic response*; discontinuation for adverse events; all cause mortality
Pregabalin in fibromyalgia:
20418173	Pain; fatigue; sleep; depressed mood; health related quality of life
19588419†	Pain; adverse effects
Tocilizumab in rheumatoid arthritis:
21097801	ACR50*
20614469	ACR50*; safety*; ACR70; ACR20; change in DAS28 score; proportion achieving good state; quality of life
Antibiotics for prevention of growth of abdominal aortic aneurysm:
20675312	Growth rate*
18365027	Growth rate*
Timing of coronary angiography in NSTEACS:
20709722	Death; MI; major bleeding; recurrent ischemia; repeat intervention; stroke; hospital stay length
21872193	Death; MI; major bleeding; recurrent ischemia; repeat intervention
Cyclophosphamide in interstitial lung disease:
20802426	FVC improvement*
18937831	FVC improvement*; DLCO improvement*
Prophylactic antibiotics in laparoscopic cholecystectomy:
21154360	Surgical site infection*; extra-abdominal infection*
21443433	Surgical site infection; extra-abdominal infection; major infection; all infections; positive bile cultures; length of hospital stay
Aspirin in diabetes:
21191260	All cause mortality; cardiovascular mortality; major adverse cardiovascular events; MI; stroke; major bleeding events
19897665	All cause mortality, fatal or non-fatal MI or stroke; any bleeding; gastrointestinal symptoms; incidence of cancer
Shortened treatment duration of peginterferon and ribavirin in HCV1 patients with rapid virologic response:
19931204	Sustained virologic response*; end of treatment virologic response; relapse rates
21674553	Sustained virologic response*; relapse rates; safety (treatment discontinuation)
Neuraxial anesthesia for lower limb revascularization:
20091615	Death*; MI*; postoperative amputation*; pneumonia
19488536	Death*; MI*; postoperative amputation*; pneumonia
Cataract surgery for fall prevention:
20117700	Vision improvement*; falls
19370674	Falls*
Statins for primary prevention in women:
18793814	Death; cardiovascular events
22300691	Death; cardiovascular events
Statins for prevention of atrial fibrillation after cardiac surgery:
19698856	Atrial fibrillation*
19559266	Atrial fibrillation*
Drug eluting stents for coronary chronic total occlusions:
20549695	Restenosis; reocclusion; death; MI; TVR; TLR; stent thrombosis
21419488	MACE*; death; MI; TVR; TLR; TVF; restenosis; reocclusion; minimal lumen diameter; late lumen loss
Antibiotics in cirrhosis with upper gastrointestinal bleed:
10347104	Infection; bacteremia and/or spontaneous bacterial peritonitis; spontaneous bacterial peritonitis; death
21707680	Death*; death from infection*; infection*; dropouts; rebleeding; length of hospitalization
Oral diflunisal for acute postoperative pain:
20393958	At least 50% pain relief*; use of rescue medication; adverse events
21901726	At least 50% pain relief*; use of rescue medication; adverse events
Corticosteroids in septic shock:
19489712	Death; shock reversal; superinfection
15289273	Death*; in hospital death; shock reversal; adverse events (gastroduodenal bleeding, superinfections, hyperglycemia, and other adverse effects)

SBP=systolic blood pressure; DBP=diastolic blood pressure; ACR(x)=x% improvement according to American College of Rheumatology criteria; NSTEACS=non-ST elevation acute coronary syndrome; DAS28=disease activity score; MI=myocardial infarction; FVC=forced vital capacity; DLCO=diffusing capacity for carbon monoxide; MACE=major adverse cardiac events; TVR=target vessel revascularization; TLR=target lesion revascularization; TVF=target vessel failure.

*Primary or unique outcome.

†Acute and chronic pain including fibromyalgia.

Among the 187 overlapping meta-analyses (49 index meta-analyses published in 2010 plus their 138 overlapping counterparts), the United Kingdom was the most common country of origin (n=47), followed by the United States (n=38) and Canada (n=19). Of the 138 overlapping meta-analyses, 17 were published before 2005, 60 in 2005-09, and 61 in or after 2010.

For 17 (23%) topics at least one author was involved in at least two of the overlapping meta-analyses. In seven of the 17 topics these overlapping meta-analyses were presented as updates. Another seven pairs of meta-analyses had partially overlapping interventions, comparators, and/or indications/settings. One was a pair of a meta-analysis of published literature versus one with patient level data contributed by primary study investigators. One pair pertained to a meta-analysis of only randomized controlled trials and a meta-analysis of both randomized controlled trials and observational studies. Finally, in one pair it was unclear whether there were any differentiating characteristics.

Comparison of index meta-analyses with v without additional meta-analyses

There was no significant association between the evaluated characteristics of index meta-analyses and the potential for redundancy, including venue of publication, type of condition, type of intervention, and country of origin (table 3).

Table 3.

 Association between characteristics of index meta-analysis and possibility for redundancy in a topic

Characteristic	Topics with redundancy/total No of topics	Fisher’s exact P value
Cochrane review:
Yes	15/27	0.09
No	34/46
Type of condition:
Cardiovascular diseases	12/14	0.33
Gastroenterology	6/9
Neurology	7/8
Hematology/oncology	3/7
Psychiatry	4/6
Pulmonary diseases	4/5
Surgery	3/5
Obstetrics and gynecology	2/5
Other	8/14
Type of intervention:
Drugs and biologics	33/49	0.42
Surgical and interventional	10/12
Behavioral/psychological	2/5
Physical therapy	3/5
Other	1/2
Country of origin:
United Kingdom	12/18	0.71
United States	5/11
Canada	7/8
China	5/7
Australia	3/5
Other	17/24

CDSR=Cochrane Database of Systematic Reviews.

Topics with many overlapping meta-analyses

There were four topics with eight or more overlapping meta-analyses (table 4). For statins for atrial fibrillation after cardiac surgery, all meta-analyses considered the same interventions (all available statins were eligible) and the same setting (cardiac surgery), except for one early meta-analysis that also included a single trial in a different setting (acute coronary syndrome). Conversely, for three other topics (chemoprevention for colorectal neoplasia, pharmacotherapy for fibromyalgia, antiepileptics for refractory epilepsy), the index meta-analysis had typically considered several drugs or interventions, while some of the other meta-analyses on the same topic considered only one of these interventions. For fibromyalgia there were also two meta-analyses that considered a wider range of causes of pain (besides fibromyalgia) than the index meta-analysis.

Table 4.

 Topics with eight or more overlapping meta-analyses*

Topic	No of meta-analyses	Journal (No of studies included)	Comments
Chemoprevention of colorectal neoplasia	13	1 Nutr Hosp 2012 (n=7); 2 Clin Ther 2010 (n=6); 3 Health Technol Assess 2010 (n=44)†; 4 Pak J Pharm Sci 2010 (n=4); 5 Aliment Pharmacol Ther 2010 (n=6); 6 Med Oncol (2010 (n=5); 7 J Natl Cancer Inst 2009 (n=4); 8 Colorectal Dis 2009 (n=3); 9 CDSR 2008 (n=2); 10 Zhonghua Yi Xue Za Zhi 2007 (n=6); 11 Aliment Pharmacol Ther 2006 (n=8); 12 Am J Gastroenterol 2005 (n=3); 13 Lancet 2004 (n=14)	Index meta-analysis evaluated NSAIDs, folic acid, calcium, vitamin D, and antioxidants including vitamin E. Other meta-analyses evaluated one of these interventions, thus they all partially overlap with index meta-analysis
Pharmacotherapy in fibromyalgia	12	1 CNS Drugs 2012 (n=35); 2 CDSR 2012 (n=5); 3 Rheumatology 2011 (n=19); 4 J Clin Pharm Ther 2010 (n=3); 5 BMC Musculoskelet Disord 2010 (n=4); 6 J Pain 2010 (n=8)†; 7 Rheumatology 2010 (n=4); 8 CDSR 2009 (n=3); 9 CDSR 2009 (n=5); 10 Pain 2009 (n=5); 11 Clin Rheumatol 2009 (n=5); 12 J Womens Health 2007 (n=2)	Index meta-analysis evaluated duloxetine, milnacipran, and pregabalin. Meta-analyses 1, 3, and 6 were published by same team and presented as updates. Meta-analysis 2 evaluated milnacipran only. Meta-analyses 4, 5, 7, and 10 evaluated pregabalin. 11 and 12 evaluated duloxetine. 8 and 9 are meta-analyses of duloxetine and pregabalin, respectively, for different pain syndromes, including fibromyalgia
Statins for prevention of atrial fibrillation after cardiac surgery	11	1 Br J Clin Pharmacol 2012 (n=8); 2 CDSR 2012 (n=8); 3 Arch Surg 2012 (n=9); 4 Ann Med 2011 (n=8); 5 J Am Coll Cardiol 2010 (n=6); 6 J Thorac Cardiovasc Surg 2010 (n=6)†; 7 Tex Heart Inst J 2009 (n=10); 8 J Thorac Cardiovasc Surg 2009 (n=13); 9 Ann Thorac Surg 2009 (n=2); 10 Eur Heart J 2008 (n=7); 11 J Am Coll Cardiol 2008 (n=3)	All meta-analyses considered any statin and any dosing. Studies 1 and 3 included also non-surgery trials and non-cardiac surgery trials, respectively (in separate analyses). Meta-analysis 11 analyzed cardiac surgery trials along with trial pertaining to acute coronary syndromes For additional details see table 5
Antiepileptics for refractory epilepsy	8	1 Epilepsia 2011 (n=62); 2 Can J Neurol Sci 2011 (n=8); 3 Epilepsia 2010 (n=54)†; 4 CDSR 2005 (n=4); 5 Epilepsia 2003 (n=3); 6 Epilepsia 1997 (n=29); 7 Epilepsia 1997 (n=5); 8 BMJ 1996 (n=20)	Studies 1, 3, 6, and 8 evaluated several different antiepileptics each, whereas 2 and 5 evaluated levetiracetam, 4 evaluated zonisamide, and 7 evaluated topiramate

CDSR=Cochrane Database of Systematic Reviews.

*In some topics not all meta-analyses are overlapping with each other, but they overlap to a lesser or greater degree with the index meta-analysis.

†Index meta-analysis.

Evaluation of 11 meta-analyses on statins for atrial fibrillation after cardiac surgery

Table 5 provides detailed information on overlapping meta-analyses of statins for the prevention of atrial fibrillation after cardiac surgery. We identified 11 overlapping meta-analyses, which were published over 57 months with a relatively steady appearance of new meta-analyses every few months. Their search dates differed over a span of 46 months. Eight of the 11 included only randomized trials, while three also included observational studies. With the exception of the first meta-analysis, all meta-analyses consistently showed a significant benefit of statins on the occurrence of postoperative atrial fibrillation, and the treatment effect was consistently large with summary risk ratios ranging between 0.54 and 0.57 and summary odds ratios ranging between 0.40 and 0.78.

Table 5.

 Mapping of potential redundancy in 11 meta-analyses on use of statins for prevention of atrial fibrillation after cardiac surgery

	1	2	3	4	5	6	7	8	9	10	11
Month/year published	2/2008	6/2008	7/2009	9/2009	12/2009*	8/2010	9/2010	2/2011	2/2012	4/2012	11/2012
Month/year of last search	6/2007	2/2008	5/2008	7/2008	12/2008*	8/2009	2/2010	8/2010	4/2011	5/2010	12/2010
Eligible studies	RCT†	RCT+O	RCT	RCT+O	RCT+O	RCT	RCT	RCT	RCT	RCT	RCT
Effect size (95% CI)	0.60‡ (0.27 to 1.37)	0.67‡ (0.51 to 0.88)	0.57‡ (0.42 to 0.78)	0.78‡ (0.67 to 0.90)	0.68‡ (0.59 to 0.79)	0.57§ (0.45 to 0.72)	0.54§ (0.43 to 0.68)	0.40‡ (0.29 to 0.55)	0.56§ (0.45 to 0.69)	0.40‡ (0.29 to 0.55)	0.40‡ (0.29 to 0.55)
Trials included in meta-analysis
Schwartz 2004 (RCT, ACS)	Yes	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Auer 2004 (RCT)	No	No	No	No	Yes	No	No	No	No	No	No
Pan 2004 (O)	No (D)	Yes	No (D)	Yes	Yes	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Chello 2006 (RCT)	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Patti 2006 (RCT)	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Marin 2006 (O)	No (D)	Yes	No (D)	Yes	Yes	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Mariscalco 2007 (O)	No (D, P)	Yes	No (D)	Yes	Yes	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Ozaydin 2007 (O)	No (D)	Yes	No (D)	Yes	Yes	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Powell 2007 (O)	No (D)	Yes	No (D)	Yes	No	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Thielmann 2007 (O)	No (D, P)	No	No (D)	Yes	No	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Lertsburapa 2008 (RCT)	No (P)	No (P)	Yes	Yes	Yes	No	No	No	No	No	No
Kourliouros 2008 (O)	No (D, P)	No (P)	No (D, P)	Yes	Yes	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Subramaniam 2008 (O)	No (D, P)	No (P)	No (D)	Yes	No	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Virani 2008 (O)	No (D, P)	No (P)	No (D)	Yes	No	No (D)	No (D)	No (D)	No (D)	No (D)	No (D)
Song 2008 (RCT)	No (P)	No (P)	No (P)	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Mannacio 2008 (RCT)	No (P)	No (P)	No (P)	No (P)	No	Yes	Yes	Yes	Yes	Yes	Yes
Caorsi 2008 (RCT)	No (P)	No (P)	No (P)	No	No	Yes	No	Yes	Yes	Yes	Yes
Tamayo 2008 (RCT)	No (P)	No (P)	No (P)	No	No	Yes	Yes	Yes	Yes	Yes	Yes
Ji 2009 or Sun 2009 (RCT)¶	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	Yes	Yes+	Yes	Yes+	Yes
Spadaccio 2010 (RCT)	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	No	Yes	No	Yes	Yes
Antoniades 2010 (RCT)	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	Yes	No (P)	No
Sun 2011 (RCT)	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	No (P)	Yes	No (P)	No (P)

RCT=randomized controlled trial; ACS=acute coronary syndromes; O=observational study; NA=not available; P=study published after search for eligible studies for that meta-analysis; D=observational study not included because meta-analysis considered only RCTs.

*Dates of publication and last search were not available for this meta-analysis, but given that it is a bi-monthly journal and issue number was 6 we assigned December as month of publication. We then considered that last search was performed one year before publication.

†Meta-analysis included in same analysis cardiac surgery and acute coronary syndrome trials.

‡Odds ratio.

§Risk ratio.

¶Two different publications reporting on same trial.

The main differences in included studies on statins were from eligibility criteria regarding observational studies and, to a lesser extent, the non-consideration of trials published after the search date of each meta-analysis (table 5). Nevertheless, no meta-analysis included all randomized trials available during their study period: two meta-analyses did not include one trial, four others missed two trials, and five missed three or more such trials. Three meta-analyses published in 2011-12 included exactly the same eight trials and derived an identical summary effect. The three meta-analyses published in 2012 cited four, two, and two of the eight previously published meta-analyses on the same topic.

Discussion

Our empirical evaluation shows that for two thirds of the meta-analyses published in 2010 there was at least one more additional meta-analysis on the same topic that did not represent an update by the same authors in most cases. Typically more recent meta-analyses did not evaluate important additional outcomes beyond those already assessed by an older one on the same topic. Perusal of topics with eight or more meta-analyses suggested that these often differed (but still overlapped) in their inclusion criteria regarding eligible interventions, settings, and types of studies. Thus it was common for some meta-analyses to cover subsets of the evidence covered by other meta-analyses on the same topic. In depth perusal of a topic with 11 meta-analyses with similar eligible interventions and similar eligible settings showed some variability regarding which studies were included, but the results were similar. Thus some of the observed overlap in our sample seems unnecessary and can reflect wasted efforts and inefficiency in the process of summarizing evidence.

Implications of redundancy

Some potential overlap of meta-analyses is justified or even of value on grounds of necessary updating and even independent replication. There is currently no firm consensus on how and when updating should occur, but several methods have been proposed and piloted.^{18 19 20 21 22} Prior surveys have shown that most updated reviews are Cochrane reviews—for example, nearly 38% of 125 Cochrane reviews indexed in PubMed in November 2004 were updates, while this was true for only 2.3% of 88 reviews published in journals in the same year.³ In our analysis, we specifically excluded past versions of Cochrane updates. Nearly all overlapping meta-analyses were done by entirely independent teams. While some can claim that they also serve an updating need, there is no continuity of effort by the same team. Assuming that each independent team has to start from scratch to set the protocol, perform searches, extract data, analyze results, and write the paper, it is likely that there is a lot of duplicated effort that would be avoided if the same team were responsible for the updating. This continuity would not prohibit some partial renewal and recycling in the authors involved.

Replication of systematic reviews by entirely independent teams, however, could be useful. Replication is useful in any scientific field. Examples of discordant meta-analyses on the same topic abound in the literature.^{4 5 6 7 8 9 10 11 12} They often reflect differences in eligibility criteria, types of studies selected, statistical methods, occasional errors, or even diverse subjective interpretation of otherwise similar results.^{4 5 6 7 8 9 10 11 12} The same issues also apply in meta-analyses of non-randomized studies^{23 24} and their subjective interpretation.²⁵ Most systematic reviews and meta-analyses to date are retrospective, and the data to be synthesized are already available to view informally or even scrutinize before a protocol is drafted. Therefore, replication efforts cannot be independent from each other. For many protocol decisions for a meta-analysis where different reviewer teams have acted differently, there is no clearly discernible correct versus incorrect choices. We worry that meta-analysts might sometimes try to make a case that their meta-analysis is different than others on the same topic to help publish their work. This can lead to choices on the eligibility criteria, outcomes, or methods that are not rational but are driven by the need to show some kind of novelty. Then the purpose of replication is not well served.

In our empirical evaluation, in most topics with a large number of meta-analyses, these meta-analyses differed on how broad their eligibility criteria were in terms of what interventions and settings to include. Some publications seemed to be slices of more comprehensive reviews. More inclusive publications that consider multiple, if not all, available treatment options for the same condition or even for multiple similar conditions can offer more complete pictures of the evidence and the available treatment options.²⁶ Most meta-analyses evaluate small fragments of the evidence on a clinical question of interest.²⁷ More inclusive designs, such as umbrella reviews²⁸ and networks,²⁹ might become more popular in the future. Even then, however, the problem of redundancy and inefficiency might remain—for example, there is already evidence that for some topics—such as anti-tumor necrosis factor (TNF) agents for rheumatoid arthritis—there are already multiple network meta-analyses published by independent teams.³⁰

The topic of statins for atrial fibrillation after cardiac surgery provides an example where redundancy is most clear. Despite some differences on whether observational studies should be included or not, all meta-analyses (with the exception of the first one, which was inconclusive) showed a large effect for the intervention. More recent meta-analyses might have included some more recently published trials, but their incremental value was uncertain. Three recent meta-analyses from different authors were even identical in the studies included and the obtained summary results. One wonders whether in some cases, newer meta-analyses actually build silently on other preceding meta-analyses on the same topic—for example, by using already extracted data and results. In our example of statins for postoperative atrial fibrillation, meta-analyses published in 2012 cited only two to four of the eight previous meta-analyses.

Limitations

Our work has some limitations. First, we did not try to determine for each meta-analysis whether it was truly unnecessary or not. This would have been a subjective decision, and there might be differences in opinion about the exact utility of so many overlapping meta-analyses. Their high prevalence, however, suggests that there is substantial inefficiency in the process. Second, overlap was not always absolute, and occasionally additional outcomes were assessed. Overlap for primary outcomes, however, was high and most of the newer meta-analyses did not study any more clearly different outcomes than those already covered by older ones. Some of the overlapping meta-analyses were slices of the evidence focusing on narrow outcomes. Understanding the real merits of interventions, however, requires the full picture about the outcomes they induce. Thus, meta-analyses with limited coverage of outcomes are suboptimal, and consideration of all core outcomes is preferable.^{31 32}

Conclusions

Despite these caveats, the high prevalence of overlapping meta-analyses on the same topic suggests that there is room for improving the efficiency of evidence synthesis worldwide. There are still many important topics for which no systematic review and meta-analysis has been performed.^{33 34} Conversely, for others there are already more than 10 published meta-analyses. Better coordination, communication between reviewers, and potentially registration of protocols for systematic reviews¹⁷ are options to consider. Previous meta-analyses on the same topic should also be properly acknowledged and placed into appropriate context versus a new overlapping effort. This should be part of standardized methods of meta-analysis and reporting and could be included as one of the PRISMA checklist items.³⁵

What is already known on this topic

• Systematic reviews and meta-analyses are popular study designs that are often considered to offer the highest level of evidence

• The number of meta-analyses published annually is increasing steadily, and the degree of redundancy among published meta-analyses is unknown

What this study adds

• Two thirds of published meta-analyses have at least one partially or completely overlapping meta-analysis that in most cases does not represent an update

Cite this as: BMJ 2013;347:f4501

Web Extra. Extra material supplied by the author

Appendix Details of index meta-analyses, search terms, and overlapping studies