Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study

Abstract

Objective To appraise multiple systematic reviews on the same clinical topic, focusing on predictors and correlates of quality of reporting of meta-analysis (QUOROM) scores.

Design Case study.

Setting Reviews providing at least individual quantitative estimates on role of acetylcysteine in the prevention of contrast associated nephropathy.

Data sources PubMed, the database of abstracts of reviews of effects, and the Cochrane database of systematic reviews (updated March 2005).

Main outcome measures Funding, compliance with the QUOROM checklist, scores on the Oxman and Guyatt quality index, and authors' recommendations.

Results 10 systematic reviews, published August 2003 to March 2005, were included. Nine pooled events despite heterogeneity and five recommended routine use of acetylcysteine, whereas the remaining studies called for further research. Compliance with the 18 items on the QUOROM checklist was relatively high (median 16, range 11 to 17), although shorter manuscripts had significantly lower scores (R = 0.73; P = 0.016). Reviewers who reported previous not for profit funding were more likely to score higher on the Oxman and Guyatt quality index. No association was found between QUOROM and Oxman and Guyatt scores (R = -0.06; P = 0.86), mainly because of greater emphasis of the Oxman and Guyatt scores on the appraisal of bias in selection and validity assessment (inadequate in five reviews).

Conclusions Multiple systematic reviews on the same clinical topic varied in quality of reporting and recommendations. Longer manuscripts and previous not for profit funding were associated with higher quality.

Introduction

Systematic reviews incorporating meta-analytic methods provide a unique means of searching, appraising, and summarising the results of several studies to enable informed decision making.¹,² Since 1976 a large number of such systematic reviews have been published, varying widely in quality and standards.^3-8 Anecdotal reports of multiple reviews focusing on the same clinical topic have differed in quality and methods used, leading to conflicting conclusions.⁷,⁹,¹⁰

In 1999 several authorities proposed the quality of reporting of meta-analysis (QUOROM) statement to improve and standardise reporting, as previously done with randomised controlled trials with the consolidated standards of reporting trials (CONSORT) statement.¹¹ The aim of the QUOROM checklist was to improve the standards of reporting, not strictly to avoid duplicated systematic reviews.

Since then many journal editors and reviewers, including those involved in the Cochrane Collaboration, have pursued compliance with the QUOROM checklist to provide sound and reproducible results.¹ The effect of the QUOROM guidelines on the design, conduct, and reporting of systematic reviews is, however, unclear.

Several randomised controlled trials have investigated the role of acetylcysteine in the prevention of contrast associated nephropathy, but with conflicting results. In light of this, a systematic review was carried out to provide more comprehensive and robust conclusions and reported in August 2003.¹² In the following months other systematic reviews on the same topic were published, with different findings.

We appraised the cluster of duplicate systematic reviews on the role of acetylcysteine to prevent contrast nephropathy published in peer reviewed journals, focusing on predictors and correlates of QUOROM quality scores. We explored the association between compliance with the QUOROM statement and characteristics of the manuscripts, such as length, number of trials and patients, and funding source.

Methods

We searched for systematic reviews in PubMed according to a defined strategy,¹³ and in the Cochrane database of systematic reviews and the database of abstracts of reviews of effects (updated March 2005). Key terms included “acetylcysteine”, “contrast”, “meta-analysis”, “nephropathy”, and “systematic review”.¹⁴ We included systematic reviews of randomised controlled trials that compared acetylcysteine with control for the prevention of contrast associated nephropathy, reviews published as complete reports, and reviews that reported quantitative estimates at least for individual studies. We excluded systematic reviews embedded in editorials, correspondence, or reports of randomised controlled trials and those published only in abstract form. We also excluded incompletely reported reviews (for example, those reported in an editorial or in the discussion section of another study type or reported as part of a controlled trial) to avoid limiting the chance of the article complying with the QUOROM statement and thus undermining the quality of reporting of such reviews. We applied no language restrictions.

Study abstraction and appraisal

We abstracted data on the journal in which the systematic reviews were published, dates of initial submission and publication, number of authors, article length in pages search strategies (including database searched language restrictions), study design (randomised and non-randomised controlled trials), number of trials and patients, journal impact factor,¹⁵ and whether estimates were pooled (with pertinent methods, summary estimates, and results). We also checked for use of funnel plots to appraise small sample size effects, whether heterogeneity was explored and discussed, methods used, and whether formal metaregression had been carried out and the control event rate had been used as covariate.²,¹⁶ We contacted the corresponding authors for details of funding sources (any previous for profit or not for profit funding or specific funding for the review).

As a measure of study quality we appraised the compliance of each systematic review with the QUOROM checklist.¹¹ We considered that the study had complied with any of the 18 specific items if over 50% of the requirements for each item had been met. We also used the Oxman and Guyatt index for scientific quality of research reviews, a validated score with established inter-rater agreement (see bmj.com).²,¹⁷ Finally, we assessed the main findings and conclusions and distinguished these as supporting the role of acetylcysteine in preventing contrast associated nephropathy, not supporting its role, or calling for further research. We then extracted excerpts from the authors' recommendations. Two unblinded reviewers (GGLB-Z, PA) independently appraised the studies. Discrepancies were resolved by consensus.

Statistical analysis

We used Fisher's exact test to compare the proportions of categorical variables (for example, funding source and authors' recommendations) and Mann-Whitney U or Kruskal-Wallis tests to compare the means (ranges) for continuous variables (for example, QUOROM scores according to funding source). Pearson and Spearman tests were used to assess correlations. We used Cohen κ and φ to determine the agreement between QUOROM scores, funding source, and authors' recommendations before consensus.¹⁸ Analyses were carried out using SPSS version 11.0.

Results

Figure 1 shows the flow of reviews through the study. After exclusions from the 52 citations initially identified (double hits, non-systematic or incompletely reported reviews,¹⁹ reviews that lacked quantitative effect estimates, and an ongoing Cochrane review (due for publication in late 2006),²⁰ 10 articles were included (table 1).¹²,^21-29 All were originally submitted between May 2003 and July 2004, published between August 2003 and March 2005, and with no overlap in authorship (49 investigators). Table 2 gives details of the randomised controlled trials that were pooled for each included systematic review. Reviews carried out later could identify and pool randomised controlled trials reported after the publication of the earliest reviews. In three cases the journal in which articles appeared was a general medicine journal, in three cases a cardiology journal, and in four cases a nephrology journal. The median impact factor was 3.9 (range 18.3 to 0). All the studies used the same primary end point of contrast associated nephropathy (defined as > 44.2 μmol/l or > 25% increase in serum creatinine concentration from baseline to 48 hours).

Fig 1.

Flow of reviews through study

Table 1.

Included systematic reviews comparing acetylcysteine with controls for the prevention of contrast associated nephropathy

			Article details						Funding
Database (selection criteria)	Trial and reference	Journal	Date published	Date first submitted	No of pages	Impact factor, 2003	No of authors	No of trials (patients)	For profit*	Not for profit*	Specific for review
Biosis+/RRM, Cochrane Library, Current Contents, Medline, Web of Science (RCT)	Birck et al12	Lancet	Aug 2003	NA	6	18.3	6	7 (805)	Yes	Yes	No
Biosis, Embase, PubMed, other online databases (RCT)	Isenbarger et al21†	American Journal of Cardiology	Dec 2003	May 2003	5	3.1	3	7 (805)	Yes	Yes	Yes‡
Central, CARE, HealthSTAR, Medline (RCT)	Alonso et al22	American Journal of Kidney Disease	Jan 2004	Jun 2003	9	3.9	5	12 (1312)	Yes	Yes	No
Central, Embase, Medline (RCT, non-RCT)	Kshirsagar et al23	Journal of the American Society of Nephrology	Mar 2004	Sep 2003	9	7.5	10	16 (1539)§	Yes	Yes	No
Central, Embase, Medline, Science Citation Index (RCT)	Pannu et al24	Kidney International	Apr 2004	Jul 2003	9	5.3	4	15 (1776)	Yes	Yes	No
Medline (RCT)	Guru and Fremes25†	Clinical Nephrology	Aug 2004	Dec 2003	7	1.3	2	11 (1207)	No	No	No
Central, Embase, Medline (RCT)	Bagshaw and Ghali26	BMC Medicine	Oct 2004	Apr 2004	12	NA	2	14 (1261)	No	Yes	No
Medline (RCT)	Misra et al27†	Clinical Cardiology	Nov 2004	Jul 2004	4	1.2	5	5 (643)	No	No	No
Central, Current Contents, Medline (RCT)	Nallamothu et al28	American Journal of Medicine	Dec 2004	Jan 2004	10	4.4	7	20 (2195)	No	Yes	No
Central, Medline (RCT, non-RCT)	Duong et al29	Catheterization and Cardiovascular Interventions	Mar 2005	Jun 2004	9¶	1.5	3	21 (2420)	No	No	No

Contrast nephropathy defined as >44.2 μmol/l or >25% increase in serum creatinine concentration from baseline to 48 hours, or both.

RCT=randomised controlled trial; NA=not available.

^*Any previous funding.

^†Restricted to English language.

^‡Indirect support from a not for profit source.

^§15 randomised controlled trials.

^¶Includes appendix.

Table 2.

Pooled randomised controlled trials (RCTs) for each of included systematic reviews^*

	Availability†		Systematic reviews
RCT	Source	Report type	Birck et al12	Isenbarger et al21	Alonso et al22	Kshirsagar et al23	Pannu et al24	Guru and Fremes25	Bagshaw and Ghali26	Misra et al27	Nallamothu et al28	Duong et al29
Tepel et al (2000)	Medline	Article	+	+	+	+	+	+	—	—	+	+
Adamian et al (2002)‡	ACC only	Abstract	EX	—	EX	+	—	—	—	—	—	EX
Allaqaband et al (2002)	Medline	Article	+	+	+	+	+	+	+	+	+	+
Azmus et al (2002)	TCT (Medline since 2005)	Abstract (article since 2005)	—	—	+	—	+	—	—	—	+	+
Briguori et al (2002)	Medline	Article	+	+	+	+	+	+	+	+	+	+
Diaz-Sandoval et al (2002)	Medline	Article	+	+	+	+	+	+	+	+	+	+
Durham et al (2002)	Medline	Article	+	+	+	+	+	+	+	—	+	+
Goldenberg et al (2002)	ACC (Medline since 2004)	Abstract (article since 2004)	—	—	—	+	—	—	+	—	+	+
Kahlon et al (2002)	AHA only	Abstract	—	—	—	+	+	—	—	—	+	+
Ochoa et al (2002)	ACC (Medline since 2004)	Abstract (article since 2004)	—	—	+	—	+	—	—	—	+	+
Shyu et al (2002)	Medline	Article	+	+	+	+	+	+	+	+	+	+
Vallero et al (2002)§	Medline	Article	EX	—	—	+	+	—	+	—	+	+
Baker et al (2003)	Medline	Article	—	—	+	—	+	+	+	—	+	+
Boccalandro et al (2003)‡	Medline	Article	—	—	EX	+	—	+	—	—	—	—
Efrati et al (2003)	Medline	Article	—	—	EX	—	—	—	+	—	+	EX
El Mahmoud et al (2003)¶	Medline	Article	—	—	—	—	—	—	—	—	+	—
Gomes et al (2003)	AHA (Medline since 2005)	Abstract (article since 2005)	—	—	—	—	—	—	—	—	+	+
Kay et al (2003)	Medline	Article	+	+	+	+	+	+	+	+	+	+
Kefer et al (2003)	Medline	Article	—	—	—	—	—	—	+	—	—	—
Le Feuvre et al (2003)	ACC only	Abstract	—	—	—	—	+	—	—	—	—	EX
Loutrianakis et al (2003)	ACC only	Abstract	—	—	+	+	+	—	—	—	+	+
MacNeill et al (2003)	Medline	Article	—	—	—	—	—	+	+	—	+	+
Nogareda et al (2003)‡	ACC only	Abstract	—	—	EX	+	—	—	—	—	—	EX
Oldemeyer et al (2003)	ACC (Medline since late 2003)	Abstract (article since late 2003)	—	—	+	+	+	+	+	—	+	+
Tadros et al (2003)‡	Medline	Article	—	—	EX	—	—	—	—	—	—	+
Agarwal et al (2004)	—	Unpublished	—	—	—	+	—	—	—	—	—	—
Fung et al (2004)	Medline	Article	—	—	—	—	—	—	+	—	+	+
Miner et al (2004)	Medline	Article	—	—	—	—	—	—	—	—	—	+
Webb et al (2004)	Medline	Article	—	—	—	—	—	—	—	—	—	+

ACC=American College of Cardiology conference proceedings; AHA=American Heart Association conference proceedings; EX=explicitly excluded from systematic review; TCT=Transcatheter Cardiovascular Therapeutics conference proceedings.

^*References to RCT are available from pertinent meta-analyses.

^†Study data available either as complete reports quoted in Medline, complete reports quoted in Embase, or conference proceedings.

^‡Non-RCT.

^§In Italian.

^¶In French.

Funding was from for profit sources in five cases, not for profit sources in seven cases, and specific for the review in one case (table 1).

Search strategies varied from extensive (five databases in one case) to relying on only Medline. Summary estimates used for pooling were odds ratios (three reviews), relative risks (six), and risk differences (one), by means of random effects (eight) or fixed effects (one) methods (table 3). Quantitative results were reported using both P values and confidence intervals in all cases, the practice supported by most reviewers.¹,² Exploratory metaregression was carried out in light of statistical heterogeneity in six studies. Although most metaregressions included covariates likely to accommodate variability in underlying risk, none formally tested as covariate the role of control group event rates, a practice advocated by some authors and criticised by others.¹⁶,³⁰

Table 3.

Methodological characteristics of included studies

Study	Birck et al12	Isenbarger et al21	Alonso et al22	Kshirsagar et al23	Pannu et al24	Guru et al25	Bagshaw et al26	Misra et al27	Nallamothu et al28	Duong et al29
Pooled analysis:
Carried out	+	+	+	—	+	+	+	+	+	+
Summary estimate used	Relative risk	Odds ratio	Relative risk	Risk difference	Relative risk	Odds ratio	Odds ratio	Relative risk	Relative risk	Relative risk
Means of testing	Random effects	Random effects	Random effects	Not done	Random effects	Fixed effects	Random effects	Fixed and random effects	Random effects	Random effects
Point estimate (95% Cl)	0.44 (0.22 to 0.88)	0.37 (0.16 to 0.84)	0.55 (0.34 to 0.91)	Not done	0.65 (0.43 to 1.0)	0.46 (0.32 to 0.66)	0.54 (0.32 to 0.91)	0.38 (0.21 to 0.66) 0.30 (0.11 to 0.82)	0.73 (0.52 to 1.0)	0.67 (0.47 to 0.96)
Heterogeneity:
Carried out
Assessed	+	+	—	+	+	+	+	+	+	+
Test used	Cochran Q	Der Simonian-Laird	—	Cochran Q	Cochran Q	Breslow-Day	χ2	Cochran Q	χ2	χ2
P value	0.016	0.012	—	<0.001	0.02	0.003	0.032	0.05	0.04	0.01
Metaregression:
Carried out	+	+	—	+	+	—	+	—	—	+
Tested for control event rate	—	—	—	—	—	—	—	—	—	—
Small study size:
Tested formally	+	+	—	+	+	—	+	+	+	+
Test used	Begg, Egger	Begg, Egger, Rosenthal	—	Begg, Egger	Begg	—	Begg, Egger	Rosenthal	Egger	Begg, Egger
P value	0.001, 0.007	0.88, 0.55, 0.06	—	NS	0.02	—	0.03, 0.09	27 unpublished trials needed	0.06	0.23, 0.24
Funnel plot provided	+	—	—	+	+	—	+	—	—	+

NS=not significant.

Five reviews recommended routine, or almost routine, use of acetylcysteine in the prevention of contrast nephropathy. The other five were more cautious and, although not dismissing the potential benefit and limited toxicity and costs of acetylcysteine, called for further well designed randomised controlled trials on the subject (table 4). These discrepancies occurred despite similar pooled effect estimates.

Table 4.

Authors' recommendations on current role of acetylcysteine in prevention of contrast associated nephropathy

Study	Excerpts from study discussion or conclusions	Summary of recommendation*
Birck et al12	Our meta-analysis showed a significant benefit of acetylcysteine treatment in prevention of contrast nephropathy in patients with chronic renal insufficiency... the reported association of contrast nephropathy with increased morbidity, mortality, and lengthened hospital stay might justify use of acetylcysteine for prevention of contrast nephropathy	Supports use of acetylcysteine
Isenbarger et al21	All considered, given the significant benefits obtained, the low cost... and ease of administration... it is reasonable to recommend routine prophylactic NAC in patients with stable chronic renal insufficiency or diabetes who will undergo elective angiography	Supports use of acetylcysteine
Alonso et al22	NAC is effective in preventing RCN (radiocontrast nephropathy)... Although cost-effectiveness analyses are lacking, NAC is inexpensive and has limited side effects and therefore should be recommended to subjects at high risk for RCN	Supports use of acetylcysteine
Kshirsagar et al23	Research... is too inconsistent at present to warrant a conclusion on efficacy or a recommendation for... routine use... A large, randomized, placebo-controlled trial, a pooled analysis of patient-level data, or both may resolve this issue	Larger randomised controlled trials needed
Pannu et al24	NAC may reduce the incidence of acutely increased serum creatinine after administration of intravenous contrast, but this finding was of borderline statistical significance, and there was significant heterogeneity between trials. Before NAC becomes the standard of care for all patients receiving intravenous contrast, new randomized trials evaluating its effects on clinically relevant outcomes are required	Larger randomised controlled trials needed
Guru and Fremes25	A review of NAC in the prevention of contrast dye-induced nephropathy shows that although the completed studies were heterogeneous, a majority showed a positive effect. There is controversy as to whether this observed effect of NAC as reported in the literature is simply a result of a publication bias where small, negative trials are less likely to be reported... This demonstrates the need for the completion of a large multicenter trial...	Larger randomised controlled trials needed
Bagshaw and Ghali26	These findings indicate that published studies of acetylcysteine for prevention of contrast-induced nephropathy yield inconsistent results. The efficacy of acetylcysteine will remain uncertain unless a large well-designed multi-center trial is performed	Larger randomised controlled trials needed
Misra et al27	Oral administration of NAC in addition to intravenous saline hydration has a beneficial effect in the prevention of contrast-induced nephropathy after cardiovascular procedures in patients with impaired renal function	Supports use of acetylcysteine
Nallamothu et al28	Acetylcysteine may reduce the incidence of contrast-related nephropathy, but this finding is reported inconsistently across currently available trials. High-quality, large clinical trials are needed before acetylcysteine use in this indication can be recommended universally	Larger randomised controlled trials needed
Duong et al29	While awaiting such studies, our finding of a 0.57 relative risk coupled with its relatively low cost and minimal toxicity suggests that NAC with hydration currently should be used in all patients at risk for RCIN	Supports use of acetylcysteine

RCIN=radiocontrast induced nephropathy.

^*Assessed by two independent reviewers.

Compliance with the QUOROM statement

Agreement before consensus between reviewers was high for compliance with the QUOROM items (174/180 (91%), κ = 0. 89, 95% confidence interval 0.80 to 0.97; φ = 0.89), and 100% (10 reviews) for authors' recommendations. The median compliance with the QUOROM checklist was 16 (range 11 to 17; table 5).

Table 5.

Compliance of included studies with quality of reporting of meta-analyses (QUOROM) checklist and Oxman and Guyatt index^*

Characteristic	Birck et al12	Isenbarger et al21	Alonso et al22	Kshirsagar et al23	Pannu et al24	Guru et al25	Bagshaw et al26	Misra et al27	Nallamothu et al28	Duong et al29
Identifying title†	−	+	+	+	−	−	−	−	−	−
Abstract:
Structured	+	−	+	−	+	+	+	+	+	−
Objectives	+	+	+	+	+	+	+	+	+	+
Data sources	+	−	+	+	+	+	+	−	−	−
Review methods	−	−	+	+	+	+	−	+	+	+
Results	+	−	+	−	+	+	+	−	+	+
Conclusion	+	−	+	+	+	+	+	+	+	+
Introduction	+	+	+	+	+	+	+	+	+	+
Methods:
Searches	+	+	+	+	+	+	+	+	+	+
Selection	+	+	+	+	+	+	+	+	+	+
Validity assessment	+	+	−	+	+	−	+	−	+	+
Data abstraction	+	+	+	+	+	−	+	−	+	+
Study characteristics	+	+	+	+	+	+	+	−	+	+
Quantitative data synthesis	+	+	+	+	+	+	+	+	+	+
Results:
Trial flow	−	−	+	+	−	−	+	−	−	+
Study characteristics	+	+	+	+	+	+	+	+	+	+
Quantitative data synthesis	+	+	+	+	+	+	+	+	+	+
Discussion	+	−	+	+	+	+	+	+	+	+
Total QUOROM score	15	11	17	16	16	16	16	12	15	14
Explicit quotation of QUOROM statement by authors	+	−	−	−	−	−	+	−	+	−‡
Guidelines for meta-analysis endorsed by journal	+‡	−	−	−	−	−	−	+‡§	−	−
Oxman and Guyatt score¶	7	6	3	4	6	1	7	2	6	6

+=More than 50% of requirements accomplished; −=non-compliance.

^*Compliance with QUOROM items assessed by two independent reviewers.

^†Title includes study design.

^‡Explicit reference to Cochrane Collaboration Reviewers' Handbook.¹

^§Explicit reference to QUOROM statement.

^¶Ranging from 1 for extensive flaws to 7 for minimal flaws.²

In only three cases could the report be identified from the title as a meta-analysis or systematic review of randomised controlled trials. A structured abstract was available in seven reviews, and objectives and conclusions were explicitly stated in 10 and nine, respectively. In the abstract sections, data sources, review methods, and results complied with the QUOROM statement in only six cases.

All the studies fulfilled the QUOROM requirements for the introduction and discussion sections, as they reported search strategies, selection criteria, and methods for quantitative data synthesis employed for the systematic review. Most of the studies complied with the recommendations for reporting of the assessment of the methodological quality of the primary studies (seven reviews), data abstraction (eight), and study characteristics (nine).

Despite thorough reporting of methods by most studies, no review quantified the agreement between reviewers on selection and appraisal of the methodological quality of the primary studies. A diagram showing the flow of the trial was provided in only three cases. Details on study characteristics and quantitative data synthesis in the results section were, however, provided in enough detail in all the reviews.

Quantitative analyses and predictors of QUOROM score

A significant association was found between number of published pages and overall QUOROM score (Pearson R = 0.73, P = 0.016, Spearman ρ = 0.64, P = 0.046), suggesting that the longer the manuscript, the greater the likelihood of complying with the QUOROM statement (fig 2). Although this finding might have relevant implications for editorial policies, it should be interpreted with caution given the small sample size and the fact that two studies largely drive the hypothesis tests.

Fig 2.

Scatter plot showing relation between manuscript length and quality of reporting of meta-analyses (QUOROM) score. Thorough and complete reports achieved higher scores

Language restrictions were associated with fewer trials analysed (7, range 5-11 v 15, range 7-21, P = 0.040) and fewer patients analysed (805, range 643-1207 patients v 1539, 805-2420; P = 0.033) (tables 1 and 3).

Studies from reviewers reporting previous not for profit funding were more likely to score higher on the Oxman and Guyatt index (6, range 3-7 v 2, range 1-4; P = 0.037), to search more databases for original articles (4, 3-5 v 1.3, 1-2; P = 0.014), and to be published in a journal with a greater impact factor (4.9, 3.1-18.3 v 1.3, 1.2-1.5; P = 0.020). Conversely, funding was not significantly associated with authors' recommendations supporting the routine use of acetylcysteine (43% (three of seven) for studies reporting previous not for profit funding compared with 66% (two of three) for the others; P = 0.42) or QUOROM score (16, range 11-17 for studies with previous not for profit funding compared with 14, range 12-16 for the others; P = 0.38).

Journal readership on the basis of type of journal was significantly associated with quality of reporting, with a significant trend towards increasing QUOROM scores from reviews published in cardiology journals compared with those published in general medicine and nephrology journals. Moreover, QUOROM scores and Oxman and Guyatt scores were not associated (Pearson R = -0.06, P = 0.86, Spearman ρ = -0.11, P = 0.76). In most cases this was due to lack of assessment of bias in selecting and abstracting studies (six reviews) or lack of appraisal of internal validity of individual randomised controlled trials (five).

Limits of the included studies

Clinical and statistical heterogeneity are a major source of discordance in meta-analyses. A thorough appraisal of this issue is paramount in any research synthesis. Moreover, some authorities consider heterogeneity (P < 0.05 to 0.10 with hypothesis tests or I² > 50%) a potential hurdle to the completion of a formal meta-analysis.¹,² Heterogeneity was appraised in most of the included meta-analyses (nine reviews), by means of Cochran Q, Der Simonian-Laird, Breslow-Day, or χ² tests. In all cases evidence for statistical heterogeneity was present (P values ranging from 0.05 to < 0.001), and in all studies except one²³ reviewers computed pooled effect size, on the basis of random effects methods (seven reviews), on fixed effects (one), and on both fixed and random effects (one).

A major source of bias in meta-analyses is small study size, which predisposes to a higher probability of publishing, quoting, and disseminating the results of randomised controlled trials with significant or positive results. This leads to pooled effect estimates being skewed to more positive findings, as small but significant randomised controlled trials are not counterbalanced in the statistical pooling procedure by similarly small but negative or non-significant studies, as shown graphically by funnel plots.¹,² Bias due to small study size was acknowledged and explicitly tested in most (eight) reviews, using Begg, Egger, or Rosenthal tests. Such tests provided significant results, suggesting the potential presence of such bias in five cases, prompting reporting and discussion of funnel plots in four of the five, plus another review that had none the less reported a non-significant test (table 1). Despite checking for small study bias among the same pool of seven studies, Birck et al¹² and Isenbarger et al²¹ reached disparate conclusions, respectively in favour and against the likely presence of such bias, a finding that can be partly explained by the use of risk ratios by Birck et al and odds ratios by Isenbarger et al. Moreover, despite such likelihood (P < 0.10) for small study bias at specific tests, all meta-analyses went forward with the pooled estimates thus providing potentially biased results. Duong et al carried out two separate analyses, the first limited to randomised controlled trials published as complete reports and the second also including randomised controlled trials available only as abstracts.²⁹ They thus showed that bias due to small study size was likely when selecting studies published only as complete reports (P = 0.02), but that it was no longer evident when including studies reported only as abstracts (P = 0.22).²⁹

Discussion

Overall compliance with the QUOROM checklist was relatively good by the investigators of 10 overlapping systematic reviews on the role of acetylcysteine in the prevention of contrast associated nephropathy. Shorter manuscripts were of significantly lower quality and previous not for profit funding was associated with higher quality.

Less rigorous reviews have reported positive conclusions more often than more robust reviews focusing on the same topic,¹⁰ even if other studies did not corroborate the hypothesis.⁵,³¹,³² More recently a qualitative comparison of overlapping yet heterogeneous systematic reviews on complementary medicine showed that reviews differed greatly, mainly because of subjective decisions during the planning, carrying out, and interpretation of the reviews.³³ Despite these data, the phenomenon of overlapping systematic reviews seems to become more prevalent, as highlighted by the present study series. The time has come for an open debate on this issue.

Systematic reviews have the inherent drawback that several independent meta-analysts may conceive, carry out, and submit a review on the same topic at the same time. This phenomenon of overlapping systematic reviews has already occurred.³⁴,³⁵ Some may welcome this as a means of highlighting the subjective choices and different perspectives available in carrying out research synthesis. The risk of wasting resources and providing the general medical readership with contradictory conclusions should not, however, be dismissed.

Potential reasons for the publication of 10 overlapping systematic reviews in less than two years are the preference given by investigators, editors, and readers of systematic reviews,³⁶ the lack of a pivotal large randomised controlled trial, and the potential for a dose-response gradient.³⁷

In the present study series, 49 investigators were involved in analysing the same clinical topic, exemplifying the waste of time and duplication of efforts in such research. Too many systematic reviews are being produced for diseases with a limited global burden, whereas many other potentially more relevant clinical conditions are not undergoing a thorough and systematic research synthesis.³⁸ Constraints on space reflected by manuscript length (number of pages in our study) and abstract length are one of the most important hurdles faced by meta-analysts wanting to document their work thoroughly. Our findings suggest that no meta-analysis can achieve optimal quality scores unless sufficient space is provided for all its sections. Countermeasures for this problem seem straightforward, and range from increasing limits on space for systematic overviews of randomised controlled trials to the provision of online data supplements whenever room is limited. The Cochrane Collaboration does not impose any space limit for its systematic reviews.

Funding might influence outcomes and quality of research.³⁹ Reviewers who reported previous not for profit funding were more likely to carry out higher quality systematic reviews. The association is not, however, synonymous to causality, and other explanations for our results include greater experience and expertise among review teams that have already carried out systematic reviews.

Although the QUOROM statement concerns reporting quality, the Oxman and Guyatt index was designed to focus on methodological quality.¹⁷ The relative disagreement between the scores in our study obviously stems from the different purposes of the checklists. Indeed, lack of reporting on the assessment of the internal validity of single randomised controlled trials may restrict Oxman and Guyatt scores to the low 1-3 range, even for well reported meta-analyses.²

One major difference between the systematic reviews we appraised was the number of primary studies included, with some reviews analysing as few as five studies,²⁷ despite being submitted for publication almost one year after publication of a review of 12 studies.²² Potential explanations for such discrepancies might be the extent of database searches (whether or not conference proceedings were handsearched), language restrictions, whether selection was restricted to randomised controlled trials or included non-randomised studies, and timing of review. We cannot exclude the possibility that some search strategies missed eligible trials.

Finally, almost all reviewers carried out and reported formal meta-analyses in the presence of heterogeneity and small study size. Although metaregression and sensitivity analyses were employed to tackle these issues, the conflicting recommendations may in part be a consequence of such discrepancies and, more likely in the present case, biased publication of primary studies.

What is already known on this topic

Multiple systematic reviews on the same clinical topic may produce conflicting results

The quality of reporting of meta-analysis (QUOROM) statement was developed to improve the quality of reviews, yet its effect is uncertain

What this study adds

Ten overlapping systematic reviews on acetylcysteine to prevent contrast nephropathy varied in quality of reporting and recommendations

Longer manuscripts and previous not for profit funding were associated with higher quality

Preventive strategies for multiple overlapping reviews

The best approach to avoid multiple overlapping reviews is probably that of the Cochrane Collaboration, which mandates prospective registration of titles and protocols.¹ Another option might be online registration of meta-analyses already accepted for publication by a journal, as this would not disseminate findings before completion of the study but would forewarn other researchers involved in the same topic and allow investigators to refocus on their work. Such recommendations come from personal experience, as our group has already been involved in several potential cases of overlapping meta-analyses.^40-42 Nonetheless, these preventive measures are likely to impose logistical and financial hurdles potentially proving counterproductive. Finally, some of the problems raised by our work are likely inherent practical drawbacks of systematic reviews.

Practical implications for the clinical use of acetylcysteine

Although the clinical use of acetylcysteine was not a formal goal of our work, some broad recommendations can be gleaned from the available evidence. The studies found no evidence of toxicity or adverse effects from using acetylcysteine. However the real preventive size effect cannot be definitively inferred, given the clinical and statistical heterogeneity and the lack of clear explanations for such an inconsistency.¹²,²¹,²³,²⁴,²⁶,²⁹

Study limitations

Limitations of our study include its design, its scope for generating the hypothesis, and the risk of α and β errors.⁴³ Moreover, most of this study involved the appraisal of published meta-analyses using the QUOROM checklist. Although scientific evidence supports only a subset of the items on the checklist, we believe that it can and should be used as a standard in the preparation, reporting, and appraisal of meta-analyses of randomised controlled trials on the basis of its wide dissemination and endorsement and the relative ease with which compliance can be achieved. It should be borne in mind that the QUOROM checklist was not developed as a quality measurement tool, and validation of this novel application is lacking.