Skip to main content
PMC home page
NPJ Primary Care Respiratory Medicine logoLink to NPJ Primary Care Respiratory Medicine
. 2015 Jan 8;25:14102. doi: 10.1038/npjpcrm.2014.102

Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool

Robin ST Ho 1, Xinyin Wu 1, Jinqiu Yuan 1, Siya Liu 1, Xin Lai 1, Samuel YS Wong 1, Vincent CH Chung 1,2,*
PMCID: PMC4498191  PMID: 25569783

Abstract

Background:

Meta-analysis (MA) of randomised trials is considered to be one of the best approaches for summarising high-quality evidence on the efficacy and safety of treatments. However, methodological flaws in MAs can reduce the validity of conclusions, subsequently impairing the quality of decision making.

Aims:

To assess the methodological quality of MAs on COPD treatments.

Methods:

A cross-sectional study on MAs of COPD trials. MAs published during 2000–2013 were sampled from the Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effect. Methodological quality was assessed using the validated AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool.

Results:

Seventy-nine MAs were sampled. Only 18% considered the scientific quality of primary studies when formulating conclusions and 49% used appropriate meta-analytic methods to combine findings. The problems were particularly acute among MAs on pharmacological treatments. In 48% of MAs the authors did not report conflict of interest. Fifty-eight percent reported harmful effects of treatment. Publication bias was not assessed in 65% of MAs, and only 10% had searched non-English databases.

Conclusions:

The methodological quality of the included MAs was disappointing. Consideration of scientific quality when formulating conclusions should be made explicit. Future MAs should improve on reporting conflict of interest and harm, assessment of publication bias, prevention of language bias and use of appropriate meta-analytic methods.

Introduction

Chronic obstructive pulmonary disease (COPD) is a chronic respiratory disease distinguished by airway limitation.1,2 COPD is estimated to rank fifth worldwide in disease burden,3 and it is estimated to increase in the coming decades as a consequence of increasing exposure to environmental risk factors and an ageing population.2,3 More than 5% of the population is affected with the disease and it is associated with high mortality and morbidity.4 Because of the high prevalence and chronicity, COPD patients rely heavily on medical interventions. For stable COPD, a wide variety of pharmacological treatments are used.3 Non-pharmacological and rehabilitation strategies are also commonly used to improve the functional capacity of COPD patients.3

There are numerous treatments available for COPD and it is important for clinicians to make a decision on their use on the basis of up-to-date clinical evidence.5 Meta-analysis (MA) of randomised trials is considered to be one of the best approaches for summarising high-quality evidence on the efficacy and safety of various therapeutic options.6 Although these MAs are published continually as trustworthy sources of evidence, little attention is paid to their methodological quality. Methodological flaws in the conduct of MAs could cause biased conclusions. For example, an incomprehensive literature search could lead to failure in locating unpublished data. This omission may cause changes in the direction of effect for a particular treatment.7,8 Other methodological shortcomings that can lead to biased results include the lack of independent audit during data extraction, inappropriate combination of heterogeneous study results, and failure to consider risk of bias among primary studies. As biased results from MAs can mislead clinical practice,9 it is important to assess the methodological quality of MAs on COPD treatment. Results will provide an overview of MA quality in the field of COPD, and accordingly suggestions for future methodological improvement can be made.

AMSTAR (Assessing the Methodological Quality of Systematic Reviews) is a freely accessible validated tool for assessing the methodological quality of MAs.10,11 The full version can be found in the AMSTAR official website.10 Validation studies9,11 have shown that AMSTAR is a reliable critical appraisal tool with good agreement, construct validity and feasibility. The AMSTAR is adopted by a number of research and health technology assessment groups, such as the Canadian Agency for Drugs and Technologies in Health and The Cochrane Effective Practice and Organization of Care Group.12

Using a cross-sectional study design, this study aimed to (1) describe the bibliographical characteristics of MAs on COPD therapies, (2) evaluate the methodological quality of MAs on COPD treatments by using AMSTAR and (3) examine the association between bibliographical characteristics and methodological quality.

Materials and Methods

Sampling of meta-analyses

We sampled MAs by searching the Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effect (DARE) using the keyword ‘COPD’. CDSR enabled us to locate Cochrane reviews, whereas DARE is an extensive, weekly updated systematic review database that indexes non-Cochrane MA. The use of these databases allowed us to assemble a representative sample of MAs for appraisal.

Inclusion and exclusion criteria

MAs on COPD treatments published between January 2000 and April 2013 were eligible. Narrative reviews, systematic reviews with no meta-analyses, network meta-analyses and systematic reviews on respiratory diseases other than COPD (e.g., asthma, emphysema, chronic bronchitis, bronchiectasis and so on) were excluded. Two authors independently screened the titles, abstracts and full texts of the retrieved literature to assess their eligibility. Discrepancies were discussed and solved by discussion and consensus among the authors.

Extraction of bibliographical characteristics and critical appraisal

Bibliographical characteristics were assessed with a pre-designed questionnaire. All 11 items of the AMSTAR instrument were used for appraising methodological quality.11 Judgements were given as either ‘yes’ or ‘no’ for items 1, 2 and 5–11. For items 3 and 4, an option from ‘yes’, ‘cannot answer or not reported’ or ‘no’ was chosen. No aggregate score was calculated, as the original version of AMSTAR did not provide any recommendation on using this approach10 and there is no empirical evidence to advise on the appropriate weighting for each item when calculating a total score.9 A detailed operational guide for AMSTAR can be found in Supplementary Material 1.

Data analysis

The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using multivariate logistic regression or multi-nominal logistic regression depending on the scaling of the dependent variable. Hosmer and Lemeshow tests were performed to evaluate the model fitting for all logistic regressions. All statistical analyses were conducted with SPSS 18.0, with a two-tailed significance level of 0.01 to avoid multiple test bias.

Results

MA searching and selection

The search strategy identified a total of 137 systematic reviews (73 from CDSR and 64 from DARE). No duplicates were found. After assessing the titles, abstracts and full texts, a total of 79 MAs met our inclusion criteria and were included in this study (Figure 1). A full list of included MAs can be found in Supplementary Material 2.

Figure 1.

Figure 1

Open in a new tab

Sampling of meta-analyses on COPD treatment: flow chart.

Bibliographical characteristics and methodological quality

The median year of publications was 2010 (n=44, 55.7%). There were more MAs published in the field from 2010 onwards. Fifty (63.3%) MAs had an impact factor of >5. Detailed results for bibliographical characteristics and methodological quality are shown in Tables 1 and 2.

Table 1. Bibliographical characteristics of 79 included meta-analyses on COPD treatments.

Bibliographical characteristics Results
Cochrane review 47 (59.5%)
An update of a previous meta-analysis (MA) 49 (62.0%)
Median impact factor of the journal for which the MA was published (range) 5.18 (0.00–6.25)
Median number of review authors (range) 4.00 (2–8)
Location of corresponding author
 North America 21 (26.6%)
 Europe 27 (34.2%)
 Australasia 21 (26.6%)
 Other regions 10 (12.7%)
Type of treatment
 Pharmacological 43 (54.4%)
 Non-pharmacological 36 (45. 6%)
Total number of included primary studies 1,099
Total number of participants in included primary studies 801,294
Number of MA that reported harm of the intervention 46 (58.2%)
Funding location of the MA
 Europe 23 (29.1%)
 Australia or New Zealand 9 (11.4%)
 Multiple region 9 (11.4%)
 North America 8 (10.2%)
 South America 1 (1.3%)
 Not reported 29 (36. 7%)
Number of MA that declared conflicts of interest 15 (19.0%)
Number of MA that declared no conflicts of interest 26 (32.9%)
Number of MA that did not mention conflicts of interest 38 (48.1%)
Number of MA that searched international databases 78 (98. 7%)
Median number of international databases searched (range) 4.00 (0–12)
Number of MA searched non-English databases 8 (10.1%)
Reporting of coverage year of search:
 Yes, reported both starting and ending years 26 (32.9%)
 Partially, only reported starting years 48 (60.8%)
 Not mentioned 5 (6.3%)
Search terms reported for one or more electronic databases
 Topics/free text/keywords/MeSH 62 (78.5%)
 Full Boolean 14 (17.7%)
 Readers are referred elsewhere for full search strategy 3 (3.8%)
Eligibility of study design
 RCT only 75 (94.9%)
 RCT and observational studies 4 (5.1%)
Eligibility criteria based on language of publication
 Included English publications only 11 (13.9%)
 English and languages other than English 42 (53.2%)
 Language criteria not reported 26 (32.9%)
Number of MA that included a PRISMA-Iike flow diagram 37 (46.8%)
Tools for assessing risk of bias of primary studies
 Cochrane risk of bias tool 31 (39.2%)
 Jadad scale 16 (20.3%)
 Pedro scale 6 (7.6%)
 Others 4 (5.1%)
 Two or more of the above-mentioned tools 13 (16.5%)
 Not reported 9 (11.4%)
Open in a new tab

Abbreviations: COPD, chronic obstructive pulmonary disease; MA, meta-analysis; RCT, randomised controlled trials.

Table 2. Methodological quality of 79 included meta-analyses on COPD treatments.

Individual AMSTAR items Yes (%) No (%) Cannot answer (%)
1. Was an 'a priori' design provided? 53 (67.1) 26 (32.9) NA
2. Was there duplicate study selection and data extraction? 61 (77.2) 0 (0.0) 18 (22.8)
3. Was a comprehensive literature search performed? 73 (92.4) 3 (3.8) 3 (3.8)
4. Was the status of publication used as an inclusion criterion (i.e., grey literature would be included, if located)? 47 (59.5) 5 (6.3) 27 (34.2)
5. Was a list of studies (both included and excluded) provided? 48 (60.8) 31 (39.2) NA
6. Were the characteristics of the included studies provided? 69 (87.3) 10 (12.7) NA
7. Was the scientific quality of the included studies assessed and documented? 65 (82.3) 14 (17.7) NA
8. Was the scientific quality of the included studies used appropriately in formulating conclusions? 14 (17.7) 65 (82.3) NA
9. Were the methods used to combine the findings of studies appropriate? 39 (49.4) 40 (50.6) NA
10. Was the likelihood of publication bias assessed? 28 (35.4) 51 (64.6) NA
11. Were the sources of support for both the systematic review and the included primary studies reported? 2 (2.5) 77 (97.5) NA
Open in a new tab

Abbreviations: AMSTAR, Assessing the Methodological Quality of Systematic Reviews; COPD, chronic obstructive pulmonary disease; NA, not applicable.

Association between bibliographical characteristics and methodological quality: multivariate analyses

Table 3 illustrates the results from logistic regression analyses (full results can be found in Supplementary Material 3). A higher journal impact factor was significantly associated with favourable methodological scoring on the following aspects: an ‘a priori’ design provided in the MA; duplicate study selection and data extraction; and provision of study lists. MAs published more recently were more likely to have conducted an assessment of the scientific quality of included studies, as well as an evaluation of publication bias. MAs on non-pharmacological treatments performed better in considering the scientific quality of included studies when formulating conclusions, compared with their pharmacological counterparts. Finally, MAs on non-pharmacological treatments also fared marginally better in using appropriate meta-analytic methods for combining findings. P values of Hosmer–Lemeshow tests for all logistic regression analyses were >0.05.

Table 3. Association between publication characteristics and methodological quality of MAs on COPD treatments: multivariate analyses.

AMSTAR item (dependent variable) Predictors Adjusted odds ratio (99% CI) P values
1. Was an 'a priori' design provided? (Yes versus No) Higher impact factor 4.22 (1.50–11.86) <0.0001
2. Was there duplicate study selection and data extraction? (Yes versus Cannot answer) Higher impact factor 2.01 (1.19–3.38) 0.001
5. Was a list of studies provided? (Yes versus No) Higher impact factor 6.85 (1.58–29.69) 0.001
7. Was the scientific quality of the included studies assessed and documented? (Yes versus No) More recent publication years 1.40 (0.94–2.08) 0.030
8. Was the scientific quality of the included studies used appropriately in formulating conclusions? (Yes versus No) Non-pharmacological treatment (pharmacological treatment as reference) 6.53 (0.92–46.36) 0.014
9. Were the methods used to combine the findings of studies appropriate? (Yes versus No) Non-pharmacological treatment (pharmacological treatment as reference) 4.49 (0.98–20.72) 0.011
10. Was the likelihood of publication bias assessed? (Yes versus No) More recent publication years 1.83 (1.13–2.97) 0.001
Open in a new tab

Abbreviations: AMSTAR, Assessing the Methodological Quality of Systematic Reviews; CI, confidence interval; COPD, chronic obstructive pulmonary disease; MA, meta-analysis.

Discussion

Main findings

This study assessed the methodological quality of 79 MAs on COPD treatment published between 2000 and 2013. The proportion of MAs that fulfilled each individual AMSTAR domain was similar to that of other medical fields like urology, orthodontics and nursing.13–15 An overview of results from these studies is shown in Table 4. The majority of MAs were published after 2010, demonstrating an increasing popularity of systematic review methods among researchers in the field. The quality of more recent MAs in terms of assessing and documenting the scientific quality of the included studies was higher, reflecting progress in methodological sophistication. The frequency of updating was acceptable as >60% of the included MAs represented an update of a previous report. Higher impact factor is associated with clarification of an ‘a priori’ design, and this trend could be explained by the requirement of Cochrane MA in publishing a protocol. Other MA authors should consider registering their MA protocols on PROSPERO, which is an international online platform for prospective registration of systematic review protocols.16

Table 4. Methodological quality of meta-analyses in other health-care fields assessed by AMSTAR.

Study Fields No. Main results
MacDonald et al. 13 Urology 57 49.1% SRs searched at least two databases; 31.6% SRs searched unpublished studies; 45.6% SRs provided a list of included and excluded studies; 63.2% SRs assessed and documented the methodological quality of included studies.
Papageorgiou et al. 14 Orthodontics 110 20.0% clearly reported only the review question or only the inclusion criteria; 35.5% conducted in duplicate only study selection, but not data extraction; the grey literature was not scanned for relevant articles in 54 reviews (49.1%). 65.5% did not provided excluded studies; 8.2% did not provide included or excluded studies in a list or a table at all.
Seo and Kim15 Nursing 22 13.6% SRs were performed in duplicate study selection and data extraction; 18.2% SRs used publications status as an inclusion criterion; 63.6% SRs did not provide information on both included and excluded studies; 13.6% SRs assessed and documented the quality of the included studies and drew an appropriate conclusion reflecting the scientific quality of the included studies; 72.7% SRs appropriately combined the findings of studies using meta-analytic methods.
Open in a new tab

Abbreviations: AMSTAR, Assessing the Methodological Quality of Systematic Reviews; SR, systematic review.

Interpretation of findings in relation to previously published work

Despite these progresses, the overall methodological quality of MAs on COPD treatment can only be considered disappointing. Only 17.7% of included MAs explicitly considered the scientific quality of primary studies when formulating their conclusions. As trials with high risk of bias are more likely to report larger effect sizes,17 readers of MAs on COPD treatments may consider reported conclusions more judiciously. Even when the trial’s risk of bias had been assessed, <40% of MAs had used the Cochrane risk of bias tool, a specialised instrument developed for appraising RCTs.18 In the future, proper use of Cochrane risk of bias tools should be promoted and explicit consideration on the risk of bias of primary studies and its impact on the treatment results should be made.

These improvement measures are especially important for MAs on pharmacological treatments as it appears that they perform less well compared with non-pharmacological MAs. Less than satisfactory performance in this aspect could be related to the influence of funders, as previous studies have clearly shown the relationship between industry funding and positive results from MAs.19–21 Another area of concern is the lack of reporting on conflict of interest among MA authors: in our results, only two MAs reported the funding sources for all included primary studies, as well as for the MA itself. More than 48% of the MA authors did not declare a conflict of interest. In addition, harmful effects of treatment were also under-reported in the sampled MAs. A total of 1,099 studies were summarised in these MAs, representing data from more than 800,000 patients. Despite the wealth of data reviewed, only 58% described the harmful effects of treatment. In the future, it is important for journal editors to encourage reporting in these aspects.

A substantial proportion of included MAs (64.6%) did not assess the likelihood of publication bias. As studies with positive results are published more often and more quickly compared with negative studies regardless of methodological quality,22,23 an overestimation of COPD treatment effect could be observed among the MAs.24 Treatment effect could be overestimated if publication bias existed even when the included individual trials have a low risk of bias.25 To reduce the likelihood of overestimating results, in the future, MAs should explicitly assess publication bias using proper methods,26 and appropriate adjustment on the meta-analytic results can be considered if publication bias is present.27

In addition to publication bias, language bias also increases the risk of overestimation as positive findings are more likely to be published in English language journals.28 In our MA sample, only 10% of the MAs published had searched non-English databases. Building an international, multi-lingual team for conducting MAs could reduce English language bias by widening the coverage on non-English databases. Currently, more than half of the corresponding authors are from North America and Europe, and the review team size is rather small, with a median of four people. Adding authors from non-Western countries could be a solution for reducing language bias.

Strengths and limitations of this study

One of the strengths of this study was that we identified all MAs indexed in CDSR and DARE. The CDSR included all Cochrane Reviews prepared by Cochrane Review Groups within the Cochrane Collaboration, whereas DARE covers the majority of indexed non-Cochrane MAs.29 One limitation of the present study is that the AMSTAR appraisal process was difficult to implement when the reporting quality was poor. This could be attributed to space restrictions in non-Cochrane journals, which limit detailed reporting.30 Authors and journal editors are encouraged to adhere to the PRISMA requirement, which includes all required reporting components for authors to publish a systematic review.31

Conclusions

Our findings demonstrate that the methodological quality of MAs on COPD treatment is disappointing. Areas that require improvement in the future include the consideration of scientific quality when formulating conclusions, reporting of conflict of interest, assessment of publication bias, and prevention of language bias. Also, the harmful effects of treatment were often under-reported. Clinicians should be judicious when applying the conclusions of MA results to patients, taking into account the potential bias resulting from methodological limitations. Journal editors and peer reviewers will have a strong role in ensuring continual improvement of MA quality by upholding the methodological and reporting standards recommended by AMSTAR and PRISMA.31

Acknowledgments

The authors thank the staff of Li Ping Medical Library from the Chinese University of Hong Kong for retrieving research papers.

The authors declare no conflict of interest.

Footnotes

Supplemental Information accompanies the paper on the npj Primary Care Respiratory Medicine website (http://www.nature.com/npjpcrm)

Supplementary Information
Click here for additional data file. (271KB, doc)

References

  1. Buist AS, McBurnie MA, Vollmer WM, Gillespie S, Burney P, Mannino DM. International variation in the prevalence of COPD (the BOLD Study): a population-based prevalence study. Lancet. 2007;370:741–750. doi: 10.1016/S0140-6736(07)61377-4. [DOI] [PubMed] [Google Scholar]
  2. Gershon AS, Warner L, Cascagnette P, Victor JC, To T. Lifetime risk of developing chronic obstructive pulmonary disease: a longitudinal population study. Lancet. 2011;378:991–996. doi: 10.1016/S0140-6736(11)60990-2. [DOI] [PubMed] [Google Scholar]
  3. Global initiative for chronic obstructive lung disease . Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease: Updated 2013. Available from http://www.goldcopd.org/uploads/users/files/GOLD_Report_2013_Feb20.pdf .
  4. Centers for Disease Control and Prevention (CDC) Chronic obstructive pulmonary disease among adults—United States, 2011. MMWR Morb Mortal Wkly Rep. 2012;61:938–943. [PubMed] [Google Scholar]
  5. Celli BR, MacNee W. Standards for the diagnosis and treatment of patients with COPD: a summary of the ATS/ERS position paper. Eur Respir J. 2004;23:932–946. doi: 10.1183/09031936.04.00014304. [DOI] [PubMed] [Google Scholar]
  6. Oxford Centre for Evidence-based Medicine—Levels of Evidence (March 2009). University of Oxford. Available from http://www.cebm.net/?o=1025 .
  7. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358:252–260. doi: 10.1056/NEJMsa065779. [DOI] [PubMed] [Google Scholar]
  8. Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet. 2004;363:1341–1345. doi: 10.1016/S0140-6736(04)16043-1. [DOI] [PubMed] [Google Scholar]
  9. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10. doi: 10.1186/1471-2288-7-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. AMSTAR offical website. Available from: amstar.ca.
  11. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62:1013–1020. doi: 10.1016/j.jclinepi.2008.10.009. [DOI] [PubMed] [Google Scholar]
  12. Oxman AD, Schunemann HJ, Fretheim A. Improving the use of research evidence in guideline development: 8. Synthesis and presentation of evidence. Health Res Policy Syst. 2006;4:20. doi: 10.1186/1478-4505-4-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. MacDonald SL, Canfield SE, Fesperman SF, Dahm P. Assessment of the Methodological Quality of Systematic Reviews Published in the Urological Literature From 1998 to 2008. J Urol. 2010;184:648–653. doi: 10.1016/j.juro.2010.03.127. [DOI] [PubMed] [Google Scholar]
  14. Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Evaluation of methodology and quality characteristics of systematic reviews in orthodontics. Orthod Craniofac Res. 2011;14:116–137. doi: 10.1111/j.1601-6343.2011.01522.x. [DOI] [PubMed] [Google Scholar]
  15. Seo HJ, Kim KU. Quality assessment of systematic reviews or meta-analyses of nursing interventions conducted by Korean reviewers. BMC Med Res Methodol. 2012;12:129. doi: 10.1186/1471-2288-12-129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Centre for Reviews and Dissemination—The University of York . Welcome to PROSPERO International prospective register of systematic reviews:: National Institute for Health Research; [updated 21/2/2014]. Available from http://www.crd.york.ac.uk/PROSPERO/ .
  17. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008;336:601–605. doi: 10.1136/bmj.39465.451748.AD. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. doi: 10.1136/bmj.d5928. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Jorgensen AW, Hilden J, Gotzsche PC. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006;333:782. doi: 10.1136/bmj.38973.444699.0B. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326:1167–1170. doi: 10.1136/bmj.326.7400.1167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ. 2007;335:1202–1205. doi: 10.1136/bmj.39376.447211.BE. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337:867–872. doi: 10.1016/0140-6736(91)90201-y. [DOI] [PubMed] [Google Scholar]
  23. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009. p. MR000006. [DOI] [PMC free article] [PubMed]
  24. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE. 2008;3:e3081. doi: 10.1371/journal.pone.0003081. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J. GRADE guidelines: 5. Rating the quality of evidence—publication bias. J Clin Epidemiol. 2011;64:1277–1282. doi: 10.1016/j.jclinepi.2011.01.011. [DOI] [PubMed] [Google Scholar]
  26. Sterne JAC, Egger M, Moher D. Chapter 10: Addressing reporting biases. Part 2: General methods for Cochrane reviews 2011. Available from www.cochrane-handbook.org .
  27. Mueller K, Meerpohl J, Briel M, Antes G, Ev Elm, Lang B. Detecting, quantifying and adjusting for publication bias in meta-analyses: protocol of a systematic review on methods. Syst Rev. 2013;2:60. doi: 10.1186/2046-4053-2-60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet. 1997;350:326–329. doi: 10.1016/S0140-6736(97)02419-7. [DOI] [PubMed] [Google Scholar]
  29. Centre for Reviews and Dissemination—The University of York . Welcome to the CRD Database: National Institute for Health Research; [updated 21/2/2014]. Available from http://www.crd.york.ac.uk/CRDWeb/AboutPage.asp .
  30. Deeks JJ. Word limits best explain failings of industry supported meta-analyses. BMJ. 2006;333:1021. doi: 10.1136/bmj.39024.372662.1F. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–269. doi: 10.7326/0003-4819-151-4-200908180-00135. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Information
Click here for additional data file. (271KB, doc)

Articles from NPJ Primary Care Respiratory Medicine are provided here courtesy of Primary Care Respiratory Society UK/Macmillan Publishers Limited

RESOURCES