Skip to main content
PLOS Computational Biology logoLink to PLOS Computational Biology
editorial
. 2019 May 16;15(5):e1006922. doi: 10.1371/journal.pcbi.1006922

Ten simple rules for carrying out and writing meta-analyses

Diego A Forero 1,2,*, Sandra Lopez-Leon 3, Yeimy González-Giraldo 4, Pantelis G Bagos 5
Editor: Scott Markel6
PMCID: PMC6521986  PMID: 31095553

Introduction

In the context of evidence-based medicine, meta-analyses provide novel and useful information [1], as they are at the top of the pyramid of evidence and consolidate previous evidence published in multiple previous reports [2]. Meta-analysis is a powerful tool to cumulate and summarize the knowledge in a research field [3]. Because of the significant increase in the published scientific literature in recent years, there has also been an important growth in the number of meta-analyses for a large number of topics [4]. It has been found that meta-analyses are among the types of publications that usually receive a larger number of citations in the biomedical sciences [5,6]. The methods and standards for carrying out meta-analyses have evolved in recent years [79].

Although there are several published articles describing comprehensive guidelines for specific types of meta-analyses, there is still the need for an abridged article with general and updated recommendations for researchers interested in the development of meta-analyses. We present here ten simple rules for carrying out and writing meta-analyses.

Rule 1: Specify the topic and type of the meta-analysis

Considering that a systematic review [10] is fundamental for a meta-analysis, you can use the Population, Intervention, Comparison, Outcome (PICO) model to formulate the research question. It is important to verify that there are no published meta-analyses on the specific topic in order to avoid duplication of efforts [11]. In some cases, an updated meta-analysis in a topic is needed if additional data become available. It is possible to carry out meta-analyses for multiple types of studies, such as epidemiological variables for case-control, cohort, and randomized clinical trials. As observational studies have a larger possibility of having several biases, meta-analyses of these types of designs should take that into account. In addition, there is the possibility to carry out meta-analyses for genetic association studies, gene expression studies, genome-wide association studies (GWASs), or data from animal experiments. It is advisable to preregister the systematic review protocols at the International Prospective Register of Systematic Reviews (PROSPERO; https://www.crd.york.ac.uk/Prospero) database [12]. Keep in mind that an increasing number of journals require registration prior to publication.

Rule 2: Follow available guidelines for different types of meta-analyses

There are several available general guidelines. The first of such efforts were the Quality of Reports of Meta-analyses of Randomized Controlled Trials (QUORUM) [13] and the Meta-analysis of Observational Studies in Epidemiology (MOOSE) statements [14], but currently, the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) [15] has been broadly cited and used. In addition, there have been efforts to develop specific guidelines regarding meta-analyses for clinical studies (Cochrane Handbook; https://training.cochrane.org/handbook), genetic association studies [16], genome-wide expression studies [17], GWASs [18], and animal studies [19].

Rule 3: Establish inclusion criteria and define key variables

You should establish in advance the inclusion (such as type of study, language of publication, among others) and exclusion (such as minimal sample size, among others) criteria. Keep in mind that the current consensus advises against strict criteria concerning language or sample size. You should clearly define the variables that will be extracted from each primary article. Broad inclusion criteria increase heterogeneity between studies, and narrow inclusion criteria can make it difficult to find studies; therefore, a compromise should be found. Prospective meta-analyses, which usually are carried out by international consortia, have the advantage of the possibility of including individual-level data [20].

Rule 4: Carry out a systematic search in different databases and extract key data

You can carry out your systematic search in several bibliographic databases, such as PubMed, Embase, The Cochrane Central Register of Controlled Trials, Scopus, Web of Science, and Google Scholar [21]. Usually, searching in several databases helps to minimize the possibility of failing to identify all published studies [22]. In some specific areas, searching in specialized databases is also worth doing (such as BIOSIS, Cumulative index to Nursing and Allied Health Literature (CINAHL), PsycINFO, Sociological Abstracts, and EconLit, among others). Moreover, in other cases, direct search for the data is also advisable (i.e., Gene Expression Omnibus [GEO] database for gene expression studies) [23]. Usually, the bibliography of review articles might help to identify additional articles and data from other types of documents (such as theses or conference proceedings) that might be included in your meta-analysis. The Web of Science database can be used to identify publications that have cited key articles. Adequate extraction and recording of key data from primary articles are fundamental for carrying out a meta-analysis. Quality assessment of the included studies is also an important issue; it can be used for determining inclusion criteria, sensitivity analysis, or differential weighting of the studies. For example the Jadad scale [24] is frequently used for randomized clinical trials, the Newcastle–Ottawa scale [25] for nonrandomized studies, and QUADAS-2 for the Quality Assessment of Diagnostic Accuracy Studies [26]. It is recommended that these steps be carried out by two researchers in parallel and that discrepancies be resolved by consensus. Nevertheless, the reader must be aware that quality assessment has been criticized, especially when it reduces the studies to a single “quality” score [27,28]. In any case, it is important to avoid the confusion of using guidelines for the reporting of primary studies as scales for the assessment of the quality of included articles [29,30].

Rule 5: Contact authors of primary articles to ask for missing data

It is common that key data are not available in the main text or supplementary files of primary articles [31], leading to the need to contact the authors to ask for missing data. However, the rate of response from authors is lower than expected. There are multiple standards that promote the availability of primary data in published articles, such as the minimum information about a microarray experiment (MIAME) [32] and the STrengthening the REporting of Genetic Association Studies (STREGA) [33]. In some areas, such as genetics, in which it was shown that it is possible to identify an individual using the aggregated statistics from a particular study [34], strict criteria are imposed for data sharing, and specialized permissions might be needed.

Rule 6: Select the best statistical models for your question

For cases in which there is enough primary data of adequate quality for a quantitative summary, there is the option to carry out a meta-analysis. The potential analyst must be warned that in many cases the data are reported in noncompatible forms, so one must be ready to perform various types of transformations. Thankfully, there are methods available for extracting and transforming data regarding continuous variables [3537], 2 × 2 tables [38,39], or survival data [40]. Frequently, meta-analyses are based on fixed-effects or random-effects statistical models [20]. In addition, models based on combining ranks or p-values are also available and can be used in specific cases [4144]. For more complex data, multivariate methods for meta-analysis have been proposed [45,46]. Additional statistical examinations involve sensitivity analyses, metaregressions, subgroup analyses, and calculation of heterogeneity metrics, such as Q or I2 [20]. It is fundamental to assess and, if present, explain the possible sources of heterogeneity. Although random-effects models are suitable for cases of between-studies heterogeneity, the sources of between-studies variation should be identified, and their impact on effect size should be quantified using statistical tests, such as subgroup analyses or metaregression. Publication bias is an important aspect to consider [47], since in many cases negative findings have less probability of being published. Other types of bias, such as the so-called “Proteus phenomenon” [48] or “winner’s curse” [49], are common in some scientific fields, such as genetics, and the approach of cumulative meta-analysis is suggested in order to identify them.

Rule 7: Use available software to carry metastatistics

There are several very user-friendly and freely available programs for carrying out meta-analyses [43,44], either within the framework of a statistical package such as Stata or R or as stand-alone applications. Stata and R [5052] have dozens of routines, mostly user written, that can handle most meta-analysis tasks, even complex analyses such as network meta-analysis and meta-analyses of GWASs and gene expression studies (https://cran.r-project.org/web/views/MetaAnalysis.html; https://www.stata.com/support/faqs/statistics/meta-analysis). There are also stand-alone packages that can be useful for general applications or for specific areas, such as OpenMetaAnalyst [53], NetworkAnalyst [54], JASP [55], MetaGenyo [56], Cochrane RevMan (https://community.cochrane.org/help/tools-and-software/revman-5), EpiSheet (krothman.org/episheet.xls), GWAR [57], GWAMA [58], and METAL [59]. Some of these programs are web services or stand-alone software. In some cases, certain programs can present issues when they are run because of their dependency on other packages.

Rule 8: The records and study report must be complete and transparent

Following published guidelines for meta-analyses guarantees that the manuscript will describe the different steps and methods used, facilitating their transparency and replicability [15]. Data such as search and inclusion criteria, numbers of abstracts screened, and included studies are quite useful, in addition to details of meta-analytical strategies used. An assessment of quality of included studies is also useful [60]. A spreadsheet can be constructed in which every step in the selection criteria is recorded; this will be helpful to construct flow charts. In this context, a flow diagram describing the progression between the different steps is quite useful and might enhance the quality of the meta-analysis [61]. Records will be also useful if, in the future, the meta-analysis needs to be updated. Stating the limitations of the analysis is also important [62].

Rule 9: Provide enough data in your manuscript

A table with complete information about included studies (such as author, year, details of included subjects, DOIs, or PubMed IDs, among others) is quite useful in an article reporting a meta-analysis; it can be included in the main text of the manuscript or as a supplementary file. Software used for carrying out meta-analyses and to generate key graphs, such as forest plots, should be referenced. Summary effect measures, such as a pooled odds ratios or the counts used to generate them, should be always reported, including confidence intervals. It is also possible to generate figures with information from multiple forest plots [63]. In the case of positive findings, plots from sensitivity analyses are quite informative. In more-complex analyses, it is advisable to include in the supplementary files the scripts used to generate the results [64].

Rule 10: Provide context for your findings and suggest future directions

The Discussion section is an important scientific component in a manuscript describing a meta-analysis, as the authors should discuss their current findings in the context of the available scientific literature and existing knowledge [65]. Authors can discuss possible reasons for the positive or negative results of their meta-analysis, provide an interpretation of findings based on available biological or epidemiological evidence, and comment on particular features of individual studies or experimental designs used [66]. As meta-analyses are usually synthesizing the existing evidence from multiple primary studies, which commonly took years and large amounts of funding, authors can recommend key suggestions for conducting and/or reporting future primary studies [67].

As open science is becoming more important around the globe [68,69], adherence to published standards, in addition to the evolution of methods for different meta-analytical applications, will be even more important to carry out meta-analyses of high quality and impact.

Funding Statement

YG-G is supported by a PhD fellowship from Centro de Estudios Interdisciplinarios Básicos y Aplicados CEIBA (Rodolfo Llinás Program). DAF is supported by research grants from Colciencias and VCTI. PGB is partially supported by ELIXIR-GR, the Greek Research Infrastructure for data management and analysis in the biosciences. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1.Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, et al. (2014) How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature. JAMA 312: 171–179. 10.1001/jama.2014.5559 [DOI] [PubMed] [Google Scholar]
  • 2.Garg AX, Hackam D, Tonelli M (2008) Systematic review and meta-analysis: when one study is just not enough. Clin J Am Soc Nephrol 3: 253–260. 10.2215/CJN.01430307 [DOI] [PubMed] [Google Scholar]
  • 3.Greco T, Zangrillo A, Biondi-Zoccai G, Landoni G (2013) Meta-analysis: pitfalls and hints. Heart Lung Vessel 5: 219–225. [PMC free article] [PubMed] [Google Scholar]
  • 4.Ioannidis JP, Chang CQ, Lam TK, Schully SD, Khoury MJ (2013) The geometric increase in meta-analyses from China in the genomic era. PLoS ONE 8: e65602 10.1371/journal.pone.0065602 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Uthman OA, Okwundu CI, Wiysonge CS, Young T, Clarke A (2013) Citation classics in systematic reviews and meta-analyses: who wrote the top 100 most cited articles? PLoS ONE 8: e78517 10.1371/journal.pone.0078517 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Patsopoulos NA, Analatos AA, Ioannidis JP (2005) Relative citation impact of various study designs in the health sciences. JAMA 293: 2362–2366. 10.1001/jama.293.19.2362 [DOI] [PubMed] [Google Scholar]
  • 7.Sutton AJ, Higgins JP (2008) Recent developments in meta-analysis. Stat Med 27: 625–650. 10.1002/sim.2934 [DOI] [PubMed] [Google Scholar]
  • 8.Hedges LV (2015) The early history of meta-analysis. Res Synth Methods 6: 284–286. 10.1002/jrsm.1149 [DOI] [PubMed] [Google Scholar]
  • 9.Glass GV (2015) Meta-analysis at middle age: a personal history. Res Synth Methods 6: 221–231. 10.1002/jrsm.1133 [DOI] [PubMed] [Google Scholar]
  • 10.Pautasso M (2013) Ten simple rules for writing a literature review. PLoS Comput Biol 9: e1003149 10.1371/journal.pcbi.1003149 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Siontis KC, Hernandez-Boussard T, Ioannidis JP (2013) Overlapping meta-analyses on the same topic: survey of published studies. BMJ 347: f4501 10.1136/bmj.f4501 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Booth A, Clarke M, Dooley G, Ghersi D, Moher D, et al. (2013) PROSPERO at one year: an evaluation of its utility. Syst Rev 2: 4 10.1186/2046-4053-2-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, et al. (2000) Improving the Quality of Reports of Meta-Analyses of Randomised Controlled Trials: The QUOROM Statement. Onkologie 23: 597–602. 10.1159/000055014 [DOI] [PubMed] [Google Scholar]
  • 14.Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, et al. (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283: 2008–2012. [DOI] [PubMed] [Google Scholar]
  • 15.Moher D, Liberati A, Tetzlaff J, Altman DG, Group P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6: e1000097 10.1371/journal.pmed.1000097 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Sagoo GS, Little J, Higgins JP (2009) Systematic reviews of genetic association studies. Human Genome Epidemiology Network. PLoS Med 6: e28 10.1371/journal.pmed.1000028 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Ramasamy A, Mondry A, Holmes CC, Altman DG (2008) Key issues in conducting a meta-analysis of gene expression microarray datasets. PLoS Med 5: e184 10.1371/journal.pmed.0050184 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Evangelou E, Ioannidis JP (2013) Meta-analysis methods for genome-wide association studies and beyond. Nat Rev Genet 14: 379–389. 10.1038/nrg3472 [DOI] [PubMed] [Google Scholar]
  • 19.Vesterinen HM, Sena ES, Egan KJ, Hirst TC, Churolov L, et al. (2014) Meta-analysis of data from animal studies: a practical guide. J Neurosci Methods 221: 92–102. 10.1016/j.jneumeth.2013.09.010 [DOI] [PubMed] [Google Scholar]
  • 20.Kavvoura FK, Ioannidis JP (2008) Methods for meta-analysis in genetic association studies: a review of their potential and pitfalls. Hum Genet 123: 1–14. 10.1007/s00439-007-0445-9 [DOI] [PubMed] [Google Scholar]
  • 21.Falagas ME, Pitsouni EI, Malietzis GA, Pappas G (2008) Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J 22: 338–342. 10.1096/fj.07-9492LSF [DOI] [PubMed] [Google Scholar]
  • 22.Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA (2005) Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol 58: 867–873. 10.1016/j.jclinepi.2005.03.004 [DOI] [PubMed] [Google Scholar]
  • 23.Barrett T, Troup DB, Wilhite SE, Ledoux P, Evangelista C, et al. (2011) NCBI GEO: archive for functional genomics data sets—10 years on. Nucleic Acids Res 39: D1005–1010. 10.1093/nar/gkq1184 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, et al. (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 17: 1–12. [DOI] [PubMed] [Google Scholar]
  • 25.Stang A (2010) Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol 25: 603–605. 10.1007/s10654-010-9491-z [DOI] [PubMed] [Google Scholar]
  • 26.Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, et al. (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155: 529–536. 10.7326/0003-4819-155-8-201110180-00009 [DOI] [PubMed] [Google Scholar]
  • 27.Greenland S, O'Rourke K (2001) On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics 2: 463–471. 10.1093/biostatistics/2.4.463 [DOI] [PubMed] [Google Scholar]
  • 28.Juni P, Witschi A, Bloch R, Egger M (1999) The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282: 1054–1060. [DOI] [PubMed] [Google Scholar]
  • 29.da Costa BR, Cevallos M, Altman DG, Rutjes AW, Egger M (2011) Uses and misuses of the STROBE statement: bibliographic study. BMJ Open 1: e000048 10.1136/bmjopen-2010-000048 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Harrison JK, Reid J, Quinn TJ, Shenkin SD (2017) Using quality assessment tools to critically appraise ageing research: a guide for clinicians. Age Ageing 46: 359–365. 10.1093/ageing/afw223 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Ioannidis JP, Allison DB, Ball CA, Coulibaly I, Cui X, et al. (2009) Repeatability of published microarray gene expression analyses. Nat Genet 41: 149–155. 10.1038/ng.295 [DOI] [PubMed] [Google Scholar]
  • 32.Brazma A, Hingamp P, Quackenbush J, Sherlock G, Spellman P, et al. (2001) Minimum information about a microarray experiment (MIAME)-toward standards for microarray data. Nat Genet 29: 365–371. 10.1038/ng1201-365 [DOI] [PubMed] [Google Scholar]
  • 33.Little J, Higgins JP, Ioannidis JP, Moher D, Gagnon F, et al. (2009) STrengthening the REporting of Genetic Association Studies (STREGA): an extension of the STROBE statement. PLoS Med 6: e22 10.1371/journal.pmed.1000022 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Homer N, Szelinger S, Redman M, Duggan D, Tembe W, et al. (2008) Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS Genet 4: e1000167 10.1371/journal.pgen.1000167 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Chene G, Thompson SG (1996) Methods for summarizing the risk associations of quantitative variables in epidemiologic studies in a consistent form. Am J Epidemiol 144: 610–621. [DOI] [PubMed] [Google Scholar]
  • 36.Hozo SP, Djulbegovic B, Hozo I (2005) Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol 5: 13 10.1186/1471-2288-5-13 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.da Costa BR, Rutjes AW, Johnston BC, Reichenbach S, Nuesch E, et al. (2012) Methods to convert continuous outcomes into odds ratios of treatment response and numbers needed to treat: meta-epidemiological study. Int J Epidemiol 41: 1445–1459. 10.1093/ije/dys124 [DOI] [PubMed] [Google Scholar]
  • 38.Di Pietrantonj C (2006) Four-fold table cell frequencies imputation in meta analysis. Stat Med 25: 2299–2322. 10.1002/sim.2287 [DOI] [PubMed] [Google Scholar]
  • 39.Hirji KF, Fagerland MW (2011) Calculating unreported confidence intervals for paired data. BMC Med Res Methodol 11: 66 10.1186/1471-2288-11-66 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Parmar MK, Torri V, Stewart L (1998) Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints. Stat Med 17: 2815–2834. [DOI] [PubMed] [Google Scholar]
  • 41.Begum F, Ghosh D, Tseng GC, Feingold E (2012) Comprehensive literature review and statistical considerations for GWAS meta-analysis. Nucleic Acids Res 40: 3777–3784. 10.1093/nar/gkr1255 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Tseng GC, Ghosh D, Feingold E (2012) Comprehensive literature review and statistical considerations for microarray meta-analysis. Nucleic Acids Res 40: 3785–3799. 10.1093/nar/gkr1265 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Dimou NL, Pantavou KG, Braliou GG, Bagos PG (2018) Multivariate Methods for Meta-Analysis of Genetic Association Studies. Methods Mol Biol 1793: 157–182. 10.1007/978-1-4939-7868-7_11 [DOI] [PubMed] [Google Scholar]
  • 44.Kontou PI, Pavlopoulou A, Bagos PG (2018) Methods of Analysis and Meta-Analysis for Identifying Differentially Expressed Genes. Methods Mol Biol 1793: 183–210. 10.1007/978-1-4939-7868-7_12 [DOI] [PubMed] [Google Scholar]
  • 45.Mavridis D, Salanti G (2013) A practical introduction to multivariate meta-analysis. Stat Methods Med Res 22: 133–158. 10.1177/0962280211432219 [DOI] [PubMed] [Google Scholar]
  • 46.Jackson D, Riley R, White IR (2011) Multivariate meta-analysis: potential and promise. Stat Med 30: 2481–2498. 10.1002/sim.4172 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Rothstein HR, Sutton AJ, Borenstein M (2006) Publication bias in meta-analysis: Prevention, assessment and adjustments. Hoboken, NJ: John Wiley & Sons. [Google Scholar]
  • 48.Ioannidis JP, Trikalinos TA (2005) Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol 58: 543–549. 10.1016/j.jclinepi.2004.10.019 [DOI] [PubMed] [Google Scholar]
  • 49.Kraft P (2008) Curses—winner's and otherwise—in genetic epidemiology. Epidemiology 19: 649–651; discussion 657–648. 10.1097/EDE.0b013e318181b865 [DOI] [PubMed] [Google Scholar]
  • 50.Sterne JA, Bradburn MJ, Egger M (2001) Meta‒Analysis in Stata™ In: Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: meta‐analysis in context. Hoboken, NJ: Wiley; pp. 347–369. [Google Scholar]
  • 51.Quintana DS (2015) From pre-registration to publication: a non-technical primer for conducting a meta-analysis to synthesize correlational data. Front Psychol 6: 1549 10.3389/fpsyg.2015.01549 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Polanin JR, Hennessy EA, Tanner-Smith EE (2017) A review of meta-analysis packages in R. Journal of Educational and Behavioral Statistics 42: 206–242. [Google Scholar]
  • 53.Wallace BC, Schmid CH, Lau J, Trikalinos TA (2009) Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data. BMC Med Res Methodol 9: 80 10.1186/1471-2288-9-80 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Xia J, Benner MJ, Hancock RE (2014) NetworkAnalyst—integrative approaches for protein-protein interaction network analysis and visual exploration. Nucleic Acids Res 42: W167–174. 10.1093/nar/gku443 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Quintana DS, Williams DR (2018) Bayesian alternatives for common null-hypothesis significance tests in psychiatry: a non-technical guide using JASP. BMC Psychiatry 18: 178 10.1186/s12888-018-1761-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Martorell-Marugan J, Toro-Dominguez D, Alarcon-Riquelme ME, Carmona-Saez P (2017) MetaGenyo: a web tool for meta-analysis of genetic association studies. BMC Bioinformatics 18: 563 10.1186/s12859-017-1990-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57.Dimou NL, Tsirigos KD, Elofsson A, Bagos PG (2017) GWAR: robust analysis and meta-analysis of genome-wide association studies. Bioinformatics 33: 1521–1527. 10.1093/bioinformatics/btx008 [DOI] [PubMed] [Google Scholar]
  • 58.Magi R, Morris AP (2010) GWAMA: software for genome-wide association meta-analysis. BMC Bioinformatics 11: 288 10.1186/1471-2105-11-288 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Willer CJ, Li Y, Abecasis GR (2010) METAL: fast and efficient meta-analysis of genomewide association scans. Bioinformatics 26: 2190–2191. 10.1093/bioinformatics/btq340 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Ioannidis JP, Boffetta P, Little J, O'Brien TR, Uitterlinden AG, et al. (2008) Assessment of cumulative evidence on genetic associations: interim guidelines. Int J Epidemiol 37: 120–132. 10.1093/ije/dym159 [DOI] [PubMed] [Google Scholar]
  • 61.Vu-Ngoc H, Elawady SS, Mehyar GM, Abdelhamid AH, Mattar OM, et al. (2018) Quality of flow diagram in systematic review and/or meta-analysis. PLoS ONE 13: e0195955 10.1371/journal.pone.0195955 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Ioannidis JP (2007) Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol 60: 324–329. 10.1016/j.jclinepi.2006.09.011 [DOI] [PubMed] [Google Scholar]
  • 63.Neyeloff JL, Fuchs SC, Moreira LB (2012) Meta-analyses and Forest plots using a microsoft excel spreadsheet: step-by-step guide focusing on descriptive data analysis. BMC Res Notes 5: 52 10.1186/1756-0500-5-52 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Osborne JM, Bernabeu MO, Bruna M, Calderhead B, Cooper J, et al. (2014) Ten simple rules for effective computational research. PLoS Comput Biol 10: e1003506 10.1371/journal.pcbi.1003506 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Russo MW (2007) How to Review a Meta-analysis. Gastroenterol Hepatol (N Y) 3: 637–642. [PMC free article] [PubMed] [Google Scholar]
  • 66.Khan KS, Kunz R, Kleijnen J, Antes G (2003) Five steps to conducting a systematic review. J R Soc Med 96: 118–121. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.Zhang W (2014) Ten simple rules for writing research papers. PLoS Comput Biol 10: e1003453 10.1371/journal.pcbi.1003453 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Wallach JD, Boyack KW, Ioannidis JPA (2018) Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLoS Biol 16: e2006930 10.1371/journal.pbio.2006930 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69.Masum H, Rao A, Good BM, Todd MH, Edwards AM, et al. (2013) Ten simple rules for cultivating open science and collaborative R&D. PLoS Comput Biol 9: e1003244 10.1371/journal.pcbi.1003244 [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from PLoS Computational Biology are provided here courtesy of PLOS

RESOURCES