Validation is Critical for GWAS-based Associations

One goal of a genome wide association study (GWAS) is to associate an allele, or multiple alleles, of a single nucleotide polymorphism (SNP) to a clinical outcome or phenotype. The hope is to identify risk alleles in individuals to better predict their genetic predisposition to that clinical outcome and individualize treatment to decrease this risk. To this end there have been many successful associations which are documented at the National Human Genome Research Institute-European Bioinformatics Institute (NHGRI-EBI) catalog of published genome-wide association studies (http://www.ebi.ac.uk/gwas/). Unfortunately there have been many more GWAS based results that have been later found to be false positives. This occurs with far too great a frequency and has even been given its own designation for initial GWAS associations later found to be incorrect: The “Winners Curse”, due in part to an overestimation of the genetic effect of a variant (1). This appears to be the case in the report by Philstrøm et al. entitled “Single nucleotide polymorphisms and long term clinical outcome in renal transplant patients. A validation study,” where two SNPs associated with 5-year creatinine and graft survival in kidney recipients could not be validated in an independent cohort (2).

In what was claimed to be the first GWAS using kidney transplant recipients, O’Brien and colleagues reported the association of two SNPs from approximately 511K SNPs (after data cleaning) and DNA from 326 kidney transplant recipients (3). The first outcome reported was 5-year creatinine levels. The SNP rs6565887, with a p-value of 4.04e⁻⁸, was reported to explain 8.8% of the variance of 5-year creatinine levels and rs3811321, with a p-value of 7.63e⁻⁸, was reported to explain 11.3% of the variance. For this analysis, 63 recipients needed to be dropped due to graft failure before the 5-year time point, leaving 263 recipients with 5-year creatinine levels to be analyzed. Both variants were also found to be predictors of 10-year graft survival. The ‘A’ allele of rs3811321 was associated with increased graft survival (70% vs. 30%, p = 0.004 as was the ‘A’ allele of rs6565887 (70% vs. 45%, p = 0.025).

In this issue, Philstrøm and colleagues attempted to validate these findings for death-censored graft loss and all-cause mortality. Their attempt, using a cohort of over 1,600 recipients, was unsuccessful for both outcomes. A lack of association was found for both SNPs using either univariate or multivariate Cox regression analysis. Additionally, mean serum creatinine levels (mean = 4.6 years after transplantation) were also not found to be significantly associated with these two SNPs in recipients as reported in the initial publication.

The fact that initially reported SNPs in a GWAS failed validation is not new. In response to this common occurrence individuals may be unconcerned believing that erroneous results will eventually be discarded through subsequent studies (4). It is this view that must be must be discarded. Pashler and Harris correctly stated “Unfortunately, however there is every reason to believe that the great majority of errors that do enter the literature will persist uncorrected indefinitely, given current practices. Errors will be propagated through textbooks and review articles, and people interested in a topic will be misinformed for generations.”(4) At least 13 publications have so far cited the results of O’Brien et al., though not all reported specific data from the original publication. For those that did report specific results, their statements ranged from acceptance of the results as fact to stating the need for validation in an independent cohort. None stated the possibility of the study results (associations) being false positives or the low statistical power of the study. O’Brien et al. reported that they had only 80% power to detect an effect size of 14%. The authors stated that replication of their study was needed and would require a larger cohort but this was not related when their results were cited in later publications. Citing results from a GWAS, especially when no validation of those results is being offered, should include a critical evaluation of those results including disclosure of the power of the original study to detect the association. The reporting of erroneous results will continue to occur and is the nature of scientific discovery, but continued citation of false positive results, especially in the absence of a full appraisal of these results, can turn fiction into fact.

The publication by Philstrøm et al., correctly points to the need for greater collaboration among transplant researchers to conduct meta-analysis and cross-validation of GWAS data. Though there are no formal criteria for the validation of GWAS results, typically such a study attempts to validate candidate variants utilizing an independent cohort and similar phenotypes resulting in a statistically significant association (p-value < 0.05 for a single SNP) with a similar effect size and direction. To provide for these types of studies, investigators in transplantation have formed the International Genetics & Translational Research in Transplantation Network (iGeneTRAiN) Consortium and have invited other investigators to join their efforts (5). Through this type of effort, we can have greater confidence in the outcomes related to GWAS produced associations in transplantation research.