Association of Direct-to-Consumer Genome-Wide Disease Risk Estimates and Self-Reported Disease

Abstract

The ongoing controversy surrounding direct-to-consumer (DTC) personal genomic tests intensified last year when the U.S. Government Accountability Office (GAO) released results of an undercover investigation of four companies that offer such testing. Among their findings, they reported that some of their donors received DNA-based predictions that conflicted with their actual medical histories. We aimed to more rigorously evaluate the relationship between DTC genomic risk estimates and self-reported disease by leveraging data from the Scripps Genomic Health Initiative (SGHI). We prospectively collected self-reported personal and family health history data for 3,416 individuals who went on to purchase a commercially available DTC genomic test. For 5 out of 15 total conditions studied, we found that risk estimates from the test were significantly associated with self-reported family and/or personal health history. The 5 conditions, included Graves’ disease, Type 2 Diabetes, Lupus, Alzheimer’s disease, and Restless Leg Syndrome. To further investigate these findings, we ranked each of the 15 conditions based on published heritability estimates and conducted post-hoc power analyses based on the number of individuals in our sample who reported significant histories of each condition. We found that high heritability, coupled with high prevalence in our sample and thus adequate statistical power, explained the pattern of associations observed. Our study represents one of the first evaluations of the relationship between risk estimates from a commercially available DTC personal genomic test and self-reported health histories in the consumers of that test.

Direct-to-Consumer (DTC) personal genomic tests utilize high-throughput genotyping of >500,000 bases of an individual’s DNA and provide an individual with information about their genetic risk for between 20 to over 40 (depending on the company) complex diseases. These tests are highly controversial and there is a lack of empirical data to inform various aspects of the debate. The ongoing controversy surrounding DTC genomic tests intensified last year when the U.S. Government Accountability Office (GAO) released results of an undercover investigation of four companies that offer such testing [Kutz 2010]. As part of their investigation, the GAO selected five donors and sent two DNA samples from each donor to each company. Among their findings, they reported that some of their donors received DNA-based predictions that conflicted with their actual medical histories.

We aimed to more rigorously evaluate the relationship between DTC genomic risk estimates and self-reported disease by leveraging data from the Scripps Genomic Health Initiative (SGHI). In the context of the SGHI, a study originally initiated to assess the impact of personal genomic tests on consumer behavior [Bloss, et al. 2011], we prospectively collected self-reported medical history data for 3,639 individuals, 3,416 of whom went on to undergo testing with and receive results from the Navigenics Health Compass, a commercially available genomic test (Supplemental Figure I). We evaluated the extent to which the risk estimates provided by Navigenics for 15 common, complex conditions were associated with both self-reported (a) family and (b) personal health history of the same conditions.

The study was approved by the Scripps Office for the Protection of Research Subjects and Institutional Review Boards, and the details of our methods have been previously published [Bloss, et al. 2010; Bloss, et al. 2011]. Briefly, prior to undergoing genomic risk assessment, participants were administered a baseline (i.e., pre-risk-disclosure) web-based health assessment that included items to assess demographics, family health-span history, personal/individual health-span history, and attitudes about genetic testing. With regard to family and personal health history, we assessed a total of 51 clinical conditions, which included 9 heart-related, 15 cancer-related, 15 brain/mental health-related, 3 lung-related, and 9 related to other areas (see Supplemental Methods). These clinical conditions were initially selected based on input from and consensus among the clinician members of our research team to represent a comprehensive disease susceptibility assessment for our participants. A total of 15 of the conditions were also included in the context of the Navigenics genomic risk test, and thus it is these 15 conditions we were able to evaluate in the context of the current study. Demographic information for the cohort as a whole is shown in Supplemental Table I.

With respect to assessment of family history (FH), participants were asked to indicate whether or not each of the following individuals in their family had ever had the condition in question: maternal/paternal grandparents, mother/father, maternal/paternal aunt/uncle, sibling, and child. A participant was defined as having a positive FH (+FH) if the condition was reported as present in any of the family members listed. Participants were also asked to indicate whether or not they had a personal history (+PH) of any of the conditions assessed.

Participants then underwent genomic risk testing with the Navigenics Health Compass [Navigenics] (Supplemental Figure II), which they purchased at a subsidized rate. Initially, this genomic risk test provided results for a total of 23 conditions. At a later point in the study, participants were provided results for 5 additional conditions. In addition, the Health Compass returns risk results in different formats, including both estimated lifetime risk (ELTR) for each of the conditions tested, as well as color-coded risk (see Supplemental Table II). Since the various risk formats are highly correlated, we chose to focus our analysis on ELTR, which is prominently featured in the risk report provided back to consumers. Please see the Supplemental Methods for information regarding how Navigenics calculates ELTR. Importantly, Navigenics compares each consumer’s genetic profile to a Caucasian reference population given that most genome-wide association studies (GWAS) to date have been performed in individuals of European ancestry. In other words, when calculating risks, they do not account for the fact that the ancestry of some consumers may differ from Caucasian.

DTC genome-wide risk testing is generally thought of as a form of predictive testing, which is testing performed in asymptomatic individuals to identify genetic susceptibility to future disease. Since FH is known to be a significant and independent risk factor conferring susceptibility for major common, chronic diseases [Valdez, et al. 2010], in our initial analysis, we assessed the extent to which Navigenics ELTR estimates were associated with FH status in our cohort. We also focused on FH because the SGHI includes participants across a wide age range (18–85 years; Supplemental Table I) and thus, a positive +FH across the diseases assessed is much more common in the sample relative to a +PH of disease. We did, however, conduct similar analyses looking at PH for conditions reported as present in more than 1% of participants.

Receiver Operating Curve (ROC) analyses were conducted to assess the extent to which Navigenics ELTR estimates were associated with +FH and +PH in our sample. The 15 conditions we evaluated are listed in the first column of Table I and are sorted according to published heritability estimates (for heritability references see Supplemental Table III). We also list the percentage of our sample for which a +FH and a +PH was reported, as well as published prevalence rates based on European study samples. Results of these analyses suggest that for a subset of 5 conditions with relatively high heritability and for which we had reasonable statistical power to detect an effect (Supplemental Table IV), Navigenics ELTR estimates were significantly associated with FH and PH status. It is likely that we did not see a statistically significant association for the two most highly heritable conditions, Crohn’s disease and Abdominal Aortic Aneurysm, because the prevalence of a +FH for these conditions in our sample was quite low. Even with respect to the significant associations, however, Area Under the Curve (AUC) for significant results ranged from .542 to .675, which is well below that of .85 to .90, the range that would generally be required of a “good” medical test [Pepe 2003].

Table I.

Extent to which Navigenics ELTR estimates predict (a) self-reported FH of disease and (b) self-reported PH of disease

Condition	Heritability (published)	Prevalence (published)b	Percentage in SGHI with Family Hx	Family Hx ROC Analysis)		Percentage in SGHI with Personal Hx	Personal Hx ROC Analysis (conditions >1%)
				AUC (CI)	p		AUC (CI)	p
Crohn’s Disease	0.80	0.0020	2.52	.520 (.456–.583)	.537	0.326	-	-
Abdominal (Aortic) Aneurysm	0.72	0.0265	6.43	.472 (.432–.511)	.159	0.148	-	-
Graves’ Disease/Thyroid	0.64	0.011515	22.0	.578 (.555–.601)	<.001	7.97	.675 (.644–.707)	<.001
Type 2 Diabetes/Diabetes	0.64	0.0665	47.7	.542 (.523–.562)	<.001	2.94	.579 (.521–.638)	.007
Lupus	0.62	0.00045	3.00	.567 (.514–.619)	.027	0.416	-	-
Alzheimer’s Disease	0.62	0.0150	23.3	.562 (.539–.585)	<.001	0.0294	-	-
Restless Leg Syndrome	0.60	0.0270	7.94	.586 (.552–.620)	<.001	5.07	.543 (.501–.585)	.058
Heart Attack (Males)	0.57	0.055	56.8	.517 (.488–.545)	.249	1.12	.568 (.436–.699)	.322
Multiple Sclerosis	0.48	0.000583	3.14	.510 (.456–.565)	.715	0.416	-	-
Prostate Cancer (Males)	0.42	0.0150	17.0	.531 (.493–.569)	.107	1.32	.527 (.399–.656)	.666
Heart Attack (Females)	0.38	0.029	57.6	.499 (.472–.526)	.966	0.166	-	-
Colon Cancer	0.35	0.0027	19.1	.496 (.471–.522)	.765	0.294	-	-
Breast Cancer (Females)	0.27	0.0089	30.8	.502 (.473–.531)	.878	2.83	.524 (.447–.602)	.553
Melanomaa	0.21	0.001762	13.2	.512 (.469–.554)	.585	1.65	.541 (.430–.651)	.476
Lung Cancer	0.14	0.000884	22.7	.492 (.469–.515)	.484	0.118	-	-
Glaucoma	0.13	0.0180	20.1	.516 (.492–.541)	.183	0.884	-	-

^a“Updated” condition; based on subset of sample (see Supplemental Methods)

^bTo evaluate the validity of our family history assessment, we correlated published prevalence rates (which are based on samples of individuals with European ancestry) with family history prevalence rates observed in our sample, which is predominately self-reported Caucasian; this analysis indicated a statistically significant positive correlation (r=.704, p=.002) suggesting that our observed prevalence rates were roughly consistent with published prevalence rates in populations of similar genetic background

Nevertheless, given these statistically significant results, we then aimed to evaluate the extent to which Navigenics ELTR estimates were associated with PH status over and above traditional risk factors, including FH, gender, age, and ethnicity (descriptive statistics shown in Supplemental Tables V and VI). These analyses were conducted for conditions present in more than 1% of our sample and performed via logistic regression. Results are shown in Table II. As shown, for the two conditions with relatively high heritability and prevalence (in our sample), Graves Disease and Type 2 Diabetes, Navigenics ELTR estimates are significantly associated with PH after accounting for FH, gender, age, and ethnicity. Given this finding, we further investigated the “added value” of the genomic test for these conditions by assessing the change in ROC AUC based on genomic testing. After accounting for the traditional disease risk factors listed above, however, the net reclassification based on the addition of the Navigenics ELTR estimates did not achieve statistical significance for either condition (see Supplemental Figure III; Graves Disease: Difference in AUC −0.005, Standard Error (SE) 0.0184, Z-statistic −0.272, p = 0.786; Type 2 Diabetes: Difference in AUC −0.007, SE 0.0325, Z-statistic −0.215, p = 0.830).

Table II.

Extent to which Navigenics ELTR estimates contribute to prediction of self-reported PH of disease independently of traditional risk factors, including FH, gender, age, and ethnicity

Condition (conditions >1%)	Nagelkerke R Square	Navigenics ELTR		Family Historyb		Genderc		Age		Ethnicityd
		OR (95% CI)	p	OR (95% CI)	p	OR (95% CI)	p	OR (95% CI)	p	OR (95% CI)	p
Graves’ Disease/Thyroid	.214	1.194 (1.068–1.335)	.002	5.286 (4.037–6.921)	<.001	2.686 (1.823–3.958)	<.001	1.052 (1.040–1.065)	<.001	.880 (.610–1.270)	.495
Type 2 Diabetes/Diabetes	.119	1.035 (1.016–1.054)	<.001	5.507 (3.265–9.289)	<.001	.598 (.394–.909)	.016	1.050 (1.031–1.069)	<.001	.608 (.369–.999)	.050
Restless Leg Syndrome	.110	1.022 (.957–1.090)	.519	6.749 (4.744–9.600)	<.001	1.329 (.956–1.848)	.091	1.016 (1.003–1.030)	.020	1.916 (1.099–3.342)	.022
Heart Attack (Males)	.171	1.031 (.966–1.100)	.364	1.953 (.678–5.628)	.215	NA	NA	1.123 (1.073–1.176)	<.001	.605 (.129–2.843)	.525
Prostate Cancer (Males)	.255	1.033 (.974–1.096)	.282	2.212 (.751–6.515)	.150	NA	NA	1.146 (1.096–1.199)	<.001	NA	NA
Breast Cancer (Females)	.115	1.043 (.962–1.130)	.306	1.363 (.753–2.465)	.306	NA	NA	1.089 (1.059–1.119)	<.001	1.118 (.454–2.754)	.808
Melanomaa	.121	.932 (.658–1.319)	.690	3.292 (1.380–7.848)	.007	.650 (.281–1.503)	.314	1.071 (1.034–1.110)	<.001	NA	NA

^a“Updated” condition; based on subset of sample (see Supplemental Methods)

^bFamily history coded as follows: 1=+FH, 0=−FH

^cGender coded as follows: 1=Female, 0=Male

^dEthnicity coded as follows: 1=self-reported Caucasian, 0=self-reported non-Caucasian (see Supplemental Methods

We emphasize that our study was not originally designed to address questions pertaining to associations between the genomic test-derived risk estimates and self-reported disease history. As such, a major limitation of this analysis is the small numbers of individuals reporting a +FH and, to a greater extent, +PH of many of the conditions studied. Another limitation is the reliance on self-reported medical histories; however, we note that in this case, the probable imprecision of this method of assessment would likely have biased us against detecting significant associations, and thus if anything, may render our findings more compelling. A strength of our study is the prospective assessment of family and personal health history in a non-clinic-based sample unselected for any particular disease or trait [Ritchie, et al. 2010]. While a limitation of our study is that our study sample is not representative of the broader U.S. population given the high median income and education level of participants, we note that our sample has been shown to be representative of the current population of consumers of genomic tests [Bloss, et al. 2011], which we consider to be a strength. Another strength of this study is that we evaluated a currently marketed, commercially available test.

A remarkable number of commentaries on consumer personal genomic testing have been published to date [Hunter, et al. 2008]. An often-cited concern among them is that different genomic testing companies have been shown to produce inconsistent risk estimates for the same DNA sample [Ng, et al. 2009], as well as that DNA-based predictions from these companies may conflict with consumers’ actual medical history information [Kutz 2010]. In this context it is important to emphasize that Navigenics ELTR estimates are calculated using a unique algorithm [Navigenics 2010] based on a set of genetic markers previously identified through GWAS to be associated with different conditions. Hence, in the current study, beyond issues associated with statistical power, it is possible that associations between risk estimates and self-reported health histories for some diseases may not have been validated due to the use of a non-optimal algorithm or non-optimal set of markers, or that at this point in time, insufficient knowledge exists on which to base a good predictive test for some of the conditions, including a test with “added value” beyond the use of traditional risk factors.

Our study represents one of the first evaluations of the relationship between risk estimates from a commercially available DTC personal genomic test and self-reported health histories from the individuals who purchased the test and thus may plausibly use the results in future health-related decisions. Although the effect sizes we observed are small, our results do provide evidence of association between Navigenics ELTR estimates and self-reported health histories, and more broadly suggest that DTC genomic risk estimates may be most consistent with actual health histories for conditions with high heritability and prevalence. We also observed that the risk estimates for two conditions (Graves Disease and Type 2 Diabetes) are significantly associated with clinical status over and above FH of disease. Although further analysis based on this initial result suggests little “added value” of genomic testing beyond the use of traditional risk factors, in terms of the implications of our findings for consumers of DTC personalized genomic tests, our results indicate that testing may still be useful when traditional risk factor information, namely family history, is not available (e.g., in the case of adoption).