Portability of 245 polygenic scores when derived from the UK Biobank and applied to 9 ancestry groups from the same cohort

Summary

The low portability of polygenic scores (PGSs) across global populations is a major concern that must be addressed before PGSs can be used for everyone in the clinic. Indeed, prediction accuracy has been shown to decay as a function of the genetic distance between the training and test cohorts. However, such cohorts differ not only in their genetic distance but also in their geographical distance and their data collection and assaying, conflating multiple factors. In this study, we examine the extent to which PGSs are transferable between ancestries by deriving polygenic scores for 245 curated traits from the UK Biobank data and applying them in nine ancestry groups from the same cohort. By restricting both training and testing to the UK Biobank data, we reduce the risk of environmental and genotyping confounding from using different cohorts. We define the nine ancestry groups at a sub-continental level, based on a simple, robust, and effective method that we introduce here. We then apply two different predictive methods to derive polygenic scores for all 245 phenotypes and show a systematic and dramatic reduction in portability of PGSs trained using Northwestern European individuals and applied to nine ancestry groups. These analyses demonstrate that prediction already drops off within European ancestries and reduces globally in proportion to genetic distance. Altogether, our study provides unique and robust insights into the PGS portability problem.

Introduction

Ever larger genetic datasets are becoming more readily available. This enables researchers to derive polygenic scores (PGSs), which summarize an individual’s genetic component for a particular trait or disease by aggregating information from many genetic variants into a single score. In human genetics, polygenic scores are usually derived from summary statistics from a large meta-analysis of multiple genome-wide association studies (GWASs) and an ancestry-matched linkage disequilibrium (LD) reference panel.¹ Polygenic scores can also be derived directly from individual-level data when available, i.e., from the genetic and phenotypic information of many individuals.² When using a single individual-level dataset with only moderate sample size, deriving polygenic scores usually results in poor prediction for most phenotypes, e.g., for autoimmune diseases with moderately large effects.^3,4 Fortunately, biobank datasets such as the UK Biobank now link genetic data for half a million individuals with phenotypic data for hundreds of traits and diseases.⁵ Thanks to the availability of these large datasets and to efficient methods recently developed to handle such data,^4,6,7 individual-level data may be used to derive competitive PGSs for hundreds of phenotypes.

A major concern about PGSs is that they usually transfer poorly to other ancestries, e.g., a PGS derived from individuals of European ancestry is not likely to predict as well in individuals of African ancestry. Prediction in another ancestry has been shown to decay with genetic distance to the training population^8,9 and with increasing proportion of admixture with a distant ancestry.^10,11 This portability issue is suspected to be primarily due to differences in LD and allele frequencies between populations, and not so much about differences in effects and positions of causal variants.^9,11 Individual-level data from the UK Biobank offers an opportunity to further investigate this problem of PGS portability in a more controlled setting.^9,12 Indeed, while the UK Biobank data contain genetic information for more than 450K British or European individuals, it also contains the same data for tens of thousands of individuals of non-British ancestry.⁵ Of particular interest, those individuals of diverse ancestries all live in the UK and had their genetic and phenotypic information derived in the same way as people of UK ancestry. Our study design circumvents potential confounding bias that might arise in comparative analyses from independent studies and makes the UK Biobank data very well suited for comparing and evaluating predictive performance of derived PGSs in diverse ancestries and across multiple phenotypes. Indeed, the UK Biobank has been shown to offer a much more controlled setting (compared to published GWAS meta-analyses) in the case of studying (for example) polygenic adaptation.^13,14 Note that these analyses are not completely free of bias since, on average, genotyped variants are more common and imputed variants are more accurately imputed in European ancestries. We also acknowledge that some residual structure may remain when deriving PGSs.¹⁵

To investigate portability of PGSs to other ancestries, we must first define groups of different ancestries from the data. Principal component analysis (PCA) has been widely used to correct for population structure in association studies and has been shown to mirror geography in Europe.^16,17 Due to its popularity, many methods have been developed for efficiently performing PCA^{18, 19, 20} as well as appropriately projecting samples onto a reference PCA space,^20,21 making it possible to perform these analyses for ever increasing datasets. Naturally, PCA has also been used for ancestry inference.^{21, 22, 23} However, among the studies where we have seen PCA used for ancestry inference, there does not seem to be a consensus on what is the most appropriate method for inferring ancestry using PCA. For example, there are divergences on which distance metric to use and the number of PCs to use to compute these distances. The ancestry of an individual can also be inferred based on other approaches, including the ADMIXTURE model, its various extensions, and haplotype-based methods.^{24, 25, 26, 27, 28, 29, 30, 31} However, we focus on PCA here because it is very fast and effective.

In this study, we examine the extent to which PGSs are transferable between ancestries by deriving 245 polygenic scores from the UK Biobank data and applying them in nine ancestry groups from the same cohort. We first propose simple, robust, and effective methods for global ancestry inference and grouping from PCA of genetic data, and we use them to define nine ancestry groups in the UK Biobank data. We then apply a computationally efficient implementation of penalized regression⁴ to derive PGSs for 245 traits using the UK Biobank genetic and phenotypic data only. As an alternative method, we also run LDpred2-auto,³² for which we directly derive the summary statistics from the individual-level data available. We show a dramatically low portability of PGSs from UK ancestry to other ancestries. For example, on average, the phenotypic variance explained by the PGSs is only 64.7% in South Asia (the “India” ancestry group defined here), 48.6% in East Asia (“China”), and 18% in West Africa (“Nigeria”) compared to in individuals of Northwestern European ancestry (“United Kingdom”). These results are presented at a finer scale than the usual continental level, which allows us to show that prediction already drops within Europe, e.g., for Northeast and South Europe (the “Poland” and “Italy” ancestry groups) compared to Northwest Europe. We find that this decay in variance explained by the PGSs is roughly linear in the PC distance to the training population and is remarkably consistent across most phenotypes and for both prediction methods applied. The few exceptions include traits such as hair color, tanning, and some blood measurements. We also explore using more than HapMap3 variants when fitting PGSs, it proves useful when large effects are poorly tagged by HapMap3 variants, e.g., for lipoprotein(a), but not in the general case. We also explore the performance of PGS trained using a mixture of European and non-European ancestry samples, but do not observe any significant gain in prediction here.

Material and methods

Data

We derive polygenic scores for 245 phenotypes using the UK Biobank (UKBB) data only.⁵ We read dosages data from UKBB BGEN files using function snp_readBGEN() of R package bigsnpr.¹⁹ We divide the UKBB data in eight ancestry groups (Note A) and restrict to 437,669 individuals without second-degree relatives (KING kinship <2^−3.5). We also define a ninth ancestry group composed of 1,709 unrelated Ashkenazi (see below). For the variants, we use 1,040,096 HapMap3 variants used in the LD reference provided in Privé et al.³² and that were also present in the iPSYCH2015 data³³ with imputation INFO score larger than 0.6. Even though the iPSYCH data is not used in this study, we plan to use the PGSs derived here for iPSYCH in the future.

To define phenotypes, we first map ICD10 and ICD9 codes (UKBB fields 40001, 40002, 40006, 40013, 41202, 41270, and 41271) to phecodes using R package PheWAS.^34,35 We filter down to 142 phecodes of interest that showed potential genetic signals in the PheWeb results from the SAIGE UKBB GWAS.^36,37 We further filter down to 106 phecodes with sufficient power for penalized regression to include at least a few variants in the predictive models. We then look closely at all 2,408 UKBB fields that we have access to and filter down to defining 111 continuous and 28 binary phenotypes based on manual curation.

Additional data: Genotyped data

For the genotyped data used in some follow-up analyses, we restrict to variants that have been genotyped on both chips used by the UK Biobank, that pass quality control (QC) for all batches and QC for possible mismappings,³⁸ with a minor allele frequency (MAF) larger than 0.01 and imputation INFO score of 1. There are 586,534 such high-quality variants, which we read from the BGEN imputed data so that there is no missing value.

Additional data: 8M+ variants

We also design a larger set of imputed variants to compare against using only HapMap3 variants for prediction. We first restrict to UKBB variants with MAF > 0.01 and INFO > 0.6. We then compile frequencies and imputation INFO scores from other datasets, iPSYCH, and summary statistics for breast cancer, prostate cancer, coronary artery disease, and type 1 diabetes.^{33,39, 40, 41, 42} We restrict to variants with a mean INFO > 0.5 in these other datasets and also compute the median frequency. To exclude potential mismappings in the genotyped data³⁸ that might have propagated to the imputed data, we compare median frequencies in the external data to the ones in UKBB (Figure S20). As we expect these potential errors to be localized around errors in the genotype data (confirmed in Figure S21), we apply a moving-average smoothing on the frequency differences to increase power to detect these errors and also reduce false positives. We define the threshold on these smoothed differences based on visual inspection of their histogram. This is the same method we have previously applied to PC loadings to detect long-range LD regions when computing PCA.^19,20 This results in a set of 8,238,692 variants.

Ashkenazi Jewish ancestry group

First, we refer the reader to Note A on ancestry grouping for the details on how we define the other eight ancestry groups, and also to better understand how we infer the “Ashkenazi Jewish” ancestry group. Briefly, we project the UKBB data onto the PCA space of a reference dataset composed of many Jewish and non-Jewish individuals.⁴³ We then compute the robust center (geometric median) of the Ashkenazi Jewish reference individuals and compute the PC distance to this center for all projected UKBB individuals. Based on visual inspection of the histogram of these distances and on the fact that the closest non-Ashkenazi Jewish reference individual, an Italian Jew (Figure S22), is at distance 12.7, we use a threshold of 12.5 under which to assign to the “Ashkenazi Jewish” ancestry group. 1,709 unrelated UKBB individuals are then assigned to this group. Note that, within the already defined eight ancestry groups, the closest individual to this new group belongs to the Italian group, and is at distance 17.3, so this new Ashkenazi group is not overlapping with any of the other groups defined previously.

Penalized regression

To derive polygenic scores based on individual-level data from the UKBB, we use the fast implementation of penalized linear and logistic regressions from R package bigstatsr.⁴ We have also considered the recently developed R package snpnet for fitting penalized regressions on large genetic data; however, we provide theoretical and empirical evidence that bigstatsr is much faster than snpnet (Note B). Our implementation allows for lasso and elastic-net penalizations; yet, for the sake of simplicity and because the UKBB data is very large, we have decided to only use the lasso penalty.⁴ We recall that fitting a penalized linear regression with lasso penalty corresponds to finding the vector of effects β (also μ and γ) that minimizes

$$L\left(\lambda \right)=\underset{\text{Loss function}}{\underbrace{\Vert y-\left(\mu +G\beta +X\gamma \right){\Vert }_2^2}}+\underset{\text{Penalisation}}{\underbrace{\lambda \Vert \beta {\Vert }_1}},$$

where μ is an intercept, G is the genotype matrix, X is the matrix of covariates, y is the (quantitative) phenotype of interest, and λ is a hyper-parameter that controls the strength of the regularization and needs to be chosen. We use sex (field 22001), age (field 21022), birth date (fields 34 and 52), Townsend deprivation index (field 189), and the first 16 genetic principal components (field 22009),²⁰ as unpenalized covariates when fitting the lasso models.

We have extended our implementation in two ways by allowing for using different penalties for the variants (i.e., having $\sum _j{\lambda }_j\left|{\beta }_j\right|$ instead of $\lambda \Vert \beta {\Vert }_1$). First, this enables us to use a different scaling for genotypes. By default, variants in G are implicitly scaled. By using ${\lambda }_j\propto {\left({\text{SD}}_j\right)}^{\left(\xi -1\right)}$, this effectively scales variant j by dividing it by ${\left({\text{SD}}_j\right)}^{\xi }$ in our implementation. The default uses $\xi =1$ but we also test $\xi =0$ (no scaling) and $\xi =0.5$ (Pareto scaling). We introduce a new parameter power_scale for which the user can provide a vector of values to test; the best value is chosen within the Cross-Model Selection and Averaging (CMSA) procedure.⁴ We also introduce a second parameter, power_adaptive, which can be used to put less penalizition on variants with the largest marginal effects;⁴⁴ we try three values here (0 the default, 0.5, and 1.5) and the best one is also chosen within the CMSA procedure.

LDpred2-auto

Using the individual-level data from the training set in the UK Biobank, we run a linear regression GWAS using function big_univLinReg of R package bigstatsr,¹⁹ accounting for the same covariates as in the penalized regression above. As LD reference, we use the one provided in Privé et al.³² based on UKBB data for European ancestry. We use these summary statistics and this LD reference as input for LDpred2-auto. LDpred2 assumes a point-normal mixture distribution for effect sizes, where only a proportion of causal variants p contributes to the SNP heritability h². In LDpred2-auto, these two parameters are directly estimated from the data.³² We use the sparse option in LDpred2-auto to also obtain a vector of effects that is potentially sparse, i.e., effects of some variants are exactly 0. Also note that, as we use linear regression for all phenotypes, we use the total sample size instead of the effective sample size $\left(4/\left(1/n_{\text{case}}+1/n_{\text{control}}\right)\right)$ for binary phenotypes as input to LDpred2. This means that heritability estimates from both LD score regression and LDpred2-auto must be transformed to the liability scale using both the prevalence in the GWAS and in the population; this can be performed using function coef_to_liab from R package bigsnpr. For simplicity, we assume here that the prevalence in the population is the same as the prevalence in the training set.

New formula used in LDpred2

We also slightly modify the formula used in Privé et al.;³² we have previously used

$$\text{se}{\left({\hat{\gamma }}_j\right)}^2=\frac{{\left(\breve{\mathit{y}}-{\hat{\gamma }}_j{\breve{\mathit{G}}}_{\mathit{j}}\right)}^T\left(\breve{\mathit{y}}-{\hat{\gamma }}_j{\breve{\mathit{G}}}_{\mathit{j}}\right)}{\left(n-K-1\right){\breve{\mathit{G}}}_{\mathit{j}}^T{\breve{\mathit{G}}}_{\mathit{j}}}\approx \frac{{\breve{\mathit{y}}}^T\breve{\mathit{y}}}{n{\breve{\mathit{G}}}_{\mathit{j}}^T{\breve{\mathit{G}}}_{\mathit{j}}}\approx \frac{\text{var}\left(\mathit{y}\right)}{n\text{var}\left({\mathit{G}}_{\mathit{j}}\right)},$$

where ${\hat{\gamma }}_j$ is the marginal effect of variant j, and where $\breve{\mathit{y}}$ and ${\breve{\mathit{G}}}_{\mathit{j}}$ are the vectors of phenotypes and genotypes for variant j residualized from K covariates, e.g., centering them. The first approximation expects ${\hat{\gamma }}_j$ to be small, while the second approximation assumes the effects from covariates are small. However, we have found here that some variants can have very large effects, e.g., one variant explains about 30% of the variance in bilirubin log-concentration. Then, instead we compute

$${\left(\breve{\mathit{y}}-{\hat{\gamma }}_j{\breve{\mathit{G}}}_{\mathit{j}}\right)}^T\left(\breve{\mathit{y}}-{\hat{\gamma }}_j{\breve{\mathit{G}}}_{\mathit{j}}\right)={\breve{\mathit{y}}}^T\breve{\mathit{y}}-2{\hat{\gamma }}_j{\breve{\mathit{G}}}_{\mathit{j}}^T\breve{\mathit{y}}+{\hat{\gamma }}_j^2{\breve{\mathit{G}}}_{\mathit{j}}^T{\breve{\mathit{G}}}_{\mathit{j}}={\breve{\mathit{y}}}^T\breve{\mathit{y}}-{\hat{\gamma }}_j^2{\breve{\mathit{G}}}_{\mathit{j}}^T{\breve{\mathit{G}}}_{\mathit{j}},$$

which now gives

$$\left(n-K-1\right)\text{se}{\left({\hat{\gamma }}_j\right)}^2=\frac{{\breve{\mathit{y}}}^T\breve{\mathit{y}}-{\hat{\gamma }}_j^2{\breve{\mathit{G}}}_{\mathit{j}}^T{\breve{\mathit{G}}}_{\mathit{j}}}{{\breve{\mathit{G}}}_{\mathit{j}}^T{\breve{\mathit{G}}}_{\mathit{j}}}=\frac{{\breve{\mathit{y}}}^T\breve{\mathit{y}}}{{\breve{\mathit{G}}}_{\mathit{j}}^T{\breve{\mathit{G}}}_{\mathit{j}}}-{\hat{\gamma }}_j^2\approx \frac{\text{var}\left(\breve{\mathit{y}}\right)}{\text{var}\left({\mathit{G}}_{\mathit{j}}\right)}-{\hat{\gamma }}_j^2,$$

finally giving (note the added term ${\hat{\gamma }}_j^2$)

$$\text{sd}\left({\mathit{G}}_{\mathit{j}}\right)\approx \frac{\text{sd}\left(\breve{\mathit{y}}\right)}{\sqrt{n\text{se}{\left({\hat{\gamma }}_j\right)}^2+{\hat{\gamma }}_j^2}}.$$ (Equation 1)

Figure S23 shows that the updated formula Equation 1 is better; we now use it in the code of LDpred2, and also recommend using it for the QC procedure proposed in Privé et al.³²

Using more than HapMap3 variants in LDpred2

Here we also run LDpred2 using more than HapMap3 variants, based on a set of 8M+ variants (see above). However, LDpred2 cannot be run on 8M variants because the implementation is quadratic with the number of variants in terms of time and memory requirements. Thus, we employ another strategy consisting in keeping only the 1M most significant variants. To correct for winner’s curse, we employ the maximum likelihood estimator used in Zhong and Prentice⁴⁵ and Shi et al.:⁴⁶

$$Z=Z^{\text{∗}}+\frac{\phi \left(Z^{\text{∗}}-Z_{thr}\right)-\phi \left(-Z^{\text{∗}}-Z_{thr}\right)}{\text{Φ}\left(Z^{\text{∗}}-Z_{thr}\right)+\text{Φ}\left(-Z^{\text{∗}}-Z_{thr}\right)},$$

where φ is the standard normal density function, $\text{Φ}$ is the standard normal cumulative density function, Z is the Z-score obtained from the GWAS, $Z_{thr}$ is the threshold used on (absolute) Z-scores for filtering, and $Z^{\text{∗}}$ is the corrected Z-score that we estimate and use. As input for LDpred2, instead of using β (along with $\text{SE}\left(\beta \right)$ and N), we use ${\beta }^{\text{∗}}=\beta \cdot Z^{\text{∗}}/Z$ where $Z=\beta /\text{SE}\left(\beta \right)$. This is now implemented in function snp_thr_correct of package bigsnpr.

Performance metric

Here we use the partial correlation as the performance metric, which is the correlation between the PGS and the phenotype after they have been both residualized using the covariates used in this paper, i.e., sex, age, birth date, deprivation index, and 16 PCs. To derive 95% confidence intervals for these correlations, we use Fisher’s Z-transformation. We implement this in function pcor of R package bigstatsr and use it here.

Results

Overview of study

Here, we use the UK Biobank (UKBB) data only.⁵ We first infer nine ancestry groups in the UKBB. Then we use 391,124 individuals of Northwestern European ancestry to train polygenic scores (PGSs) for 245 phenotypes (about half being diseases; see categories in Figure S1) based on UKBB individual-level genotypes and phenotypes, and we assess portability of these PGSs in the remaining individuals of diverse ancestries (Table 1). As additional analyses, we also investigate using more variants than the HapMap3 variants used in the main analyses, and we train models using a mixture of multiple ancestries. To derive PGSs in this study, we use two different methods, penalized regression and LDpred2-auto, and finally compare them.

Table 1.

Overview of sets of individuals used in this study

Set	UK1	UK2	UK3	Poland	Italy	Iran	India	China	Caribbean	Nigeria	Ashkenazi Jewish
Training 1	367,063	24,061	–	–	–	–	–	–	–	–	–
Test 1	–	–	20,000	4,136	6,660	1,200	6,331	1,810	2,484	3,924	1,709
Training 2	367,063	–	–	4,136	6,660	1,200	6,331	1,810	–	3,924	–
Test 2	–	–	20,000	–	–	–	–	–	2,484	–	–

In total, 439,378 unrelated individuals are used here. Most analyses in this paper use UK1 + UK2 (391,124 individuals) as training set and the other groups as test sets. Secondary analyses in section “Training with a mixture of ancestries” involve multiple ancestry training and keep only the UK3 and Caribbean groups as test sets; UK2 is removed from the training so that sample size from training 2 is the same as training 1 (391,124 individuals). Note that the names of the first eight ancestry groups we define here refer to the country names from the UK Biobank (field 20115) that we use to define the centers of each ancestry group; therefore, these groups also include individuals from nearby countries. For example, the “United Kingdom” ancestry group also includes many individuals who self-identify as Irish, and the “India” ancestry group also includes many individuals who self-identify as Pakistani (Note A).

Ancestry grouping

We investigate various approaches to classify individuals in ancestry groups based on principal component analysis (PCA) of genome-wide genotype data. Detailed results can be found in the corresponding Note A; we recall main results here. First, we show that (squared) Euclidean distances in the PCA space of genetic data are approximately proportional to F_ST between populations, and we therefore recommend using this simple distance. We also provide evidence that using only two PCs, or even four PCs, is not enough to distinguish between some less-distant populations, and we recommend using all PCs visually capturing some population structure. Then, we use this PCA-based distance to infer ancestry in the UK Biobank and the POPRES datasets. We propose two solutions to do so, either relying on projection of PCs to reference populations such as the 1000 Genomes Project, or by directly using internal data only. We show that these solutions are simple, robust, and effective methods for inferring global ancestry and for grouping genetically homogeneous individuals.

Here, we first use the second solution presented in Note A, relying on PCs computed within the UK Biobank and individual information on the countries of birth, for inferring the first eight ancestry groups presented in Table 1. These groups were chosen on the basis of being distant enough from the other groups, and including enough individuals (e.g., >1,000) to draw meaningful conclusions. Note that the names of the ancestry groups we define here refer to the country names from the UK Biobank (field 20115) that we use to define the centers of each ancestry group; therefore, these groups also include individuals from nearby countries. For example, the “United Kingdom” ancestry group also includes many individuals who self-identify as Irish, and the “India” ancestry group also includes many individuals who self-identify as Pakistani (Note A). Then, for inferring the “Ashkenazi Jewish” ancestry group, we use the first solution, projecting UKBB individuals onto the PCA space of a reference dataset composed of many Jewish and non-Jewish individuals.⁴³ We identify a ninth group of 1,709 unrelated individuals, which is entirely non-overlapping with the other eight groups previously defined (Material and methods). This group is largely overlapping with the 1,719 presumably British Jews identified from IBD segments in Naseri et al.⁴⁷ (personal correspondence with the authors). Finally, we run ADMIXTURE (with k = 8 and k = 5) on 200 individuals from each of the nine ancestry groups defined here.²⁴ The results are consistent with the PCA analysis (Figure 1), e.g., showing that the Caribbean group we define is mostly composed of admixed individuals with mostly African ancestry and some small percentage of European ancestry (Figure S2). Moreover, the other groups we define have distinct ADMIXTURE profiles (consistently with being distinct on PCA), except for the “United Kingdom” and “Poland” ancestry groups, which cannot be distinguished based on this analysis.

Figure 1.

The first eight PC scores of the UK Biobank (field 22009) colored by the homogeneous ancestry group we infer for these individuals

Only 50,000 individuals are represented at random. “NA” means that the corresponding individual is not categorized in any of the nine ancestry groups.

Portability of polygenic scores to other ancestries

Figure 2 presents the results when fitting penalized regression using a training set composed of Northwestern European individuals from the UK Biobank (“United Kingdom,” hereinafter also referred to as “the UK individuals” or “the UK” for simplicity purposes) and testing in nine different ancestry groups from the same cohort (Table 1). Averaged over 245 phenotypes, compared to prediction performance in individuals of Northwestern European ancestry, relative predictive ability in terms of partial-r² (Material and methods) is 93.8% in the “Poland” ancestry group (Northeast Europe), 85.6% in “Italy” (South Europe), 72.2% in “Iran” (Middle East), 64.7% in “India” (South Asia), 48.6% in “China” (East Asia), 25.2% in the “Caribbean,” 18% in “Nigeria” (West Africa), and 85.7% for the Ashkenazi Jewish group. As a follow-up analysis to ensure that this drop in performance in other ancestries is not due to differences in imputation quality across ancestries, we perform the same analysis for 83 of the continuous phenotypes using high-quality genotyped variants only (Material and methods) instead of the (mostly imputed) HapMap3 variants; results are highly consistent (Figure S3). We also run the previous follow-up analysis while removing third-degree relatives, which leaves us with 349,991 individuals for training (instead of 391,124) and 43,631 for testing (instead of 46,545); results are practically unchanged (Figure S4). These results are also very similar when using LDpred2-auto instead of penalized regression for training predictive models for all phenotypes (Figure S5). A few phenotypes deviate from this global trend, e.g., prediction of bilirubin concentration ranges between 0.537 and 0.619 (partial-r) for all ancestries except for “China,” for which it is 0.415 (95% CI: 0.374–0.453, see Material and methods). In contrast, for example for hair and skin color, partial correlations decrease quickly and are not significantly different from 0 for both “China” and “Nigeria,” while of 0.420 (95% CI: 0.409–0.432) for “darker hair” in the “United Kingdom” ancestry group (Figure 2). Overall, relative predictive performance decreases approximately linearly with PC distance to the training set (Figure 3). A similar pattern is observed when computing PCA based on more balanced ancestry groups, as recommended in Privé et al.²⁰ (Figure S6).

Figure 2.

Partial correlation and 95% CI in the UK test set versus in a test set from another ancestry group

Each point represents a phenotype and training has been performed with penalized regression on UK individuals (training 1 in Table 1) and HapMap3 variants. The slope (in blue) is computed using Deming regression accounting for standard errors in both x and y, fixing the intercept at 0. The square of this slope is provided above each plot, which we report as the relative predictive performance compared to testing in the “United Kingdom” ancestry group.

Figure 3.

Relative variance explained compared to the UK versus PC distance from the UK

PC distances are computed using Euclidean distance between geometric medians of the first 16 reported PC scores (field 22009) of each ancestry group. Relative performance values are the ones reported in Figure 2. The slope and standard errors are computed internally by function geom_smooth(method = “lm”) of R package ggplot2.

Using more than HapMap3 variants?

We investigate some of the outlier phenotypes in Figure 2, especially the ones from blood biochemistry which have some variants with large effects. We hypothesize that using a denser set of variants could improve tagging of the causal variants with large effect sizes, resulting in an improved prediction in all ancestries. We focus on “total bilirubin,” “lipoprotein(a)” (lipoA), and “apolipoprotein B” (apoB). We perform a localized GWAS which includes all variants around the most significant variant (hereinafter denoted as “top hit”) from the GWAS in the training set 1 (UK individuals and HapMap3 variants only) in each of the first eight ancestry groups defined here. More precisely, we include all variants with an imputation INFO score larger than 0.3 and within a window of 500 kb from the HapMap3 top hit in the UK; there are approximately 30K such variants for all three phenotypes. For bilirubin, the overall top hit is a HapMap3 variant and explains around 30% of the phenotypic variance (Figure S8). Effects from the three top hits are fairly consistent within all ancestry groups (Figure S9) explaining why genetic prediction is highly consistent in all ancestries, except for “China” (Figure 2), for which these variants are rarer. For lipoA, results are very different across ancestries; HapMap3 variants are far from being the top hits for the UK individuals, where the top HapMap3 variant explains 5% of phenotypic variance compared to 29% for the (non-HapMap3) top hit (Figure 4). Note that this top hit is more than 200 kb away from the HapMap3 top hit from the UK group. Moreover, the three top hits for lipoA do not have very consistent effect sizes across ancestries (Figure S10). Finally, for apoB, effects from the three top hits, which are not part of HapMap3 variants, are fairly consistent across ancestries and explain up to 8.5% of the phenotypic variance (Figures S11 and S12).

Figure 4.

Zoomed Manhattan plot for lipoprotein(a) concentration

The phenotypic variance explained per variant is computed as $r^2=t^2/\left(n+t^2\right)$, where t is the t-score from GWAS and n is the degrees of freedom (the sample size minus the number of variables in the model, i.e., the covariates used in the GWAS, the intercept, and the variant). The GWAS includes all variants with an imputation INFO score larger than 0.3 and within a 500 kb radius around the top hit from the GWAS performed in the UK training set and on the HapMap3 variants, represented by a vertical dotted line.

We then investigate whether the use of a larger set of variants than the HapMap3 set is beneficial; we use more than 8M common variants (Material and methods) and apply LDpred2-auto after restricting to the 1M most significant variants and applying winner’s curse correction (Material and methods). Except for lipoA for which we get a large improvement in predictive accuracy compared to using HapMap3 variants only, it is not beneficial for the other seven phenotypes analyzed here (Figure 5). Remarkably, while the partial correlation for lipoA is about 75% in the UK test set when using this prioritized set of variants, it is still not different from 0 when applied to the “Nigeria” group. For height and BMI, estimated SNP heritability is reduced when using this set of most significant variants only, and all these variants are estimated to be causal, i.e., the estimate of the proportion of causal variants p is 1 (Table S1). As height and BMI are very polygenic traits (p is estimated to be ∼2% and ∼4%, respectively, when using HapMap3 variants), contribution from less significant causal variants is missed due to this thresholding selection. For the three binary phenotypes of breast cancer (phecode: 174.1), prostate cancer (185), and coronary artery disease (411.4), although heritability estimates are larger when using this set of prioritized variants (Table S1), predictive accuracy does not improve compared to when using HapMap3 variants (Figure 5).

Figure 5.

Predictive performance with LDpred2-auto for eight phenotypes, when using either HapMap3 variants or the 1M most significant variants

One phenotype shown in each panel. Bars represent the 95% confidence intervals. Phecode 174.1: breast cancer; 185: prostate cancer; 411.4: coronary artery disease. HM3, HapMap3; top1M, the 1M most significant variants out of more than 8M common variants (see Material and methods).

Training with a mixture of ancestries

We hypothesize that using individuals from diverse ancestries could improve tagging of the causal variants, resulting in an improved prediction in all ancestries. Indeed, power improvements for both association and prediction have been reported when using even a small set of individuals from different ancestries.^11,48,49 Here we use all ancestry groups except for the Caribbean and Ashkenazi for training penalized regressions; we remove the same number of UK individuals to keep the same training sample size as before (training 2 in Table 1). We recall that Caribbean individuals are mostly admixed between African, European, and Native American ancestries,⁵⁰ which are almost all represented here in the training set 2. In Figure S13, we investigate nine phenotypes of interest, either because they are highly studied diseases or are outliers in Figure 2: breast cancer (phecode: 174.1), prostate cancer (186), type 2 diabetes (250.2), hypertension (401), coronary artery disease (411.4), skin tone, total bilirubin concentration, lipoprotein(a) concentration, and years of education. We predict in the test sets from the UK and the Caribbean (test set 2); overall, the predictive performance is highly similar when using this multi-ancestry training compared to when using only UK individuals, in both the UK and the Caribbean target samples. Prediction is only improved for lipoprotein(a) concentration when the mixed ancestry training data is used in application to the Caribbean target data (Figure S13). Discrepancies between our results and results from Márquez-Luna et al.⁵¹ and Cavazos and Witte¹¹ may be explained by the fact that we use the exact same sample size when training with multiple ancestries (by removing some UK individuals; see Table 1), whereas these studies use extra (non-European) individuals, making it hard to know if the improved predictions come from using non-European individuals, or just from using more individuals. We also run the newly developed PRS-CSx method⁴⁹ using individuals from training 2, deriving the GWAS summary statistics from the UK Biobank individual-level data (as for LDpred2-auto). PRS-CSx provides lower predictive performance than using the penalized regression on training 2 for both the UK and Caribbean test sets, except when predicting years of education for both sets as well as “darker skin” and coronary artery disease (phecode 411.4) in the Caribbean test set (Figure S13). Predictive performance of PRS-CSx is particularly lower for traits with large effects (bilirubin and lipoprotein(a) concentrations) and moderate effects (breast and prostate cancers; phecodes 174.1 and 185).

Comparison of predictive models

Penalized regression and LDpred2-auto provide approximately similar predictive performance across all traits and ancestries considered here (Figure S14); there are only four pairs of phenotype-ancestry (out of nearly 2,000 pairs) for which 95% CIs for partial-r from penalized regression and LDpred2 are not overlapping: “615: endometriosis” in the “China” ancestry group with 0.065 (0.0074 to 0.122) versus −0.051 (−0.108 to 0.0068); “hard falling asleep” in UK with −0.0349 (−0.742 to 0.0045) versus 0.071 (0.031 to 0.110); height in UK with 0.634 (0.626 to 0.643) versus 0.613 (0.605 to 0.622); and log-bilirubin in “Nigeria” with 0.546 (0.523 to 0.569) versus 0.475 (0.449 to 0.500). For prediction in UK ancestry, penalized regression tends to provide better predictive performance than LDpred2 for phenotypes for which partial-r > 0.3, and LDpred2 tends to outperform penalized regression for phenotypes harder to predict (Figure S14).

Both methods allow for fitting sparse effects, i.e., some resulting effects are exactly 0. Sparse models may be beneficial because they may be more easily implemented. The sparse option in LDpred2-auto provides similar performance as LDpred2-auto without this option (Figure S15). Sparsity of resulting effects follows a very different pattern for penalized regression compared to LDpred2-auto-sparse. Indeed, penalized regression tends not to include variants if it is uncertain that they have a non-zero effect, i.e., when effects are very small and prediction is difficult (Figure S16). In contrast, LDpred2-auto-sparse tends not to discard variants, only when h² is large enough it sets lots of effects to 0 if p is small (Figure S17). Finally, running each penalized regression model takes between a few minutes and a few days depending on the number of non-zero effects in the resulting model (Figure S18). In contrast, LDpred2-auto should take the same computation time for all phenotypes; it completed under 7 h for most phenotypes (Figure S19).

Discussion

In this paper, we have conducted an extensive assessment of PGS portability across ancestries using hundreds of phenotypes. Our analysis demonstrates a canonical relation between genetic distance and predictive performance for most phenotypes. The reported poor portability is in agreement with three previous studies;^9,52,53 we show a relative predictive performance compared to Europeans of ∼18% for Africans (versus 22%, 42%, and 24%), ∼49% for East Asians (versus 50%, 95%, and 64%), and ∼65% for South Asians (versus 60%, 62.5%, and 72%). However, our results also provide a significant addition to the current literature in many ways. First, we show that the portability issue remains strong even when PGSs are derived and applied in the same cohort. Second, the presented results are averaged over 245 phenotypes, which is much more than what has been typically used, and should capture a broad range of the phenotypic spectrum. Portability results are highly consistent across most phenotypes (with a few exceptions) and could therefore be used to predict the expected loss of accuracy for other phenotypes. Third, we provide this result at a finer scale than the usual continental level by proposing a simple, robust, and effective method for grouping UKBB individuals in nine ancestry groups. This allows us to show, for example, that predictive performance already decreases within Europe with only ∼94% for Northeast Europe and ∼86% for South Europe of the performance reached within Northwest Europe.

We showcase two methods for deriving polygenic scores when large individual-level datasets are available. Although LDpred2-auto is a method based on summary statistics, it provides good predictive performance compared to penalized regression, when applied to individual-level data. Moreover, portability results shown here are similar when using either the individual-level penalized regression or the summary statistics based LDpred2 method. Fitting of penalized models is relatively fast when using 1M HapMap3 variants. We have also tried fitting penalized regression using 8M variants (>3 TB of data); this was possible but took several days for the phenotypes we tried, so we have not investigated this further. To the best of our knowledge, we use the most efficient penalized regression implementation currently available. Recently, Qian et al.⁷ proposed snpnet, a new R package for fitting penalized regressions on large individual-level genetic datasets, but we have found it to be much less efficient than R package bigstatsr on UKBB data (Note B). As for LDpred2, it currently cannot be run using 8M variants, but we show how to use a subset of 1M prioritized variants out of these 8M. Using this new set of variants provides a large improvement in predicting lipoprotein(a) concentration (lipoA), but not for the other seven phenotypes studied in this analysis. This improvement for lipoA is not surprising given that the top HapMap3 variant explains 5% of phenotypic variance compared to 29% for the (non-HapMap3) top hit (Figure 4).

Here we use only the UK Biobank data to fit polygenic scores. We do not use external information such as functional annotations; those could be used to improve the heritability model assumed by predictive methods in order to improve predictive performance.⁵⁴ Moreover, we do not use external summary statistics, which means that polygenic scores derived from large GWAS meta-analyses would probably outperform the ones we derived here. Nevertheless, Albiñana et al.⁵⁵ have shown that an efficient strategy to improve predictive ability of polygenic scores consists in combining two different polygenic scores, one derived using external summary statistics and another one derived using internal individual-level data. Therefore, the polygenic scores we derived here could be combined with polygenic scores derived using external summary statistics; we will release these PGSs publicly and share them in databases such as the PGS Catalog and the Cancer-PRSweb.^56,57

Published: January 6, 2022

Data and code availability

The UK Biobank data are available through a procedure described at https://www.ukbiobank.ac.uk/using-the-resource/. All code used for this paper is available at https://github.com/privefl/UKBB-PGS/tree/main/code. Links to the code used for the Notes A and B are provided there. Code to reproduce our nine ancestry groups is available at https://github.com/privefl/UKBB-PGS#code-to-reproduce-ancestry-groups.

We have extensively used R packages bigstatsr and bigsnpr¹⁹ for analyzing large genetic data, packages from the future framework⁵⁸ for easy scheduling and parallelization of analyses on the HPC cluster, and packages from the tidyverse suite⁵⁹ for shaping and visualizing results. We have also used R package deming for fitting Deming regressions.

Web resources

• bigsnpr, tutorial on LDpred2, https://privefl.github.io/bigsnpr/articles/LDpred2.html

• bigstatsr, tutorial on penalized regressions, https://privefl.github.io/bigstatsr/articles/penalized-regressions.html

• PGS Catalog, effect sizes of PGSs derived here, https://www.pgscatalog.org/publication/PGP000263/

• UK Biobank, quality control information on genetic variants, https://biobank.ctsu.ox.ac.uk/crystal/crystal/auxdata/ukb_snp_qc.txt

• UKBB-PGS, description of the 245 phenotypes used in this study, https://github.com/privefl/UKBB-PGS/blob/main/phenotype-description.xlsx

• UKBB-PGS, other information on the phenotypes (e.g., sample sizes), https://github.com/privefl/UKBB-PGS/blob/main/phenotype-info.xlsx