Likelihood ratio statistics for gene set enrichment in Alzheimer’s disease pathways

Abstract

INTRODUCTION:

The study of Alzheimer’s disease (AD) has revealed biological pathways with implications for disease neuropathology and pathophysiology. Pathway-level effects may be mediated by covariates such as age or sex.

METHODS:

Gene set enrichment methods test hypotheses at the level of biological pathways. We introduce a method for quantifying gene set enrichment ( gsLRT ), which accounts for covariate effects. We test for age and sex interactions with protein expression in the context of AD and compare results between human and mouse species.

RESULTS:

Gene sets identified by gsLRT are validated by previous AD studies. Differences between gsLRT results on mouse and human datasets are observed.

DISCUSSION:

Characterizing biological pathways involved in AD builds on important work involving single gene drivers. Our method highlights commonalities and differences between human AD and mouse models, which may inform the development of higher fidelity models for the study of AD.

1. Background

Strategies for translational research in Alzheimers disease have moved from the characterization of individual variants (SNPs, metabolites, proteins) to the understanding of larger-scale genomic pathways that form the molecular basis of the disease [1–6]. A pathway model of late-onset Alzheimers Disease (LOAD) must integrate numerous genetic, clinical, and environmental factors to more accurately represent the etiology of human LOAD. Relevant factors include: (1) genetic risk factors, e.g. APOE genotype or sex, (2) environmental factors including education level and diet, (3) heterogeneity of biological-behavioral symptoms, and (4) prolonged degenerative processes with pathophysiology beginning years before the onset of detectable symptoms [7] and with accompanying adaptive changes. The motivation for this study is to propose a pathway-based statistical approach to AD that accounts for these factors while informing the development and validation of clinically-relevant mouse models for late-onset or sporadic Alzheimers disease (LOAD).

The use of model systems to identify, describe, and design treatments for human diseases has expanded dramatically over the past 30 years. This increase has kept pace with the technological advancements that have permitted a wide range of advanced approaches to the study of human disease mechanisms. While critical to improving human health, the use of model systems comes with many assumptions. Of primary importance is the assumption that the model system’s pathophysiological pathways are either identical or closely related to those of humans. This assumption leads to the belief that experimental phenomena observed in an animal model, such as a mouse, will forecast similar phenomena in humans. However, clear exceptions exist, which impact the translation of animal model data to humans with disease. These exceptions may require re-evaluation of data generated in mouse studies, redesigning of mouse models, or the use of statistical methodologies that permit evaluation of the differences. In this study, we have developed and tested such a methodology, which permits pathway-level evaluation of differences between mouse and human responses to normal and disease conditions.

Our method uses gene-level data (proteomics, transcriptomics, imaging) collected from human or mouse models to generate significance scores at the level of biological pathways. Informally, the significance scores report the novelty of the observed experimental outcome under the assumption that there is no difference between normal and disease states. Moreover, our approach generates these scores while accounting for age-specific, sex-specific and APOE genotype-specific differences in the human subjects or model systems. It can also test for the significance of the relationship between pathway-level interactions with each of these factors and presence of AD. We validate this method using human proteomic data on Alzheimers disease development and test for interactions with the factors of age, sex and APOE genotype. By applying the same approach to CVN-mouse proteomics data, we are able to compare the pathway significance scores from mouse models with a human-like immune background to those obtained from human Alzheimers patients.

2. Methods

The driver of our pathway-based approach to understanding LOAD in humans and mice is a novel approach to gene set enrichment. Gene set enrichment methods are popular tools for addressing the challenge of finding biologically relevant relationships between sample phenotypes and underlying genomic factors. These methods condense gene-level statistics into gene set scores, each of which corresponds to a group of genes with a common annotation. Gene set annotations may be derived from experiment, computation, or literature-search, though the gene set annotations considered in this article are based on membership to signaling pathways.

Many gene set methods assume that the only piece of information known about a sample is its class label, for instance normal versus AD. However, experimental datasets often include additional sample information, which may be relevant to the particular biological mechanism through which a complex diesease manifests. Motivated by documented age- and sex-driven differences in Alzheimer’s Disease (AD) presentation, we developed a framework for gene set enrichment that takes this information into account. Unlike the method proposed by [8] (ROAST), which relies on the assumption of normally-distributed gene expression profiles, our method (gsLRT) generates likelihood ratio statistics from a pair of nested logistic regression models. The procedure is flexible enough to test for a wide number of complicated gene set hypotheses while adjusting for variation explained by the covariates included in both models.

2.1. Enrichment statistics and p-values

Existing methods that use linear models for gene set enrichment treat each gene expression profile as the response variable in a linear model [8, 9]. They assume that each gene expression profile is normally distributed within categories determined by the diagnosis and experimental design. We remove the assumption of normality by instead treating the diagnosis as the response variable for each of our models. Specifically, we assume a matrix $G\in {ℝ}^{n\times m}$ of measurement values (e.g. mRNA expression) for m genes and n samples, a matrix $X\in {ℝ}^{n\times d}$ of covariates for the samples, and a binary vector $y\in {ℝ}^{n\times 1}$, which codes for the diagnosis (e.g. cognitively normal or AD) for each sample. For each gene j = 1,…,m, we fit the nested models

$${\mathcal{M}}_0:\log \left(\frac{\mathbb{P}\left[y_i=1\right]}{1-\mathbb{P}\left[y_i=1\right]}\right)=x_i{\beta }^j$$ (1)

$${\mathcal{M}}_1:\log \left(\frac{\mathbb{P}\left[y_i=1\right]}{1-\mathbb{P}\left[y_i=1\right]}\right)=x_i{\beta }^j+g_i^j{\alpha }^j$$ (2)

and find the maximum likelihood estimators of the model coefficients ${\widehat{\beta }}_{{\mathcal{M}}_0}^j$, ${\widehat{\beta }}_{{\mathcal{M}}_1}^j$, ${\widehat{\alpha }}_{{\mathcal{M}}_1}^j$.

We define statistics measuring the additional explanatory power of gene measurement profile g_j over that of the fixed covariate matrix X as a difference in model log-likelihoods operating at the gene level, which we define as

$${\mathrm{\Lambda }}^j=2\left[\ell \left({\widehat{\beta }}_{{\mathcal{M}}_1}^j,{\widehat{\alpha }}_{{\mathcal{M}}_1}^j\right)-\ell \left({\widehat{\beta }}_{{\mathcal{M}}_0}^j\right)\right]$$ (3)

where ℓ(·) denotes the log-likelihood of the logistic model. We define enrichment score for gene set γ_k as

$$E{S}^k=\sum _{j\in {\gamma }_k}{\mathrm{\Lambda }}^j$$ (4)

Obtaining appropriate p-values for enrichment scores has been a matter of intense discussion in the literature on gene set testing [10–13]. Most of the discussion has centered around the choice of null hypothesis concerning the relationship among genes. While the assumption that the gene-level statistics Λ^j are independent leads to a computationally efficient testing procedure, correlation among gene expression profiles in a gene set—possibly due to a violation of independence—will give the corresponding enrichment statistic higher variance, producing overly optimistic levels of significance (Figure 1). We choose to report more conservative p-values obtained by permutation sampling, which is computationally expensive. In practice, one may want to use the independence assumption on larger data sets when computational speed and ranking of gene sets are priorities over type I error control.

Figure 1:

Comparison of p-values obtained from the independence assumption test (x-axis) and the permutation test (y-axis) run on 542 gene sets in Alzheimer’s proteomics data (Section 5). Black color indicates that neither test rejects, gold indicates that both tests reject, and light blue indicates that only one test rejects. Note the difference in ranges on the axes: the permutation test is more conservative than the independence assumption test.

The enrichment scores produced by gsLRT quantify the degree to which including the terms in ${\mathcal{M}}_1$ alters the model log-likelihood. They capture how much the inclusion of effects from the genes within a gene set improves the ability to discriminate between diagnoses. They are directionless in the sense that the likelihood ratio test will reject when a significant improvement in the model likelihood is made, regardless of whether the expression profiles within a gene set are positively or negatively associated with the diagnosis.

The permutation p-values report a significance level with respect to the second null hypothesis described by [10], or equivalently the focused testing described by [8]. They represent the significance of the cumulative improvement in model log-likelihood with respect to the genes in a gene set without reference to other genes in the full collection. As such, permutation p-values from a focused test are particularly appropriate for assigning significance to gene sets when many genes in a dataset have a putative association with the diagnosis of interest, as is the case in our Alzheimer’s study. Gene sets with low permutation p-values are unlikely to contain only genes whose measurement levels have no association to the diagnosis.

2.2. Description of datasets used

After conducting simulation benchmarks (Appendix B), we applied gsLRT to the ADNI Biomarkers Consortium CSF Proteomics MRM data set. These data were obtained from the Alzheimers Disease Neuroimaging Initiative (ADNI¹) database (adni.loni.usc.edu). The ADNI was launched in 2003 as a publicprivate partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimers disease (AD). The ADNI study was conducted at multiple sites within North America with retrospective assessment of imaging (MRI and FDG-PET) and biomarkers from cognitively normal controls (CN), MCI subjects, and subjects with AD. MCI subjects defined by the ADNI protocol included subjects with MMSE scores >23, isolated memory impairment based on education adjusted memory scores on the Wechsler Memory Scale Logical Memory III scale and a Clinical Dementia Rating (CDR) [14] global score of 0.5. Study participants with AD were diagnosed using standard criteria (NINCDS/ADRDA criteria for probable AD [15]) and were of mild severity at enrollment, with MMSE scores between 20–26 and CDR scores between 0.5 and 1.0. The cohort in our study was comprised of 152 individuals, 86 classified as cognitively normal and 66 with a diagnosis of AD (following ADNI protocol of NINCDS/ADRDA criteria for probable AD).

Experimental details for the multiplexed proteomics assay including quality control criteria, normalization and the statistical analysis plan are described in [16]. The assay was designed to determine the ability of a panel of peptides in the cerebrospinal fluid (CSF) to discriminate among disease states (normal cognition, MCI and AD), using a targeted mass spectrometry-based assay to qualify the candidate AD biomarkers. The proteomics panel consisted of 567 peptides representing 222 proteins with 2 mass transitions monitored per peptide. The Biomarkers Consortium completed data processing, peptide quantification, mapping of peptides to proteins and normalization [16]. We used the normalized protein intensity data in our analysis.

The cohort of 12 mice used in our cross-species analysis contained 6 wildtype mice and 6 APPSwDI/Nos2^−/− (CVN) mice. Each cohort had 3 males and 3 females with a mean age across cohorts of 56.3 months, SD 4.6. Numerous mouse models have been generated to recreate the brain pathology observed in humans with Alzheimers disease. The most common approach to modeling has been to recreate the dominant AD pathologies, aggregated Abeta peptides and amyloid deposits and tau pathology. To generate these models, others have incorporated into mice mutations in specific genes that lead to amyloid or phospho tau accumulation in human brains. This approach clearly shows the accumulation of AD-like pathological deposits in mouse brain. Unfortunately, recreating human amyloid deposition has not led to the loss of neurons leading to brain volume changes in mice. These critical pathologies of human AD are not observed in the widely used AD mouse models. Thus, the commonly used available mouse models do not recreate human brain disease pathology and are unlikely to provide critical information to understand the underlying pathology of AD in humans.

It has been observed that mouse and human immune responses were different, both in the induction of the NOS2 gene and in the generation of NO by the NOS2 protein. Thus, we used a mouse strain that more closely represented human NOS2 activation and human NO production [17]. When this novel mouse strain was crossed to mice expressing a mutation in APP to generate the APPSwDI/mNos2−/− (CVN-AD) line, this relatively limited, but striking difference in NOS2 and NO production impacted immune-regulated metabolic profiles in the brain [18–20]. These changes lead to loss of neurons, loss of regional connectivity and loss of regional brain volumes, which has been shown using Magnetic Resonance Imaging [21, 22]. Thus, brain changes in the CVN-AD mice closely mimic critical human AD pathology, which other commonly used models do not, that is, neuronal loss leading to significant brain volume changes, and loss of regional connectivity. Because of these structural and pathological commonalities between the APPSwDI/mNos2−/− mice and human AD, we have pushed further in our generation of mouse models by replacing the mouse NOS2 gene with the Human NOS2 gene which recreates a similar profile as our previous model.

The use of the APPSwDI/mNos2−/− model in this study was to compare mice that expressed AD-like pathology in the brain with available human data sets of the same kind. The importance was to uncover functional gene/protein networks that were significantly different from humans as well as to find those areas that were notably similar. The analysis methodology developed here and presented in this manuscript has enabled us to see where this mouse mimics corresponding human data sets and where it fails to mimic them. The data have pointed out very clearly that we have accomplished some of our modeling goals, but are missing other key elements. It shows us where to test and where to modify the genetic/protein landscape further. Overall, at the least, this combined approach of innovative modeling brings us closer to having in hand a more useful, testable model for drug development and translation of drug discovery to humans with AD.

We have used the human CSF for comparison with the mouse brain because CSF is robust surrogate of brain chemistry. The brains interstitial fluid is in direct contact with the CSF and the chemical composition of the CSF reflects the cellular chemical environment in the CNS, including metabolites that are entering or leaving. Because the CSF directly receives metabolites from the entire brain, its chemical composition is an average of all the different brain regions, but as we used whole brain extracts, this is not a major concern for our study. In confirmation of our arguments, [23] obtained comparable results in brain and CSF metabolomes from APPswe/PS1deltaE9 mice, validating the use of CSF as an accurate surrogate of the human brain for use in this study. Moreover, future studies will use analytical methods developed in this publication to compare different stages of disease, using CSF from living human subjects for comparison with animal models. Preparation of mouse brain tissue and subsequent proteomics analysis are each described in Appendix C.

3. Results

We first used gsLRT to test for AD gene set enrichment, including each individual’s age, sex, and APOE genotype as covariates, but not including any interactions between these covariates and protein expression levels. We then tested for AD gene set enrichment with respect to protein expression interactions with sex and age, respectively. For these latter tests, the reference models in the nested logistic regressions were those fitted with only main effects for the covariates and the protein expression profile. The gene sets considered in our analysis were downloaded from MSigDB [24, 25], a repository of annotated gene sets. The gene set collections used were the Hallmark, C2, and C5 collections, which include GO [26, 27], KEGG [28], and REACTOME [29] pathways, among others. We then limited our reference set to include only those gene sets with an overlap of greater than 5 genes and fewer than 100 genes with the ADNI data set, yielding a total of 542 gene sets. Our results therefore extend only to the genes that appear in the filtered gene sets. Below we report the gene set annotations as they appeared in their original collections, but our interpretation focuses on the member genes that overlapped with those included in our study.

The study of sex-specific differences in the pathophysiology and epidemiology of AD has generated a large body of research [30–38]. We are particularly interested in how female biological sex interacts with age and APOE ϵ4 genotypes to affect susceptibility to AD. This is a contentious topic [35, 38], and we believe that addressing the interaction between these factors will help resolve the controversy, and possibly lead to strategies for AD prevention and treatment. We hypothesized that female biological sex and APOE genotype interact at an underlying, immune-regulated metabolic pathway in the brain. Immunity, like sex, age, and APOE genotype, is a primary disease factor in AD and impacts both the onset and pathological features of neurodegenerative events.

The five gene set enrichment results with the lowest permutation p-values for the class of models that did not include interaction terms for age, sex or APOE genotype are reported in Table 1. Importantly, because of the dependence among gene set tests with overlapping membership, the p-values reported here and in the following tables are not adjusted for multiple testing, and therefore do not have theoretical guarantees for control of False Discovery Rate (FDR). A group of related protein signatures involving lipid metabolism show the strongest enrichment. Most of these signatures include several apolipoproteins (APOE, APOB, APOD) and fatty acid binding protein (FABP3) (Figure 2). The signatures range in size from 5–14 proteins, constituting relatively small to moderate size biological networks. Several metabolic signatures are not identified as significantly enriched for this model. These signatures include glucose metabolism (p=0.34), metabolism of carbohydrates (p=0.37) and heme metabolism (p=0.38). The Gene Ontology pathway for locomotion is a good negative control since this pathway is not identified as significantly enriched (p=0.20) despite inclusion of the APOB and APP proteins in the pathway. This pathway consists of 14 genes and the lack of a strong or moderate difference in protein concentration between the AD and control groups for neuronal and neural cell adhesion proteins (NRCAM, NCAM1), L1 cell adhesion (L1CAM) and neurexins (NRXN1, NRXN3) accounts for the lack of significance at the pathway level.

Table 1:

gsLRT results on ADNI data with no interaction term

Gene Set	p
GO_GLYCEROLIPID_METABOLIC_PROCESS	0.0009995
GO_CELLULAR_LIPID_METABOLIC_PROCESS	0.0019990
GO_REGULATION_OF_LIPID_BIOSYNTHETIC_PROCESS	0.0029985
GO_LIPID_CATABOLIC_PROCESS	0.0029985
GO_REGULATION_OF_LIPID_METABOLIC_PROCESS	0.0044978

Figure 2:

Gene-level enrichment statistics for two gene sets with significant enrichment (a, b) and one gene set with no significant level of enrichment (c). The inclusion of FABP3 is one of the primary drivers of significance.

Protein set enrichment results for the model that included an interaction term for sex and protein level are reported in Table 2. The p-values for results including this interaction term assesses the statistical significance of the model coefficient that allows for sex-specific differences in the association between the protein expression levels and the disease state. The gene sets with the strongest association contain complement proteins (C2, C3, C4A, C5, C6, C8B, CFB). Several signatures associated with immune function are identified as statistically significant (p<0.05) for this model including immune effector process (p=0.0075) and the innate immune system (p=0.012). For the models that included interaction terms for age, no gene sets were identified as significantly associated (p<0.05) with the AD phenotype.

Table 2:

gsLRT results on ADNI data with sex interaction

Gene Set	p
GO_IMMUNE_EFFECTOR_PROCESS	0.0074963
GO_REGULATION_OF_PROTEIN_MATURATION	0.0084958
REACTOME_COMPLEMENT_CASCADE	0.0089955
HALLMARK_COAGULATION	0.0094953
REACTOME_INITIAL_TRIGGERING_OF_COMPLEMENT	0.0094953

3.1. Significance of gene sets across species

In order to compare mouse proteomics pathway signatures to those of humans, we applied gsLRT to data from a select population of mice that are useful models of human Alzheimers disease. We applied gsLRT to both the ADNI and mouse data adjusting for age and sex. We then compared the gene set significance results obtained from both datasets, limiting our analysis to only those gene sets corresponding to the intersection of the proteins contained in each dataset.

Interestingly, the mouse proteomics dataset yielded more significant pathways overall than the ADNI dataset. As shown in Figure 3, this may be explained by the fact that the human gene set results were driven by a select few proteins with very large Λ_j values—including Fatty Acid Binding Protein 3 (FABP3) and Neuronal Pentraxin Receptor (NPTXR)—while those for the mouse data were driven by a larger collection of more modest scores for a group of common AD drivers like APOE, APP, CLU, and PRDX6. Hence, pathways involving lipid metabolism were consistently observed as significantly enriched for both species and included regulation of lipid biosynthetic processes, cellular lipid metabolic process, lipid localization and lipid binding, while pathways enriched only in the mouse dataset included those corresponding to immune system processes, aging, and neuronal development. The full table of results is provided as Supplemental Table 2.

Figure 3:

Gene-level enrichment statistics for genes that appear both in the mouse and human proteomics data. The 10 proteins with greatest deviation from equality (dotted line) between species are labeled.

4. Discussion

Application of gsLRT for gene set enrichment in the absence of additional interaction terms (Table 1, Supp. Table 1) confirm known, highly relevant biological signatures and pathways involved in the development of Alzheimers disease [39–43]. Lipid metabolic changes in Alzheimers disease pathophysiology are well replicated. Genes involved in lipid metabolism are identified in large, well-powered genome-wide association studies [42, 43]. Studies of genetic pleiotropy have identified substantial overlap for genes involved with the regulation of serum lipids (LDL, HDL) and Alzheimers disease [44] or cognitive impairment [45]. The gene sets identified as not significantly associated with Alzheimers in Model 1 are not among the strongest genetic effects identified with disease pathophysiology. By contrast, there is compelling phenotypic evidence that these signatures are involved in the disease, most notable for glucose metabolism [46–48] and to a lesser extent for carbohydrate and heme metabolism [49, 50]. If the proteins (or their peptide fragments) that are linked with these signatures are leaked into the CSF, they may appear below the limits of detection in the targeted MS assays.

The inclusion of the protein FABP3 in many of the signatures identified as statistically significant is noteworthy. Increased FABP3 levels have been reported in patients with AD [51], and FABP3 mRNA was increased in microglia from 5XFAD mice compared with C57Bl/6 mice [52]. A specific FABP3 peptide was identified as the most significant peptide differentially expressed between AD and cognitively normal samples identified in the reference study of this proteomic dataset [16]. Lipids serve a variety of roles, from structural to metabolic to signaling [53, 54], and this effect is mediated in part by FABPs [55, 56]. It also has been suggested that one or more of the three FABPs in the brain (FABP3, FABP5, FABP7) may be involved in hippocampal neurogenesis, with a functional defect contributing to hippocampal loss in AD [57]. The lipid metabolism results correlate well with previously reported changes in specific AD biomarkers in the CSF including beta amyloid and tau [58, 59].

It is important to note that our analysis is based on a data set that measures differential protein expression in the CSF. Protein changes in the CSF may not mirror changes in the brain and, specifically, changes associated with glucose metabolism are specific to brain regions and protein changes in metabolism of carbohydrates and heme may be observed in serum [50] in addition to brain [47]. Identification of appropriate controls is both important and challenging for gene set enrichment analysis. Since the proteins chosen for the study had prior support for a role in the development of AD, any significant gene sets will necessarily contain AD-related proteins. However, the finding that a set of lipid metabolic gene sets, specifically (Figure 2a,b), demonstrates enrichment in contrast to the GO locomotion gene set (Figure 2c) suggests that our method of gene set testing confirms a significant role for lipid metabolic signatures involving specific sets of proteins and pathways reported in prior genetic and biochemical studies.

The observation that significant gene set enrichment results were observed for the class of models that included a sex interaction term (Table 2) suggests that the association between AD diagnosis and a subset of protein intensities differs among males and females. The involvement of the complement pathway in AD has strong precedent [60–62]. Prior studies have demonstrated significant sex related differences in complement levels and immune response in healthy individuals [61] and in mice [62]. For example, the clusterin protein is a member of the immune effector process signature, which has the strongest association in the model that includes the sex interaction. The association between clusterin and Alzheimers disease risk is reported in genetic studies [63–66] and in CSF protein level studies [67, 68]. A sex-specific effect of clusterin has been reported as a link between overall decreased bioenergetic metabolism and increased amyloid-related dyshomeostasis in female brains compared with male brains [37].

Since age is a significant risk factor for Alzheimers disease, our prior hypothesis was that we would observe significant enrichment results for the class of models including age interactions. However, the lack of significant results suggests that the interaction between age and protein expression levels does not yield substantial additional explanatory evidence for AD diagnosis beyond that provided by the marginal association between age and AD diagnosis. There are aspects of the study design that may explain the lack of a statistically-significant interaction between age and protein expression. First, The ADNI study includes participants ages 55 to 90 years of age which limits study of changes in dynamic biomarkers such as proteins that occur earlier in life. Second, although the sample size would provide sufficient power for moderate effect size differences between AD and cognitively normal groups, the cohort size may be limiting to detect age-protein concentration level differences of smaller effect sizes. Generalization of the gsLRT statistical analysis to support longitudinal outcome measures would allow more comprehensive assessment of age-specific effects that occur later in life. Prior studies have examined proteomic changes in response to age. One study identified 30 out of 300 proteins in CSF with >20% change in concentrations between older and younger individuals [69], while another study found 248 out of 800 proteins where age differences were greater than twofold over background [70]. The age ranges for these studies were larger than the ADNI study, 22–85 years and 21–85 years respectively. Separating age and disease specific changes in dynamic biomarkers is a challenging problem, requiring large sample sizes to detect interactions between factors.

4.1. Conclusion

Here, we present a statistical approach to derive enrichment scores for biological pathways while accounting for sample covariates in the context of AD. A proof of concept study illustrates cross-species comparisons at the level of pathways or signatures by contrasting the gene-set enrichment results that allowed for interactions between key covariates (age, sex APOE genotype) and protein expression. These pathway-level analyses confirm known genetic interactions and provide unique and informative insights into the relationship between a mouse model of AD and humans with AD. It is clear that overlap in the statistical relevance of specific metabolic pathways can be found. However, it is also clear that, in the data considered for our study, certain pathways clarify critical differences between mouse and human brain. The proof of concept study illustrates that considering the interaction between the covariates and specific analytes that define molecular signatures is important in the development of mouse models that are reliable and consitute a useful model of human neurodegenerative diseases.

There are several caveats and cautions on interpreting the differences between the mouse and human pathway analysis. The genes included in the studies from which these data were drawn focused on specific hypotheses and did not involve an agnostic sampling of the proteome. Measurements were made in two different tissues, brain for the mouse sample and CSF for humans. Sample sizes for the mouse experiments were not adequately powered for stratified analysis by sex or by APOE genotype. The p-values need to be interpreted with caution since there was not a random sampling by the classification factors and since there were multiple comparisons for the number of pathways. The small sample size of the mouse experiments limits the power to detect all but the largest effect sizes. However, there was a substantial overlap (260) in the number of pathways covered by the human and mouse gene sets, supporting an adequate collection of background differences between the classification factors and statistical support for specific signatures is increased by coordinated coregulation or disease-specific differences for a set of proteins, rather than a single protein. For the human data, to be consistent with prior publications based on ADNI data and this specific proteomic dataset, we used the standard diagnostic criteria for AD (NINCDS/ADRDA criteria for probable AD), however future studies may be designed around changes in specific biomarkers including beta amyloid, tau and neurodegenerative markers in accordance with the ATN NIAAA framework [71] and to enable specific comparisons with mouse models that reflect these changes in neuropathology.

Interpretation of dynamic biomarkers such as protein concentrations is complex and cross-species comparisons where there is considerable heterogeneity in the datasets introduces more complexity. As noted in this proof of concept study, there are mechanistic relationships between brain changes and plasma/blood/CSF metabolites that introduce complexities in the human/mouse comparison. Moreover, these studies are cross-sectional, rather than longitudinal; the latter type of study would potentially elucidate age-specific changes and timing/sequencing patterns as a function of disease progression in the proteins concentration. While our study focuses on interactions between critical covariates and the biological signatures, other groups have reported on approaches to compare human and mouse model data, for example to identify of shared cell types across individuals, species, and multiple modalities (gene expression, epigenetic, or spatial data) where non-negative matrix factorization is performed to define a low-dimensional space in which each cell is defined by one set of dataset-specific factors, or metagenes [72] or to examine conserved epigenomic signals in mice and humans in the development of Alzheimers disease [73].

Our future studies will continue to test hypotheses that involve the interactions of four critical factors: age, sex, APOE genotype, and immunity. How these critical factors interact to initiate or accelerate LOAD pathology remains unknown and the approach that we have developed is well-suited to address this question by the field at large. It also facilitates evaluation and comparisons across species, potentially improving the use of animal models to uncover mechanisms of disease and the more rapid development of effective therapeutics for the human population.

Table 3:

gsLRT results on ADNI data with age interaction

Gene Set	p
HALLMARK_ESTROGEN_RESPONSE_LATE	0.0569715
GO_SERINE_HYDROLASE_ACTIVITY	0.1429285
GO_COFACTOR_METABOLIC_PROCESS	0.1504248
GO_BLOOD_MICROPARTICLE	0.1559220
GO_REGULATION_OF_CELLULAR_RESPONSE_TO_STRESS	0.1739130

Highlights.

• Development of novel gene set enrichment method (gsLRT) to discover biological pathways relevant to AD.

• Gene set enrichment analysis of ADNI proteomics data.

• Test for interactions with covariates including age and sex, which may mediate pathway activity.

• Comparison of gene set enrichment between human AD and mouse models.

Research in Context.

1. Systematic review:

The authors reviewed literature describing biological pathways involved with AD. Pathway activity related to AD may be mediated by factors like age or sex.

2. Interpretation:

The manuscript proposes a novel statistical approach to derive enrichment scores for biological pathways while accounting for covariates like age and sex in the context of AD. The pathway-level analyses performed on proteomics data confirm known genetic interactions and provide informative insights into the relationship between a mouse model of AD and humans with AD.

3. Future directions:

Future studies will develop methods to integrate several genomics data modalities to identify animal models with optimal pathway-specific correspondence to human AD, while accounting for relevant covariates.

A. Implementation of gsLRT

A.1. Nested logistic regressions

For each gene in our dataset, we use the ‘glm’ function from the base R package ‘stats’ to fit the logistic regression models described above. We apply the Bartlett correction using the correction factor reported by [74]. The statistic used for testing is then

$$\frac{{\mathrm{\Lambda }}^j}{1+{\alpha }_n/n}\stackrel{d}{\to }{\chi }_1^2$$ (5)

where

$${\mathrm{\Lambda }}^j=2\left[\ell \left({\widehat{\beta }}_{{\mathcal{M}}_1}^j,{\widehat{\alpha }}_{{\mathcal{M}}_1}^j\right)-\ell \left({\widehat{\beta }}_{{\mathcal{M}}_0}^j\right)\right]$$ (6)

$${\alpha }_n=\frac{n}{2}[\mathrm{Tr}(H_{{\mathcal{M}}_1}^2)-\mathrm{Tr}(H_{{\mathcal{M}}_0}^2)]$$ (7)

Here $H_{{\mathcal{M}}_1}$ denotes the diagonal matrix of leverage statistics for the alternative model and $H_{{\mathcal{M}}_0}$ denotes the diagonal matrix of leverage statistics for the null model. For a given model $\mathcal{M}$ and corresponding design matrix $X_{\mathcal{M}}$, the diagonal leverage matrix $H_{\mathcal{M}}$ has entries defined by

$${\left[H_{\mathcal{M}}\right]}_{ii}={\left[X_{\mathcal{M}}{\left(X_{\mathcal{M}}^T{X}_{\mathcal{M}}\right)}^{-1}{X}_{\mathcal{M}}^T\right]}_{ii}$$ (8)

A.2. Permutation

Let ES_k be an enrichment statistic computed for gene set k using the original (non-permuted) data. Let the total number of permutations be B. Then the permutation p-value is given by

$$p_k=\frac{{{\sum }_{b}1}_{E{S}_k^b>E{S}_k}+1}{B+1}$$ (9)

where $E{S}_k^b$ is the value of the enrichment statistic upon permutation of the covariate to be tested. For testing main effects, the gene expression profile is the “covariate” that gets permuted. For testing interaction terms, the column containing the product of the gene expression profile and the interacting covariate is permuted.

B. Simulation study

B.1. Dataset simulation

We used a simulation procedure to produce datasets of the same dimension as the proteomics dataset that we used in our Alzheimer’s investigation. For each simulation trial, we generated gene expression data with 152 samples and 142 genes. The genes were split into 20 disjoint gene sets of random sizes. The expression values for each gene and sample g^j in gene set γ_k were drawn independently from the multivariate normal with covariance matrix drawn from an inverse-Wishart distribution

$$g^j~N\left(0,S^k\right)$$

$$S^k~IW\left(S_0^k,2\left|{\gamma }_k\right|\right)$$

where the entries of $S_0^k$ were set equal to 1 on the diagonal and set equal to u^k, a random number on the unit interval, on the off-diagonal. This construction yields a random correlation structure for genes within gene sets. For each trial, we also generated a covariate matrix X with three continuous covariates drawn i.i.d. from a standard normal distribution and three binary covariates drawn i.i.d. from a Bernoulli distribution with success probability 1/2. Three gene sets were randomly chosen to be associated with the disease outcome through the non-linear relationship

$$y_i~\text{Bernoulli}\left(\text{expit}\left[x_i\beta +\sum _{j\in \mathcal{J}}{g}_i^j{\alpha }^j\right]\right)$$

where expit(x) = e^x/(1 + e^x) and $\mathcal{J}$ denotes the set of genes included in the 3 enriched gene sets. Note that this means that each gene in the selected gene sets has some association to the disease outome, with magnitude depending on the size of the sampled coefficients α. The coefficients β and α were drawn i.i.d. from a standard normal distribution, which produced simulated disease outcome vectors with an average disease prevalence of 47%, comparable to the 43% observed in the ADNI dataset.

B.2. True and false positive rates compared to ROAST

We performed 50 trials following the procedure above and computed p-values using 200 permutations in each trial. The number of permutations was chosen to limit computation time. The three gene sets per trial that were randomly selected to be associated with the disease outcome were the true positive cases; all other gene sets were considered true negatives. We then computed the average true positive rate (TPR) and false positive rate (FPR) for gsLRT at fixed p-value thresholds. For comparison, we also ran the ROAST methodology [8] using 200 rotations in each trial. ROAST achieved better Type I Error control than gsLRT, but was less sensitive on average (Figure 4). We ran a second and third round of trials using exponentially-distributed gene expression profiles and t-distributed gene expression profiles with five degrees of freedom. For a fixed FPR, gsLRT achieved comparable or modestly higher true positive rates than ROAST in all scenarios. The improvement was most marked in the region of low Type I Error (FPR < 0.1).

Figure 4:

Evaluation of gsLRT’s performance on simulated gene expression data versus that of ROAST methodology. On average, gsLRT (darker colors) achieves a higher true positive rate than ROAST (lighter colors) at each fixed false positive rate when evaluated on normally distributed gene expression profiles (red) exponentially distributed gene expression profiles (blue) and t-distributed gene expression profiles (green).

B.3. Validation under null

We modified the simulation setup as described above in order to sample from the null distribution that gsLRT is designed to test against. Specifically, we ran simulations with the same dimensions as above, but sampled gene expression values for the samples within gene set k from

$$g^j~N\left(0,I_{\left|{\gamma }_k\right|}\right)$$

This reflects the independence assumption assumed by the gsLRT chi squared test (though this is not necessary for the permutation test). We also sampled the disease outcome independently of the gene expression profiles and covariates

$$y^i~\text{Bernoulli}(0.2)$$

so that the simulated data reflected the null hypothesis that the addition of the gene expression profiles offered no additional explanatory power. The empirical false positive rate at level 0.05 was observed to be 0.0479. In addition, we observed no bias in significance level as relates to the size of gene set (correlation between gene set size and gsLRT p-value was −0.02137, CI = [−0.09835,0.05586]).

Figure 5:

Evaluation of gsLRT’s performance on simulated gene expression data with no relationship to disease outcome (i.e. data conforming to the null model assumed by the gsLRT test). In addition to controlling Type I error at the nominal level (0.0479 vs. 0.05), the gsLRT p-values show no bias towards assigning significance to gene sets of larger size. The blue line shows the line of best-fit, whose slope is proportional to the correlation between the gene set size and −log₁₀(p).

C. Proteomics Methods

C.1. Brain tissue preparation

Brain tissue samples stored in 1.5 mL tubes were delivered to the Duke Proteomics & Metabolomics Core Facility (n = 6 per genotype). 0.5% w/v ALS-1 surfactant in 50 mM ammonium bicarbonate (AmBic) was added to each sample at a volume of 10 uL/mg wet weight of tissue. Tissue homogenization and cell lysis was performed with probe sonication (Misonix) over three pulses at power level 3 for 5 seconds each with cooling on ice between pulses. A 5 uL aliquot of homogenate was diluted 25x in AmBic for determination of protein content by Bradford assay. Based on Bradford results, samples were 0.7±0.2 mg protein / mg tissue. Following normalization (100 μg protein at 1 mg/mL protein in 0.5% ALS-1/AmBic), samples were reduced with 10 mM dithiothreitol (DTT) at 80° C with shaking for 15 minutes, alkylated with 20 mM iodoacetamide (IAA) at room temperature in the dark for 30 minutes, and digested with 2 μg sequencing grade modified trypsin (Promega) overnight at 37° C with shaking. Digestion was stopped with the addition of 12 μL 10/20/70 v/v/v TFA/MeCN/H₂O and heating at 60° C for 2 hours. Diluted further with 1/2/97 v/v/v TFA/MeCN/H₂O for a final digested protein concentration of 0.5 ug/uL. A pool of all samples (Study Pool QC, SPQC) was created from equal volume of each sample, and analyzed at regular intervals throughout the study to allow observation of any experimental drift.

C.2. Proteomics Analysis

The samples were analyzed using a nanoAcquity UPLC system (Waters) coupled to a Q Exactive HF Orbitrap high-resolution accurate-mass tandem mass spectrometer (Thermo Scientific) via a nanoelectrospray ionization source. Each sample was analyzed once, and the SPQC was analyzed approximately every 6 samples. Briefly, the sample was first trapped and desalted on a Symmetry C18 180 um x 20 mm trapping column (5 uL/min at 99.8/0.1/0.1 v/v water/acetonitrile/formic acid), then the analytical separation was performed using a 1.7 um Acquity HSS T3 C18 75 um x 250 mm column (Waters). The peptides on the column were eluted using a 90-minute gradient of 5–40% acetonitrile with 0.1% formic acid at a flow rate of 400 nanoliters/minute (nL/min) with a column temperature of 55° C. Data collection on the Q Exactive HF mass spectrometer was performed in a data-dependent MS/MS manner, using a 120,000 resolution precursor ion (MS1) scan followed by MS/MS (MS2) of the top 12 most abundant ions at 30,000 resolution. MS1 was accomplished using an automatic gain control (AGC) target of 3e6 ions and mass accumulation time of up to 50 msec. MS2 used AGC target of 5e4 ions, up to 45 msec maxiumum ion accumulation, 1.2 m/z isolation window, 27V normalized collision energy, and 20 sec dynamic exclusion.

Following the analyses, the data was imported into Rosetta Elucidator v 4.0 (Rosetta Biosoftware, Inc.), and all LC-MS files were aligned based on the accurate mass and retention time of detection ions (“features”) using a PeakTeller algorithm (Elucidator). The relative peptide abundance was calculated based on area-under-the-curve (AUC) of aligned features across all runs.

The MS/MS data was searched against a custom built database based on the SwissProt database with Mus musculus taxonomy (downloaded April 28, 2017) with additional proteins, including yeast ADH1_YEAST (surrogate standard), ALBU_BOVIN (contaminant), APOE_HUMAN (genetic substitution), and additional mutated proteins expressed in the mice with sequences provided by the investigators, were also included in the custom database. An equal number of reversed-sequence decoys were appended to this forward DB for false discovery rate determination. A total of 3084 proteins were quantified, and 2118 (69%) proteins were quantified with 2 or more peptides (Supplementary Table 3).

Data availability

ADNI data are available with approved data access request from http://adni.loni.usc.edu/.