Pre-selected class-level testing of longitudinal biomarkers reduces required multiple testing corrections to yield novel insights in longitudinal small sample human studies

Abstract

Objectives

Exploratory studies that aim to evaluate novel therapeutic strategies in human cohorts often involve the collection of hundreds of variables measured over time on a small sample of individuals. Stringent error control for testing hypotheses in this setting renders it difficult to identify statistically signification associations. The objective of this study is to demonstrate how leveraging prior information about the biological relationships among variables can increase power for novel discovery.

Methods

We apply the class level association score statistic for longitudinal data (CLASS-LD) as an analysis strategy that complements single variable tests. An example is presented that aims to evaluate the relationships among 14 T-cell and monocyte activation variables measured with CD4 T-cell count over three time points after antiretroviral therapy (n=62).

Results

CLASS-LD using three classes with emphasis on T-cell activation with either classical vs. intermediate/inflammatory monocyte subsets detected associations in two of three classes, while single variable testing detected only one out of the 14 variables considered.

Conclusions

Application of a class-level testing strategy provides an alternative to single immune variables by defining hypotheses based on a collection of variables that share a known underlying biological relationship. Broader use of class-level analysis is expected to increase the available information that can be derived from limited sample clinical studies.

Introduction

Unraveling the complex relationships among cellular level markers and clinical measures of disease progression serves as the foundation of precision medicine applications. However, the discovery of novel associations in immunological investigations remains limited due in part to relatively small sample sizes coupled with the need to control type-1 error rates when testing individually a large number of potentially informative markers. Leveraging information on known biological function offers an opportunity to reframe hypotheses in terms of groups of variables with shared features, while also providing a potentially more powerful strategy for detecting associations. In this manuscript, we apply a class-level testing approach to evaluate simultaneously whether a set of immune variables with established biological relationships is related to a clinically relevant outcome, namely CD4 recovery over time. Our findings underscore the utility of integrative analysis, as a first step before individual variable analysis, particularly in the context of small sample studies involving a larger number of variables.

The data motivating our research arise from cure-directed studies of eradication and functional cure in HIV/AIDS. A typical HIV cure-directed study is initially performed under a pilot design inclusive of hundreds of secondary variables with multiple assessments over time on as few as 12–21 individuals as listed in clinicaltrials.gov (e.g. NCT03617198, NCT03588715). Current practice in this context involves single variable tests and application of a multiple testing adjustment to control type-1 error or false discovery rate. Alternative methods that leverage established information on biologically defined collections of variables that are known to measure discrete and common functionalities, what we term “classes”, are not well-described for immunological markers.

A broad and established literature on class-level testing exists and associated methods have been applied extensively in contexts outside of immunology; however, these methods appear to be underutilized for immunological investigations. For example, several methods exist for testing the association between a complex trait and a set of common genetic variants that are positioned within a defined region of a protein-coding gene (Li et al. 2011, 2015; Liu et al. 2010; Ma, Clark, and Keinan 2013; Neale and Sham 2004; Qian et al. 2016). Further extensions of gene-levels tests have also been described for multivariate (Basu et al. 2013; Van der Sluis et al. 2015) and longitudinal (Qian et al. 2017) traits. In this manuscript, we consider a novel application of this later method, namely the Class Level Association Score Statistic for Longitudinal Data (CLASS-LD) (Qian et al. 2017), to test simultaneously whether a pre-selected collection of immune activation variables are associated with CD4 change over time after initiation of ant-retroviral therapy (ART).

An analysis pipeline that includes testing classes of targeted (pre-selected) variables is suitable to settings with established biological relationships such as present in immunological variables defining immune activation. The variables that constitute an immune activation class can vary depending on the specific context. For example, T cell activation can be evaluated by measuring the expression of numerous biologically independent activation markers at the cell surface, e.g. CD38, CD25, HLA-DR, CD69, CD32 as well as systemic markers, e.g. sCD14 (Abdel-Mohsen et al. 2018; Cockerham et al. 2014; Gandhi et al. 2017). Therefore, a functional class may combine this set of variables. Alternatively, classes can reflect physical compartments, e.g. variables measured in whole blood, plasma or tissue. An integrated analysis of multiple variables connected to a single biological process would allow investigators to capture big picture associations that derive from and transcend single variable analysis.

Here we describe a novel class-level testing strategy define hypotheses based on a collection of variables that share a known underlying biological relationship. We have chosen CD4 cell recovery after HIV suppression by ART to illustrate the impact of class-based analysis when combining frequencies of T-cell and monocyte population change as compared to individual variable analysis. The inference that a reduction in immune activation can impact CD4 recovery is known (Serrano-Villar et al. 2014). Therefore, we reanalyze a reported dataset on persons starting ART with a limited sample size study where longitudinal CD4 recovery, T-cell and monocyte activation were measured. We use this dataset to illustrate how an initial class-based analysis can yield new information otherwise not present if individual variables are tested (Patro et al. 2016; Sierra-Madero et al. 2014). In both cases, a stringent multiple testing adjustment is applied. Importantly, we focus on the setting in which classes are pre-defined based on prior biological knowledge, rendering this distinct from latent class analysis and machine learning methods aimed at defining phenotypic classes without any pre-defined direct input for biological relationships. Our goal is to illustrate the utility of one such strategy, and to describe limitations and opportunities for further work in this area.

Methods

CLASS-LD (Qian et al. 2017) is applied to a study of the association between multiple measures of immune modulation and CD4 recovery with an emphasis on classes combining T-cell plus monocyte variables. The data motivating our work arose from a subset of individuals enrolled in the CADIRIS trial (Sierra-Madero et al. 2014), a study of the effect of CCR5 antagonist Maraviroc on the occurrence of immune reconstitution inflammatory syndrome in HIV (Sierra-Madero et al. 2014). The study enrolled HIV-infected participants in Mexico and South Africa at high risk of developing Immune reconstitution syndrome (<50 CD4+ T cells/ml at entry). To assess if blocking the HIV co-receptor CCR5 resulted in protection from IRIS, participants were randomized to receive either first-line ART treatment, or ART plus Maraviroc for a year. The data used for the current investigation includes PBMCs prepared from peripheral blood collected at baseline, week 12 and week 24 from n=62 South African participants (Patro et al. 2016). The relationship between time varying variables of T-cell and/or monocyte activation after ART and the trajectory of CD4 T-cells over time has not been evaluated previously for these data.

Treating CD4 as the dependent variable, we begin by fitting separate linear mixed effects models for each immune variable including orthogonal polynomials for time to account for the non-linear CD4 trajectories. Formally, this model is given by:

$$Y=X_0\beta +X_s{\gamma }_s+Zb+ϵ$$ (1)

where Y is of length n=62 and represents natural log transformed CD4 count, $X_0$ is the fixed effects design matrix for intercept and polynomials for time, $X_s$ is the fixed effects design matrix for the time varying immunological variable, Z is the random effects design matrix, $b\sim N\left(0,D\right)$, $D={\sigma }_b^2\Omega$, $ϵ\sim N\left(0,{\sigma }_{ϵ}^2{I}_n\right)$, and $Z\subset X_0$. We use a random intercept model such that Z is a column vector of 1’s. Given the limited sample size, we do not include time by immune variable interactions, although the approach could accommodate this more complex model.

Based on the model of Equation (1), we calculate a score statistic corresponding to the null hypothesis of no main effect of the immune variable and CD4. In the model of Equation (1), the parameter γ represents the association between the immune variable and CD4 count and the score statistic corresponds to a test of $H_0:{\gamma }_s=0$ vs. $H_1:{\gamma }_s\ne 0$ without specifying β. Details for calculating this score statistic, denoted S, are provided in (Qian et al. 2017) and it can be shown that this has a ${\chi }^2$ distribution with 1 degree of freedom. Within a biologically-defined class with m variables, our model fitting and testing yields m such statistics that are potentially correlated. A class level statistic is given by a weighted sum of these score statistics across all markers in the class:

$$C=\sum _{j=1}^m{\omega }_j{S}_j\text{\hspace{0.17em},}$$ (2)

In our case, we apply equal weights $\left({\omega }_j=1/m\right)$ in the absence of prior biological information. Finally, we evaluate an approximation to the distribution of this sum (Qian et al. 2017), given by:

$$Pr\left(C>t\right)\approx Pr\left({\chi }_l^2\left(\delta \right)>t^{\text{*}}{\sigma }_{\chi }+{\mu }_{\chi }\right)\text{\hspace{0.17em},}$$ (3)

where ${\mu }_{\chi }=l+\delta$, ${\sigma }_{\chi }=a\sqrt{2}$, $t^{\text{*}}=\left(t-{\mu }_C\right)/{\sigma }_C$, ${\mu }_C=c_1$, ${\sigma }_C=\sqrt{2c_2}$, $s_1=c_3/c_2^{3/2}$, $s_2=c_4/c_2^2$ and $c_k=\text{tr}\left[{\left(\tilde{A}{\Sigma }_Y\right)}^k\right]+k\ast {\mu }_Y^T{\left(\tilde{A}{\Sigma }_Y\right)}^{k-1}\tilde{A}{\mu }_Y$, (k=1, 2, 3, 4) where $A^k=A{\left(A\right)}^{k-1}$ for k ≥ 1, $A^0=I$ and $\text{tr}[\cdot ]$ is the trace of a matrix. Finally, if $s_1^2>s_2$, $a=1/\left(s_1-\sqrt{s_1^2-s_2}\right)$, $\delta =s_1{a}^3-a^2$ and $l=a^2-2\delta$. Otherwise, bif $s_1^2\le s_2$, $a=1/s_1$, $\delta =0$ and $l=c_2^3/c_3^2$. An estimate of c_k is given by replacing μ_Y and ${\Sigma }_Y$ by their estimates under the null model.

In summary, the CLASS-LD algorithm is as follows: (1) Fit separate linear mixed effects models for each immune variable as a predictor (Fitzmaurice, Laird, and Ware 2004); (2) Calculate a score statistic corresponding to the null hypothesis of no association between this variable and CD4; (3) Calculate a weighted sum of score statistics across all immune variables within the pre-defined set; and (4) Estimate the distribution and corresponding p-value of this sum. A summary of how implementation of this approach compares to single variables testing in practice is provided in Figure 2.

Figure 2:

Example of class level testing strategy.

Results

Data example: CADIRIS trial

CD4 recovery over time for the considered subset of individuals enrolled in the CADIRIS trial is illustrated in Figure 3. The light grey lines represent individual level data and the dark line is based on a fitted model with two orthogonal polynomials for time. As expected on average, we see a rapid increase in CD4 after trial initiation from week 0 to week 12, followed by a more gradual increase or plateauing from week 12 to week 24. The 95% interval estimates for the first and second orthogonal polynomials for time are respectively (7.21, 9.35) and (−5.02, −2.90). Our interest is in testing whether time varying values of immune variables associate with this CD4 trajectory where association refers to an overall shift in the fitted curve up or down.

Figure 3:

CD4 trajectory over time.

While decreases in T-cell activation (as defined by CD38+HLADR+) are expected after ART, monocyte activation variables persisted in this cohort. It remains unclear how different monocyte subsets taken together with the change in T cell activation may best inform CD4 recovery. Therefore, we focus on testing the effects of defining classes by monocyte subsets where all classes also included variables for CD4 and CD8 T-cell activation as defined by CD38+HLADR+subsets. The 12 monocyte variables used here were defined by three monocyte groups (Intermediate/inflammatory, Classical, or Total) where four measures of activation were measured (CD169, Tetherin MFA,Zinc MFI, TRAIL). Variables used in each class (all described in (Patro et al. 2016)) are shown in Table 1. An illustration of the three overlapping groups of variables is given in Figure 1. Notably, variables within the same group are expected to change in concert as prior knowledge suggest they measure a similar underlying phenomenon. It is not, however, assumed that the direction of the change is the same.

Table 1:

Class-level testing of sets of immune variables.

Class(a)	Sum	Threshold(b)	df(c)	ncp(d)	p-Value(b)	Adj p-Value(e)
Intermediate	21.399	15.440	4.645	0	6.594E − 03	0.020
Classical	20.982	15.614	4.869	0	7.284E − 03	0.022
Total	13.544	9.578	4.366	0	6.103E − 02	0.183

^(a) Immune classes defined in Figure 1, ^(b) p-value $=Pr\left({\chi }_{\text{df}}^2\left(\text{ncp}\right)>\text{threshold}\right)$, ^(c) df = degrees of freedom, ^(d) ncp = non-centrality parameter, ^(e) Bonferroni adjusted p-value based on correction for three (3) class-level tests.

Figure 1:

Three biologically defined classes of immune variables.

Immune classes were define based on prior known biological associations; they included immunofluorescence markers of T cell activation in combination with markers of activation assessed on intermediate, classical and total monocytes, as indicated. Intermediate = [Inter.CD169, Inter.MFI.Teherin, Inter.MFI.Zinc.Control, Inter.TRAIL, CD4.CD38.HLA.DR, CD8.CD38.HLA.DR]; Classical = [Class.CD169, Class.MFI.Teherin, Class.MFI.Zinc.Control, Class.TRAIL, CD4.CD38.HLA.DR, CD8.CD38.HLA.DR]; and Total = [Total.CD169, Total.MFI.Teherin, Total.MFI.Zinc.Control, Total.TRAIL, CD4.CD38.HLA.DR, CD8.CD38.HLA.DR]

The results of class-level testing are provided in Table 1. This analysis reveals a significant association between both the “Intermediate” and the “Classical” sets of variables on CD4 change over time (adjusted p-values=0.020 and 0.022, respectively). These results suggest that the combined change in activation variables inclusive of T-cells and monocytes contribute to change in CD4 count. In order to further characterize this association, we report in Table 2 the model fitting results for each of the immune variables within these classes. Single variable testing with a multiple testing adjustment leads to identification of one of 14 variables, namely TRAIL expression in the intermediate monocyte subset, as associated with CD4 count change over time. Importantly, class-based analysis additionally detected an association with the classical set where all 6 individual variables when analyzed singly did not reach statistical significance after adjustment for multiple comparisons.

Table 2:

Single immune variable associations.

Class/Marker	Est(a)	SE	Adj 95% CI(b)	${\chi }_1^2$-stat	p-Value	Adj p-Value(b)
T-cell: Intermediate/inflammatory (non classical) monocytes:
CD4.CD38…HLA.DR.	−0.090	0.060	(−0.265, 0.086)	2.222	1.361E − 01	1.000E+00
CD8.CD38…HLA.DR.	0.122	0.061	(−0.056, 0.300)	4.008	4.529E − 02	6.341E − 01
Inter.CD169	−0.167	0.083	(−0.409, 0.076)	4.020	4.495E − 02	6.294E − 01
Inter.TRAIL	−0.142	0.045	(−0.273, −0.011)	9.975	1.586E − 03	2.221E − 02
Inter.MFI.Tetherin	−0.022	0.048	(−0.161, 0.117)	0.216	6.424E − 01	1.000E+00
Inter.MFI.Zinc.Control	0.102	0.046	(−0.034, 0.237)	4.789	2.864E − 02	4.010E − 01
T-cell: Classical monocytes:
CD4.CD38…HLA.DR.	−0.090	0.060	(−0.265, 0.086)	2.222	1.361E − 01	1.000E+00
CD8.CD38…HLA.DR.	0.122	0.061	(−0.056, 0.300)	4.008	4.529E − 02	6.341E − 01
Class.CD169	−0.202	0.083	(−0.443, 0.039)	5.970	1.455E − 02	2.037E − 01
Class.TRAIL	−0.121	0.043	(−0.245, 0.004)	8.005	4.664E − 03	6.530E − 02
Class.MFI.Tetherin	−0.028	0.047	(−0.166, 0.110)	0.350	5.541E − 01	1.000E+00
Class.MFI.Zinc.Control	0.070	0.046	(−0.065, 0.206)	2.304	1.291E − 01	1.000E+00
T-cell: Total monocytes:
CD4.CD38…HLA.DR.	−0.090	0.060	(−0.265, 0.086)	2.222	1.361E − 01	1.000E+00
CD8.CD38…HLA.DR.	0.122	0.061	(−0.056, 0.300)	4.008	4.529E − 02	6.341E − 01
Total.CD169	−0.135	0.078	(−0.363, 0.094)	2.945	8.614E − 02	1.000E+00
Total.TRAIL	−0.084	0.046	(−0.217, 0.049)	3.419	6.445E − 02	9.020E − 01
Total.MFI.Tetherin	0.011	0.048	(−0.128, 0.151)	0.057	8.114E − 01	1.000E+00
Total.MFI.Zinc.Control	0.043	0.047	(−0.094, 0.179)	0.822	3.646E − 01	1.000E+00

^(a) Coefficient estimate representing change in natural log CD4 for one unit increase in corresponding marker, ^(b) Bonferroni adjusted confidence intervals and p-values based on correction for 14 tests (the total number of immune variables considered across the three classes).

Simulation study

To further illustrate the utility of class-level testing, we conduct a simulation study under a range of underlying data generating distributions. In order to reflect a realistic setting, a quantitative outcome is simulated over time according to the observed log CD4 count distribution described in Section 3.1 assuming a sample size of n=62 with three observations per person over the same time intervals and the estimated variance components from fitting a mixed effects model under the null of no immune variable effects. We begin by simulating six immune variables based on a multivariate normal distribution with variance 1 and covariance ranging from 0 (independent) to 0.6. In the first scenario, each of the these immune variables is assumed to have an additive effect on the outcome and a range of effect sizes from 0.1 to 0.2 is considered. In the second scenario, the covariance between immune variables is set to be 0.2, and the number of immune variables associated with the outcome is varied from 1 to 4 while analysis is based on all 6. Two-hundred simulated data sets are generated for each condition. In all cases, power, defined as the proportion of datasets for which an association is detected given that one exists, is reported for the class-level test and for a univariate test with a Bonferroni adjustment.

The simulation study results are provided in Figure 4. Reported power for the univariate test is averaged across all signal variables, i.e. those immune variables that have a true association with the outcome. Here we see that the power of the class-level test is generally higher than the univariate test after adjusting for multiple comparisons. This difference is more pronounced in the low to moderate correlation settings, and when at least half of the immune variables have a signal, i.e. are associated with the outcome. As seen in Figure 4(b), if the number of signal variables is one and the number of variables in the class is 6, then the power of class-level and univariate tests is comparable. However, as the number of single variables increases to 4 of 6, the power of the class-level test is substantially higher than the univariate test.

Figure 4:

Simulation study results.

(a) Power under six signal variables for a range of correlations and effect sizes. Signal variables are assumed to have an additive effect on outcome. (b) Power based on six variables with correlation equal to 0.2, for a range of number of signal variables and effect sizes, assuming an additive effect on outcome.

Discussion

We illustrate how application of a class level test in a small pilot study with multiple secondary variables can reveal novel associations that are not identified with single variable tests. Notably, the class level testing approach represents an alternative to model building which is generally not feasible with a large number of correlated variables. We present a path to interpretation based on use of broader classes before testing individual variables. In contrast to methods aimed at reducing variables to focus on the strongest variable among correlated variables, this application to class assignment looks to join variables expected to be correlated per a shared biological rationale. However, use of this approach is restricted to class building using variables that are recognized to be biologically interrelated, based on prior knowledge (e.g. prior reports that establish a biological relationship). In the absence of a sound basis for inclusion of variables within a class, alternative methods such as unsupervised clustering (Johnson and Wichern 1988) may be appropriate for defining these sets.

Here we chose to test a known relationship (immune activation as inversely related to CD4 T-cell recovery) to illustrate the potential for this method to support interpretation of class testing. The detection of a significant association between the trajectory of CD4 T cell change and classes of immune activation variables, which was not identified using single variable testing, supports the preferability of class level testing in these circumstances. Interestingly, we identify that class analysis of monocyte subsets in conjunction with T-cell activation is potentially more informative to CD4 recovery than evaluating T-cell activation alone. Thus, class level testing has provided additional insights, showing a more complex pattern of variables affecting the change of CD4 T cell count after ART. Future validation of the results require additional longitudinal studies. A survey of studies of immune activation kinetics after ART reveals several examples where class level testing may have allowed additional insights over single variable testing (Cobos Jimenez et al. 2016; Dube et al. 2019; Iannetta et al. 2019; Kasang et al. 2016; Ruggiero et al. 2015; Serrano-Villar et al. 2013, 2016).

With the exception of pathway analysis, most of the current methodologies used to interpret datasets characterized by a large number of variables are based on unsupervised bootstrap methods, and are aimed at discovering previously unknown associations between variables. As such, these methodologies are more useful for wide-net discovery approaches rather than for hypothesis-driven approaches where a large group of variables is assessed based on its anticipated relation with a particular characteristic of interest (e.g. detection of surface molecules that have been associated with various aspects and stages of T-cell activation). By grouping such variables in classes, our approach allows the testing of the main hypothesis (e.g. are T cells activated in a certain context) using method where the stringent adjustment for type 1 error does not penalize the use of multiple tests assessing biologically interrelated outcomes. Following this test, a simple ranking of the variables based on their unadjusted p-value will allow to further characterize which individual variable(s) is responsible for the class effect. A broad literature of alternative methods for determining relative variables importance exist, including penalized regression, such as least absolute shrinkage and selection operator (LASSO) (Tibshirani 1996), and ensemble learners, such as Random Forests (Breiman 2001). Future methods development of this class based approach could include prioritizing variables within classes, including data-driven weighting of variables within classes, baseline corrections and inclusion of interactions. Addressing the impact of interactions may be warranted where a biological class is likely to alter the effect of another class [e.g. microbial translocation markers (class 1) may change the effect of immune activation (class 2) on CD4 T cell recovery].

Limitations of this method are the lack of a test or other quantifiable criteria for defining a relationship for joining variables under a class other than the rationale provided at analysis. Class building would need to be justified a priori based on a pre-defined set of biological interdependencies, which need to be drawn from prior work. However, this need is in keeping with the proposed use for limited sample settings where predefined hypotheses derived from current literature can be tested. Indeed, this limitation could also be noted as part of hypothesis building supporting creation of multiple overlapping classes (as used in this example). We suggest class variables that are joined be clearly linked by previous reports showing shared directional change relative previous to their inclusion into a class for analysis. The limits of this method per number of variables per class or total number of classes to be tested in relation to the number of variables (or time-points) in dataset also remains to be further evaluated.

HIV immune recovery after ART and cure-directed studies represents an area of research where variables selected for analysis include many measures that have been well established in the literature (e.g. immune activation, viral measures, plasma measures). Therefore, there are opportunities to apply this method to gain new insights in such studies. Taken together, application of a class-level testing strategy represents a paradigm shift from analysis of single immune variables to defining hypotheses based on a collection of variables that share a known underlying biological relationship. The use and development of CLASS-LD methods to precede single variable analysis when data already exists between measured variables to support class creation creates an opportunity to gain added insights. Broader use of pre-assigned classes in analysis is expected to increase the available information that can be derived from limited sample clinical studies.