Biomarker Combinations for Diagnosis and Prognosis in Multicenter Studies: Principles and Methods

Abstract

Many investigators are interested in combining biomarkers to predict a binary outcome or detect underlying disease. This endeavor is complicated by the fact that many biomarker studies involve data from multiple centers. Depending upon the relationship between center, the biomarkers, and the target of prediction, care must be taken when constructing and evaluating combinations of biomarkers. We introduce a taxonomy to describe the role of center and consider how a biomarker combination should be constructed and evaluated. We show that ignoring center, which is frequently done by clinical researchers, is often not appropriate. The limited statistical literature proposes using random intercept logistic regression models, an approach that we demonstrate is generally inadequate and may be misleading. We instead propose using fixed intercept logistic regression, which appropriately accounts for center without relying on untenable assumptions. After constructing the biomarker combination, we recommend using performance measures that account for the multicenter nature of the data, namely the center-adjusted area under the receiver operating characteristic curve. We apply these methods to data from a multicenter study of acute kidney injury after cardiac surgery. Appropriately accounting for center, both in construction and evaluation, may increase the likelihood of identifying clinically useful biomarker combinations.

1. Introduction

Biomedical investigations are often conducted in multiple centers (e.g., hospitals, clinics, providers). For etiologic and therapeutic studies, there is a substantial literature on the challenges of a multicenter study design. These challenges include correlations among observations from the same center and the effect of differences across centers.¹ The literature on multicenter studies is especially extensive for randomized trials, where the need for careful design and analysis of such studies is widely acknowledged.¹

Multicenter biomarker studies are increasingly common as investigators seek to increase power and generalizability (e.g., Feldstein et al.², Degos et al.³, and Nickolas et al.⁴). However, in contrast to randomized trials, the literature on multicenter biomarker studies is small. As a cause or consequence of this, the challenges and issues posed by a multicenter design appear not to be widely appreciated among biomarker researchers. Furthermore, most biomarker studies measure many biomarkers. Since biomarkers often have only modest individual performance, investigators are usually interested in constructing combinations of biomarkers. A multicenter study design can have implications for both the construction and evaluation of biomarker combinations.

Center plays a unique role in biomarker studies, where the goal is generally prediction. Center may be associated with the outcome one wants to predict, yet it cannot be used as a predictor. The reason is that center does not generalize to patients from centers not in the study, so a prediction instrument that used center as a predictor would not be broadly applicable. Recognizing this situation, it seems some biomarker investigators decide to simply ignore the fact that their data come from multiple centers. As we will demonstrate, ignoring center can produce misleading or undesirable results. Although center cannot be used as a predictor, it generally must be accounted for. However, not all methods for accounting for center are suitable for biomarker studies, and we will illustrate shortcomings with some existing methods.

We will consider the role that center can play in multicenter biomarker studies, including proposing a taxonomy that distinguishes different ways that center can be important and providing guidance to researchers on identifying the role center may play in their studies. We assess the impact of ignoring center and evaluate existing approaches for accounting for center in biomarker studies. Finally, we propose suitable methods for constructing and evaluating biomarker combinations using data from multiple centers. We restrict attention to biomarkers that will be used to identify individuals likely to have (in the diagnostic setting) or develop (in the prognostic setting) some clinical outcome; such biomarkers are sometimes referred to as “prognostic” or “diagnostic” biomarkers, as opposed to biomarkers used to predict response to treatment, which are often called “predictive” biomarkers.

This work was motivated by the Translational Research Investigating Biomarker Endpoints in Acute Kidney Injury (TRIBE-AKI) study. The TRIBE-AKI study involves 1219 cardiac surgery patients at six centers in North America.⁵ The participants were followed for diagnosis of post-operative acute kidney injury (AKI). For each patient, blood and urine were collected at multiple time points pre- and post-operatively, and about two dozen biomarkers were measured at each time point. AKI is typically diagnosed via changes in serum creatinine but these changes often do not happen until several days after the injury.⁵ One goal of the study is to identify a combination of post-operative biomarkers that can provide an earlier diagnosis of AKI.

2. Notation and Terminology

We discuss existing methods for (i) modeling clustered data and (ii) adjusting for covariates in evaluating performance. We then apply these ideas to the multicenter setting, where center can be thought of as both a clustering variable and a covariate. Below, we primarily use the term “cluster,” though this is (for our purposes) interchangeable with “covariate.”

Let C indicate cluster and suppose the population consists of M clusters where cluster c has N_c observations, c = 1, …,M. Further suppose that we observe data from m of these clusters with n_c observations from cluster c, giving n total observations. We consider a p-dimensional vector of predictors X and a binary outcome D. Cases (individuals who have or will develop the outcome) are denoted by either D = 1 or the subscript D, while controls (individuals who do not have or will not develop the outcome) are denoted by either D = 0 or the subscript $\overline{D}$. Let (X, D) be the predictors and outcome for an arbitrary observation. We use the subscript i on X and D to denote the predictors and outcome, respectively, for the i^th observation. We use the superscript c on X and D to denote the predictors and outcome, respectively, for an observation from cluster c. We denote the collection of predictors and outcomes for observations in cluster c as $\left({𝓧}^c,{𝓓}^c\right)$.

In general, in the clustered data setting, predictors may be constant for all observations in a cluster (often called cluster-level, cluster-constant, or between-cluster predictors), may vary across observations in a cluster (called cluster-varying or within-cluster predictors), or may vary both within and between clusters. We focus on predictors that have at least some variation within clusters. Throughout, we will assume a non-trivial cluster-specific prevalence of D; that is, $P(D=1|C=c):={\gamma }_c\in (0,1),\hspace{1em}c=1,\dots ,M$.

3. Models for Clustered Data

3.1. Random Intercept Logistic Regression

The random intercept logistic regression (RILR) model can be written as:

$$\mathrm{logit}\left\{P\left(D=1|\mathbf{X},C=c,b_c\right)\right\}=b_c+{\tau }_0+{\text{τ}}^{\top }\mathbf{X},\hspace{1em}{b}_c\stackrel{iid}{~}F\text{.}$$ (1)

Typically, it is assumed that $b_c~N\left(0,{\sigma }^2\right)$ so σ² is an additional parameter in this model. If we view the random intercept b_c as σz_c, where $z_c\stackrel{iid}{~}N(0,1)$, σ is the regression coefficient for this standardized omitted (cluster-level) predictor.⁶ In that sense, b_c is generally “interpreted as the combined effects of omitted cluster-level predictors.”⁷

3.1.1. Assumptions

The key assumptions made by the RILR model given in equation (1) are:^8,9

• (A1)b_c and ${𝓧}^c$ are independent
• (A2) $b_c\stackrel{iid}{~}N\left(0,{\sigma }^2\right)$
• (A3)Conditional on $\left({𝓧}^c,b_c\right)$, $D_1^c,\dots D_{n_c}^c$ are independent and $P\left(D_i^c=1|{𝓧}^c,b_c\right)=P\left(D_i^c=1|{\mathbf{X}}_i^c,b_c\right),i=1,\dots ,n_c$

Assumption (A1) can be written as $f\left(b_c|{𝓧}^c\right)=f\left(b_c\right)$, which is a fairly strong assumption in the non-randomized setting.⁶ In particular, this assumption is often implausible when the distribution of the predictors varies by cluster.

3.1.2. Estimates

It is important to distinguish between marginal and conditional modeling approaches: the conditional (or cluster-specific) approach, for example, RILR, involves modeling the probability distribution of D as a function of predictors and cluster-specific parameters (e.g., cluster-specific intercepts), while the marginal (or population-averaged) approach involves modeling the marginal expectation of D as a function of predictors.¹⁰ Due to the inclusion of cluster-specific parameters, parameter interpretation under the conditional approach is with respect to cluster.¹⁰ For predictors that vary within clusters, conditional methods are often more appropriate than marginal methods, such as generalized estimating equations.¹⁰ In particular, conditional approaches are most useful when there is interest in estimating the association between the predictor and the outcome without the influence of cluster differences.

Predictors frequently have both a between- and within-cluster component; that is, they vary both within and between clusters.¹¹ Estimates obtained via conditional methods are generally interpreted as estimates of the within-cluster association, i.e., the association within each cluster, averaged across clusters; this is typically what researchers are trying to estimate when they use these methods with predictors that vary within clusters.^1,12,13 However, as discussed below, estimated coefficients obtained from RILR may not actually represent the within-cluster association: depending upon the nature of the data, the resulting estimates are often a combination of within- and between-cluster variations.^{1,9–11,13–16} Importantly, between-cluster differences are likely to include the effects of cluster-constant confounders. ¹³

3.1.3. Violations of Assumptions

First we consider (A1); that is, independence of b_c and ${𝓧}^c$. In the context of a randomized multicenter clinical trial, the assumption holds since the distribution of the predictor, treatment, is the same across centers in large samples.^1,11 However, as noted above, it is often the case predictors are not purely within-cluster and instead have both a between-cluster component and a within-cluster component.^11,17,18 When such predictors are included in a RILR model, the assumption that b_c and ${𝓧}^c$ are independent may not hold, leading to distortions of the association of interest.^11,17

As a concrete example, suppose the following model holds for the predictor X:

$$\begin{gathered} \mathrm{logit}\left\{P\left(D=1|X,C=c,b_c^{\prime }\right)\right\}=b_c^{\prime }+{\tau }_0+{\tau }_{B}h\left({𝓧}^c\right)+{\tau }_W\left(X-h\left({𝓧}^c\right)\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=b_c^{\prime }+{\tau }_0+\left({\tau }_B-{\tau }_W\right)h\left({𝓧}^c\right)+{\tau }_{W}X, \end{gathered}$$ (2)

where $b_c^{\prime }~N\left(0,{\sigma }^2\right)$ and h(·) is some cluster-level summary of ${𝓧}^c$ such that $X-h\left({𝓧}^c\right)$ has the same distribution across clusters. Here, $X-h\left({𝓧}^c\right)$corresponds to the within-cluster component of X and $h\left({𝓧}^c\right)$ corresponds to the between-cluster component of X. If the distribution of the predictor is the same across clusters, then $\left({\tau }_B-{\tau }_W\right)h\left({𝓧}^c\right)$ will be constant in large samples, and can be combined with the fixed intercept τ₀. However, if the distribution of the predictor varies across clusters such that $h\left({𝓧}^c\right)$ varies and the RILR model given in equation (1) is fit to the data, $b_c=b_c^{\prime }+\left({\tau }_B-{\tau }_W\right)h\left({𝓧}^c\right)$ which is not independent of ${𝓧}^c$ if ${\tau }_B\ne {\tau }_W$, violating assumption (A1).

Results from research on omitted variable bias indicate that when (2) holds, and (1) is fit to the data, the estimate of τ_W will be a combination of the within- and between-effects, τ_B and τ_W.¹¹ Importantly, the combination of within- and between-effects, if these effects differ, is not of substantive interest, as it lacks clinical relevance.^1,11 Even in situations where it is thought that the between- and within-cluster effects are reasonably close to one another, there is the potential for differential confounding at the between- versus within-cluster level; thus, using both within- and between-cluster comparisons to estimate the within-cluster effect is problematic.^13,19 If cluster-level factors are associated with the predictors, as is often true in observational studies, the distribution of the predictors will vary across clusters, which may lead to correlation between the random intercepts and the predictors.^7,16

This issue is often called “confounding by cluster” since the within-cluster association, τ_W, is distorted by the between-cluster association, τ_B;^1,9,17,20,21 in the econometrics literature, it is called the “endogenous covariates problem.”⁷ In our example, omitting $h\left({𝓧}^c\right)$ leads to correlation between b_c and ${𝓧}^c$, which, as described by Greenland et al., has the effect of confounding τ_W.²² Thus, confounders are “now covariates that ‘explain’ the correlation between” b_c and ${𝓧}^c$,²² namely, the cluster-level variable $h\left({𝓧}^c\right)$.

Assumption (A2) requires that the random cluster-specific intercepts be independently and identically distributed according to a normal distribution with mean zero and variance σ². Broadly speaking, misspecifications of the random intercept distribution may lead to bias in the estimate of the fixed intercept and the coefficients for cluster-level variables but typically do not have a large effect on the estimates for cluster-varying predictors.^6,18,23,24

One solution that has been proposed to address violations of assumption (A1) is to decompose predictors into a between-cluster component and a within-cluster component.^{6,9,11,14,16,18,19,25,26} In the context of model (2), this means fitting a model with $h\left({𝓧}^c\right)$ and X as predictors. When $h\left({𝓧}^c\right)={\overline{𝓧}}^c$, the cluster mean, this approach is called the “poor man’s” method.¹¹ Using the cluster mean may be overly simplistic¹⁷ and more flexible methods have been proposed based on modeling b_c as a function of ${𝓧}^c$.²⁷ Of course, these methods require that the model for b_c be correctly specified.^16,26,27

3.1.4. Efficiency

RILR is often touted as being more efficient than alternative methods due in part to the assumption that b_c has some (parametric) distribution.^12,28 In addition, RILR can use both between- and within-cluster comparisons to estimate coefficients, which allows it to use more information in estimating these parameters.^8,9,12,29 Some studies have found reduced efficiency when the distribution of the random intercept is not normal and normality is assumed.³⁰

3.2. Fixed Intercept Logistic Regression

Fixed intercept logistic regression (FILR) can be used to model clustered data by including a fixed intercept for each cluster. These models are a special case of generalized linear models. We consider two variants of FILR: conditional (cFILR) and unconditional (uFILR). Both cFILR and uFILR have the same model form:

$$\mathrm{logit}\left\{P\left(D=1|\mathbf{X},C=c,{\beta }_0^c\right)\right\}={\beta }_0^c+{\mathit{\beta }}^{\top }\mathbf{X}\text{,}$$ (3)

where ${\beta }_0^c$ represents a cluster-specific intercept. The conditioning on ${\beta }_0^c$ in (3) is only necessary if ${\beta }_0^c$ is random. cFILR and uFILR differ in their approach to estimation: uFILR relies on the full likelihood, while cFILR uses a conditional likelihood, conditioning on the number of cases in each cluster.³¹ This allows cFILR to avoid the incidental parameters problem when the number of clusters is large.³¹

3.2.1. Assumptions

In the econometrics literature, the distinction between RILR and FILR is based not on whether the cluster-specific intercepts are fixed or random, but whether they are independent of the predictors.⁸ Thus, the key assumption for FILR is:^8,9

• (B1)Conditional on ${𝓧}^c,D_1^c,\dots D_{n_c}^c$ are independent

If the ${\beta }_0^c$ are random, then they must be independent across clusters and assumption (B1) must additionally condition on ${\beta }_0^c$.^8,9

3.2.2. Estimates

FILR consistently estimates the within-cluster effect of predictors that vary within clusters, provided (B1) is satisfied and model (3) holds.^11,12,15,17 Thus, this method avoids the issue of confounding by cluster; in fact, the resulting estimates are not subject to confounding by any unmeasured cluster-constant variable.^13,17

For both uFILR and cFILR, only within-cluster comparisons are used to estimate the coefficients, and, since clusters for which all observations have D = 1 or all observations have D = 0 (we call these “concordant clusters”) do not contribute any information to the estimation of the within-cluster effect, they are not used in estimation.^8,12 This is also true of clusters that are concordant on the predictors, though this situation is unlikely when there are multiple and/or continuous predictors.

3.2.3. Efficiency

Many investigators are hesitant to use FILR since the exclusion of concordant clusters could reduce efficiency.¹¹ However, previous research has shown that cFILR provides estimates that are efficient relative to RILR for predictors that vary predominantly within-clusters.^19,32 Indeed, as pointed out by Neuhaus and Kalbfleisch, for predictors with between- and within-cluster components, the increased efficiency of estimates from RILR that is sometimes observed is often largely due to the assumption of common within- and between-cluster effects.¹¹ If these effects are indeed equal, there will be some efficiency gain from using RILR since this approach uses both within- and between-cluster variations to estimate the coefficients.¹⁹ However, as noted above, using both types of variation in estimation is generally not recommended since the between- and within-cluster effects may not be equal and the potential exists for differential confounding. Furthermore, concordant clusters contribute to between-cluster variation and often exhibit strong between-cluster effects, which have the potential to heavily distort the estimated coefficients for predictors that vary within clusters if RILR is used.^14,15 Finally, uFILR involves estimating m cluster-specific intercepts, which may lead to reduced efficiency relative to RILR, particularly when m is large relative to the total sample size.

4. Evaluating the Performance of a Predictor

Suppose we have a predictor Z and are interested in evaluating its performance. Without loss of generality, we will assume that higher values of Z are more indicative of D. We focus on discrimination, since determining the discriminative ability of a predictor is often the first step in developing a clinically useful diagnostic or prognostic tool. Discrimination is the ability of Z to separate cases and controls, and is commonly assessed via the area under the receiver operating characteristic (ROC) curve (AUC). The ROC curve plots the true positive rate, the proportion of correctly classified cases, versus the false positive rate, the proportion of incorrectly classified controls, over the range of possible thresholds for Z.³³ The ROC curve for a useless predictor is the 45-degree line, and the corresponding AUC is 0.5.³³ The ROC curve for a perfect predictor reaches the upper left-hand corner of the unit square, and the AUC for such a predictor is 1.³³ The AUC has a probabilistic interpretation: it is the probability that, for a randomly selected case and control, the value of Z for the case is higher than the value of Z for the control, that is, $AU{C}_Z=P\left(Z_D>Z_{\overline{D}}\right)$.³³ In practice, we can estimate the AUC empirically:

$$A\widehat{U}{C}_Z=\frac{{\sum }_{i=1}^{n_D}{\sum }_{j=1}^{n_{\overline{D}}}1\left(Z_{Di}>Z_{\overline{D}j}\right)}{n_D{n}_{\overline{D}}}$$

where 1(a > b) is 1 if a > b and 0 otherwise.³³

Covariate effects could influence the evaluation of the predictor Z; in particular, associations between Z and the covariate could allow the covariate to contribute to or attenuate the discriminatory accuracy of Z.³⁴ In order to prevent the covariate from affecting the assessment of the discriminatory accuracy of Z, the covariate-adjusted AUC should be evaluated. The covariate-adjusted ROC (aROC) and corresponding covariate-adjusted AUC (aAUC) for a discrete covariate C, proposed by Janes and Pepe, can be written as aROC_Z and aAUC_Z, respectively, where³⁵

$$\begin{gathered} aAU{C}_Z={\int }_0^{1}aRO{C}_Z(t)dt={\int }_0^1\sum _{c}RO{C}_{Z|C=c}(t)P(C=c|D=1)dt \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}=\sum _c{w}_{c}AU{C}_{Z|C=c}, \end{gathered}$$ (4)

where t denotes the false positive rate, ROC_Z|C=c and AUC_Z|C=c denote the covariate-specific ROC and AUC, respectively, and w_c = P (C = c|D = 1) denotes the distribution of the covariate among cases. Thus, the aAUC is a weighted average of the covariate-specific AUCs, where the weights correspond to the proportion of cases with each covariate value.^35,36 In practice, AUC_Z|C=c is estimated empirically:

$$A\widehat{U}{C}_{Z|C=c}=\frac{{\sum }_{i=1}^{n_D^c}{\sum }_{j=1}^{n_D^c}1\left(Z_{Di}^c>Z_{\overline{D}j}^c\right)}{n_D^c{n}_{\overline{D}}^c}.$$

The empirical aAUC estimate is then

$$aA\widehat{U}{C}_Z=\sum _{c=1}^m{\widehat{w}}_{c}A\widehat{U}{C}_{Z|C=c},$$ (5)

where ${\widehat{w}}_c$ is the fraction of observed cases with covariate value c, that is, ${\widehat{w}}_c=\frac{n_D^c}{n_D}$.

Using the distribution of the covariate among cases to summarize the covariate-specific AUCs is compelling because the resulting measure is the area under the ROC curve corresponding to the true positive rate based on covariate-specific thresholds; these covariate-specific thresholds are chosen such that the false positive rate is the same for each value of the covariate.³⁵ That is, we can write $aRO{C}_Z(t)=P\left(Z>g_c(t)|D=1\right)$, where g_c(t) is the covariate-specific threshold giving a false positive rate of t for the covariate value c.³⁵

When the same data are used to construct a combination and evaluate its performance, the resulting estimate of performance is optimistic.³⁷ This can be addressed by using a bootstrapping procedure to estimate the degree of optimism.^37,38 Bootstrapping assumes observations are exchangeable, which may not be reasonable when the data are clustered; thus, bootstrap resampling of clusters has been suggested.^1,36,39,40 However, Bouwmeester et al. found similar results for the average cluster-specific AUC whether resampling was done on clusters or individual observations.³⁹

5. Biomarker Combinations in Multicenter Studies

Now, our predictors consist of a collection of biomarkers, and both the covariate and the cluster variable are center.

When the data come from a single center, common practice is to first construct a combination of the biomarkers, often using logistic regression, and evaluate its performance using measures such as the AUC. With more than one center, it is important to consider how to appropriately accommodate center in both the construction and evaluation of biomarker combinations.

Throughout, we focus on constructing a single biomarker combination; that is, we do not allow the relationship between the biomarkers and the outcome to vary across centers. In the clinical trial setting, assessing treatment-by-center interactions is usually not part of the primary analysis.²⁹ Analogously, in the diagnostic and prognostic settings, it is preferable to give a single combination that is not center-specific, as this would make combination development highly localized. We focus on constructing linear combinations via the logistic regression framework. While this may seem restrictive, Pepe et al. noted that the class of linear combinations is actually quite large (taking into consideration possible biomarker transformations and interactions) and the logistic form is fairly robust.⁴¹

5.1. The Role of Center

We consider the role of center in the context of two sets of characteristics:

1. Characteristics affecting the prevalence of D: differences in the populations served by each center could affect the prevalence of D.

2. Characteristics affecting biomarker measurements: center-level factors, including storage and handling of specimens and practices in each center, could lead to variations in biomarker measurements unrelated to D.

We focus on three possibilities for the role of center. We call center a confounder when it affects both the prevalence of D and the biomarker measurements, a case mix variable when it affects only the prevalence of D, and a calibration variable when it affects only the biomarker measurements.

In the TRIBE-AKI study, where the goal is to use biomarkers to diagnose AKI, certain centers may serve particularly unhealthy communities, resulting in differences in biomarker levels across centers; however, these differences may reflect true underlying biology. If factors such as storage and handling of biomarkers and surgical practices are standardized, such that the distribution of biomarkers is similar across centers, conditional on case status, center would be a case mix variable. If, however, these factors vary across centers (e.g., in some centers surgeons use different protocols for fluid administration) in addition to variability in disease prevalence, center would be a confounder. On the other hand, if the populations served by each center are relatively similar in terms of underlying AKI risk, but factors such as surgical protocols vary across centers and lead to variations in biomarker measurements, center would be a calibration variable.

We can also consider an example with prognostic biomarkers. Suppose carotid intima-media thickness is used to predict which patients will experience stroke. If certain centers tend to serve less healthy populations (i.e., those at greater risk for stroke), but practices for measuring carotid intima-media thickness are standardized across centers, center would be a case mix variable. If the make-up of patients in terms of underlying risk of stroke is similar across centers, but different protocols or imaging tools are used at different centers such that the distribution of intima-media thickness measurements varies across centers, center would be a calibration variable. If both the composition of patients and intima-media thickness measurement practices vary across centers, center would be a confounder.

In Figure 1, we present graphical and probabilistic depictions of center as a case mix variable, a calibration variable, and a confounder for diagnostic or prognostic biomarkers X. Diagnostic biomarkers represent some underlying disease or disease process, that is, D → X, while prognostic biomarkers cause some future outcome, that is, X → D.

Figure 1:

Select potential roles of center in biomarker studies.

It is important to distinguish center as a confounder, as defined in Figure 1, from “confounding by cluster” in the context of a RILR model. Our use of “confounding” in Figure 1 is in line with standard epidemiological notions of confounding, where a variable C distorts the effect of interest, the causal association between X and D. The idea of “confounding by cluster” for RILR models, on the other hand, is specific to the RILR framework: “confounding by cluster” occurs when the random intercepts and the biomarkers are not independent, leading to distortion of the effect of interest, the within-cluster association. As we will see, there are situations where center is not a confounder by the definitions in Figure 1, but the random intercepts and the biomarkers are not independent, so the estimates from the RILR model are susceptible to “confounding by cluster.”

5.2. Ignoring Center

Clinical researchers frequently ignore center in the construction and/or evaluation of combinations of diagnostic or prognostic biomarkers (e.g., Shapiro et al.⁴² and Vuilleumier at al. ⁴³). This is likely due to the fact that investigators acknowledge that center should not naïvely be included as a predictor, but are not familiar with methods for accommodating center or the repercussions of ignoring it.

First, we consider ignoring center in combination construction. Suppose the linear-logistic model holds:

$$\mathrm{logit}\left\{P\left(D=1|\mathbf{X},C=c,{\beta }_0^c\right)\right\}={\beta }_0^c+{\mathit{\beta }}^{\top }\mathbf{X}.$$ (6)

Such a model could arise from the following data-generating model for two biomarkers, X = (X₁, X₂):

$$\left(\left.\begin{array}{c}{X}_1\\ X_2\end{array}\right|D=d,C=c\right)\sim N\left(\left(\begin{array}{c}{f}_{X_1}(c)+{\mu }_{X_1}d\\ f_{X_2}(c)+{\mu }_{X_2}d\end{array}\right),\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)\right)$$ (7)

where ${\mu }_{X_1}$ and ${\mu }_{X_2}$ are related to the center-specific AUC for each marker: ${\mu }_{X_1}=\sqrt{2}{\mathrm{\Phi }}^{-1}\left({\lambda }_1\right)$ and ${\mu }_{X_2}=\sqrt{2}{\mathrm{\Phi }}^{-1}\left({\lambda }_2\right)$, where Φ is the standard normal distribution function and λ₁ and λ₂ are the center-specific AUCs for X₁ and X₂, respectively. Thus, we consider constant center-specific AUCs for X₁ and X₂, and allow for center effects on biomarker levels via conditional mean shifts (f(c)). Equation (7) gives

$$\mathrm{logit}\left\{P\left(D=1|\mathbf{X},C=c,{\beta }_0^c\right)\right\}={\beta }_0^c+{\beta }_1{X}_1+{\beta }_2{X}_2\text{.}$$

where ${\beta }_0^c$ is a center-specific offset and, as shown in the Supplementary Material (S1),

$$\begin{gathered} {\beta }_0^c=\frac{-{\mu }_{X_1}^2-{\mu }_{X_2}^2}{2\left(1-{\rho }^2\right)}+\frac{\rho {\mu }_{X_1}{\mu }_{X_2}+\rho {\mu }_{X_1}{f}_{X_2}(c)+\rho {\mu }_{X_2}{f}_{X_1}(c)}{1-{\rho }^2} \\ \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}-\frac{\left\{{\mu }_{X_1}{f}_{X_1}(c)+{\mu }_{X_2}{f}_{X_2}(c)\right\}}{1-{\rho }^2}+\log \left(\frac{{\gamma }_c}{1-{\gamma }_c}\right)\text{,} \end{gathered}$$

and

$${\beta }_1=\frac{{\mu }_{X_1}-\rho {\mu }_{X_2}}{1-{\rho }^2},\hspace{1em}{\beta }_2=\frac{{\mu }_{X_2}-\rho {\mu }_{X_1}}{1-{\rho }^2}.$$

Recall that γ_c is the center-specific prevalence, γ_c = P (D = 1|C = c).

Returning to the general linear-logistic model given in (6), suppose that the model holds, but ${\beta }_0^c$ is not allowed to vary across centers. That is, suppose we fit the following model to the data pooled across centers:

$$\mathrm{logit}\{P(D=1|\mathbf{X})\}={\alpha }_0+{\mathit{\alpha }}^{\top }\mathbf{X}.$$ (8)

When C and D are independent conditional on X or C and X are independent conditional on D, and model (6) holds, we have collapsibility,⁴⁴ so the conditional and marginal coefficients are the same (α = β) and the marginal logit, logit{P (D = 1|X)}, is still linear. Therefore, in these situations, the relationship between the biomarkers and the outcome is the same whether or not we condition on center. Furthermore, under model (6), when C and D are independent conditional on X, ${\beta }_0^c$ will not vary across centers, so ${\alpha }_0={\beta }_0^c$.

However, when model (6) holds but C and D are not independent conditional on X and C and X are not independent conditional on D, we may no longer have α = β. Furthermore, the linear-logistic model (6) may not hold, in which case the results on collapsibility will no longer be expected to apply. More generally, ignoring center in the construction of the biomarker combination potentially allows center to be predictive; that is, part of the effect of center may be included in the estimates of the biomarker coefficients when center is omitted.

Center may also be ignored in the evaluation of combinations. Suppose we have a linear combination, $L_{\mathit{\theta }}(\mathbf{X})={\mathit{\theta }}^{\top }\mathbf{X}$. When center is ignored in the evaluation of L_θ(X), the data are pooled across centers, giving the marginal AUC, $AUC(\mathit{\theta })=P\left(L_{\mathit{\theta }}\left({\mathbf{X}}_D\right)>L_{\mathit{\theta }}\left({\mathbf{X}}_{\overline{D}}\right)\right)$ is associated with center, the marginal AUC may not reflect the center-specific AUC.³⁴

5.3. Accounting for Center

Analogous to center-adjusted measures of association in multicenter etiologic studies and center-adjusted treatment effects in multicenter randomized trials, methods that stratify (condition) on center can be used in both the construction and evaluation of biomarker combinations. By stratifying on center, we avoid allowing center differences to affect the construction of the biomarker combination or the assessment of its performance.

5.3.1. Construction

We consider two methods for constructing combinations that involve conditioning on center, namely, RILR and FILR; for FILR, we will consider both cFILR and uFILR. For concreteness, we consider p = 2 in the discussion below.

To the extent that the literature has acknowledged the potential role of center in the prediction setting, RILR is often the approach used in constructing combinations:⁴⁵

$$\mathrm{logit}\left\{P\left(D=1|\mathbf{X},C=c,b_c\right)\right\} =b_c+{\tau }_0+{\tau }_1{X}_1+{\tau }_2{X}_2,b_c\stackrel{iid}{~}F\left(0,{\sigma }^2\right),$$ (9)

where F is the distribution of the random center-specific intercepts. In general, when the distribution of X₁ or X₂ varies by center, assumption (A1) may not hold and the estimates $\left({\widehat{\tau }}_0,{\widehat{\tau }}_1,{\widehat{\tau }}_2\right)$ may not be meaningful.

An option that has been discussed at length in the literature on multicenter randomized trials,^1,11,17 but has been largely (if not entirely) neglected in the prediction literature is FILR, using uFILR when the number of centers is modest and cFILR when the number of centers is large:³¹

$$\text{logit}\left\{P\left(D=1\mid \mathbf{X},C=c,{\beta }_0^c\right)\right\}={\beta }_0^c+{\beta }_1{X}_1+{\beta }_2{X}_2.$$

Random intercept models are, at first glance, appealing in the context of prediction when the data arise from multiple centers: these models are thought to represent a situation where there exists a large population of centers, and the data at hand constitute a random draw of centers from that population. This intuition may make investigators more comfortable with generalizing their results to centers not included in their data, typically the goal of prediction research, and thus more likely to use RILR. However, the key distinction between random and fixed intercept models is not necessarily whether the center-specific intercepts are random or fixed, but rather whether they are associated with the biomarkers.⁸ Thus, while the notion of center-specific intercepts as random quantities may have intuitive appeal, this is outweighed by the statistical reality that random intercept models rely on potentially untenable assumptions.

Researchers may also be drawn to RILR since it gives an estimate of the overall intercept τ₀ and, since the center-specific intercepts b_c are typically assumed to be normally distributed with mean 0, researchers may use ${\widehat{\tau }}_0+{\widehat{\tau }}_1{X}_1+{\widehat{\tau }}_2{X}_2$ to provide predicted probabilities for patients in new centers not used in model fitting. However, assuming b_c = 0 in new centers generally leads to poor calibration; that is, it does not provide useful estimates of P (D = 1|X).²¹ Even if a valid estimate of b_c is available, the estimate of τ₀ from RILR can be badly biased if the random intercept distribution is misspecified.

The “poor man’s” method has been proposed as an alternative to standard RILR. Even if the distributions of the mean-centered predictors are the same across centers (which would help to address violations of assumption (A1)), this method is not particularly compelling in the prediction setting since application of the model to new centers requires estimates of the center-specific biomarker means; such reliance on information from the new center makes external validation and clinical application (if predicted probabilities are sought) more challenging. In addition, since the “poor man’s” method still relies on a RILR model, the estimate of the fixed intercept may face the same challenges as with the standard RILR model.

The goal of the poor man’s method is to transform the biomarkers into predictors that are independent of b_c. This is an attempt to force the model to estimate the within-center effect of the biomarkers, as opposed to a combination of the within- and between-center effects. However, FILR estimates the within-center effect with no further assumptions or transformations of the data. This is compelling as estimates of biomarker associations (and thus, fitted biomarker combinations) that are unaffected by center differences are most useful in identifying promising combinations for further development.

Conversely, an obvious criticism of FILR is that it does not allow predicted probabilities to be calculated either in new centers (for uFILR) or at all (for cFILR). However, as discussed above, RILR does not necessarily solve this problem. Furthermore, the biomarker combination can still be useful, for example, to stratify patients within each center according to likelihood of having or developing the outcome.

5.3.2. Evaluation

The marginal AUC would be an appropriate measure of performance if between-center heterogeneity could be used in making decisions, but this is not typically true.⁴⁶ Thus, some summary of the conditional, or center-specific, AUCs should be used to avoid allowing center differences to influence the evaluation of performance; we propose using the center-adjusted AUC defined in equation (4). For a given combination $L_{\mathit{\theta }}(\mathbf{X})={\mathit{\theta }}^{\top }\mathbf{X}$, the center-adjusted AUC can be written as $aAUC(\mathit{\theta })={\sum }_{c=1}^M{w}_{c}AU{C}_c(\mathit{\theta })$ where the center-specific AUC is $AU{C}_c(\mathit{\theta })=P\left(L_{\mathit{\theta }}\left({\mathbf{X}}_D^c\right)>L_{\mathit{\theta }}\left({\mathbf{X}}_{\overline{D}}^c\right)\right)$. Note that only discordant centers contribute to the empirical estimate of the center-adjusted AUC defined in equation 5.

When the ROC curve varies by a covariate, it is generally recommended that a separate ROC curve be estimated for each value of the covariate.³⁶ In the case of center, where only a fraction of the centers are observed, this is not possible. However, it is reasonable to assess the heterogeneity in the center-specific AUCs, as this provides some indication of how the predictor may perform in a new center.⁴⁵

Finally, as a consequence of focusing on the center-specific AUC, summarized via the aAUC, we do not need an estimate of the center-specific intercept to evaluate a combination, as the center-specific AUC is a rank-based measure and so would be unaffected by such offsets. This allows for identification of promising combinations of biomarkers for further development without the need for center-specific intercept estimates.

5.3.3. Asymptotic Properties

Our proposal involves constructing linear combinations of biomarkers by estimating θ and evaluating the performance of these combinations with the aAUC. We would like to demonstrate consistency of this estimate of performance; that is, we would like to say that $aA\widehat{U}C(\widehat{\mathit{\theta }})$ converges in probability to aAUC(θ₀) if $\widehat{\theta }$ converges in probability to θ₀. This is shown by Lemma 1 and Theorems 1 and 2, which are stated and proved in the Supplementary Material (S2).

5.4. Combining Construction and Evaluation

When constructing and evaluating biomarker combinations, there are two binary decisions to make regarding center, giving four possibilities (using the notation of models (6) and (8)):

1. Pool the data across centers for both construction and evaluation, giving AUC(α)

2. Pool the data across centers for construction, but stratify by center for evaluation, giving aAUC(α)

3. Stratify by center for construction, but pool across centers for evaluation, giving AUC(β)

4. Stratify by center for both construction and evaluation, giving aAUC(β)

Proposition 1, given in the Supplementary Material (S3), follows directly from Pepe³³ and shows that the marginal and center-adjusted AUCs of a combination based on some θ are equivalent if C and L_θ(X) are independent among controls. If C and X are independent conditional on D, then C and L_θ(X) will be independent among controls. Thus, if model (6) holds and C and X are independent conditional on D, then AUC(β) = aAUC(β) = aAUC(α) = AUC(α), since α = β by collapsibility. Proposition 2, given in the Supplementary Material (S3), also follows directly from Pepe³³ and shows that when the prevalence and center-specific AUC do not vary with center and the center-specific ROC curves are concave, the aAUC for a given biomarker combination will be at least as large as the marginal AUC. In general, the center-specific ROC curves will be concave if for a given θ, in each center, increasing L_θ(X) increases the likelihood that D = 1.³⁷

When model (6) holds, optimality of the risk score $P\left(D=1|\mathbf{X},C=c,{\beta }_0^c\right)$ implies that the combination based on β is optimal within each center, in terms of maximizing the center-specific AUC.^33,47 Thus, under this model,

$$aAUC(\mathit{\beta })\ge aAUC(\mathit{\theta }),$$

for any θ. Furthermore, by the collapsibility results discussed above, when model (6) holds and C and D are independent conditional on X, α = β, so

$$AUC(\mathit{\alpha })=AUC(\mathit{\beta })$$

$$aAUC(\mathit{\alpha })=aAUC(\mathit{\beta }).$$

5.5. Identifying the Role of Center

When biomarker data from multiple centers are available, graphical displays and other data summaries may be useful for identifying whether center is a case mix variable, calibration variable, or confounder. We therefore propose investigating:

1. The distribution of the biomarkers across center (to assess whether C ⊥ X)

2. The distribution of the biomarkers across center stratified by the outcome (to assess whether C ⊥ X|D)

3. The prevalence of the outcome across centers (to assess whether C ⊥ D)

4. The prevalence of the outcome across centers, stratified by biomarker categories (to assess whether C ⊥ D|X)

Not all of these tools will be useful in all settings; for example, in the case of diagnostic biomarkers, items (2) and (3) will be most useful, while items (1) and (4) will be most useful for prognostic markers. Also, implementing item (4) may be challenging as it requires the designation of biomarker categories, which necessarily results in a loss of information. Alternatively, this relationship could be assessed with a regression model. Some of these suggestions are similar in spirit to those made by Berlin et al. in the setting of a single binary predictor.¹⁷

It is important to keep in mind that there are no “rules” for interpreting the results of the data summaries proposed above. Rather, they should be used as a guide, in conjunction with knowledge about the design and conduct of the study, to assess the role of center.

6. Simulations

6.1. Ignoring Center

We studied the impact of ignoring center in the construction and/or evaluation of biomarker combinations. We considered diagnostic markers, and allowed center to be a case mix variable, a calibration variable, or a confounder (as summarized in Figure 1). The two biomarkers X₁ and X₂ were distributed as described in equation (7) with ρ = 0.5, and λ₁ = 0.6 and λ₂ = 0.65 in all centers.

Throughout, $f_{X_1}\left(c\right)=f_{X_2}\left(c\right)=f\left(c\right)$. When center was a case mix variable, $\mathrm{logit}\left({\gamma }_c\right)~N\left(0,{\sigma }_{{\gamma }_c}^2\right)$ and f (c) = 0. When center was a calibration variable, γ_c = 0.5 and $f(c)~N\left(0,{\sigma }_{f(c)}^2\right)$. Finally, when center was a confounder,

$$\left(\begin{array}{c}\mathrm{logit}\left({\gamma }_c\right)\\ f(c)\end{array}\right)~N\left(\left(\begin{array}{l}0\hfill \\ 0\hfill \end{array}\right),\left(\begin{array}{cc}{\sigma }_{{\gamma }_c}^2& \delta {\sigma }_{{\gamma }_c}{\sigma }_{f(c)}\\ \delta {\sigma }_{{\gamma }_c}{\sigma }_{f(c)}& {\sigma }_{f(c)}^2\end{array}\right)\right).$$

We considered ${\sigma }_{{\gamma }_c}^2=1$, ${\sigma }_{f(c)}^2=5$, and δ ∈ {−0.75, 0.75}.

We constructed combinations in a training dataset consisting of either 6 centers with 200 observations each or 500 centers with 20 observations each. The combinations were constructed via logistic regression, where center was either ignored or incorporated using FILR (uFILR for m = 6 and cFILR for m = 500). These estimates correspond to estimates of α and β (defined in equations (8) and (6)), respectively.

We evaluated fitted combinations via the conditional AUC, AUC_c(·), in a large test dataset with a single center, and the marginal AUC, AUC(·), in a large test dataset with multiple centers. As shown in the Supplementary Material (S4), the conditional AUC is constant across centers under our data-generating model, so AUC_c(·) = aAUC(·). The test set used to evaluate the conditional AUC consisted of a single center with 200 000 observations while the test set used to evaluate the marginal AUC included either 6 centers with 30 000 observations each or 500 centers with 400 observations each, depending on the structure of the training data. The observations in the test data represent subjects from new centers, i.e., not the same centers as used in the training data. The true coefficients β = (β₁, β₂) and AUC_c(β) were determined analytically for comparison. The simulations were repeated 500 times.

Figure 2 presents the results of the simulations with 500 centers. These simulations support the conclusions given above: that is, when center was a case mix variable, the combination and its performance (in terms of the AUC) were not affected by ignoring center in construction and/or evaluation. When center was a calibration variable, ignoring center during construction led to biomarker combinations with reduced predictive capacity in new centers, and ignoring center during evaluation yielded a measure of performance that was lower than the actual performance of the combination in a new center. These results are consistent with the relationships described above, that is, $AU{C}_c(\widehat{\mathit{\beta }})\ge AU{C}_c(\widehat{\mathit{\alpha }})$, $AU{C}_c(\widehat{\mathit{\beta }})\ge AUC(\widehat{\mathit{\beta }})$ and $AU{C}_c(\widehat{\mathit{\alpha }})\ge AUC(\widehat{\mathit{\alpha }})$.

Figure 2:

Simulation results for training data with 500 centers. The first column, $AUC(\widehat{\mathit{\alpha }})$, is the marginal AUC based on the combination constructed by ignoring center and the second column, $AU{C}_c(\widehat{\mathit{\alpha }})$, is the conditional AUC based on the combination constructed by ignoring center. The third column, $AUC(\widehat{\mathit{\beta }})$, is the marginal AUC based on the combination constructed by stratifying by center and the fourth column, $AU{C}_c(\widehat{\mathit{\beta }})$, is the conditional AUC based on the combination constructed by stratifying by center. For each, the median and middle 90% of the distribution across simulations are shown. Different colors and shapes correspond to different roles for center: blue circles indicate center is a case mix variable, red squares indicate center is a calibration variable, purple triangles indicate center is a confounder with positive correlation (0.75) between logit(γ_c) and f(c), and green diamonds indicate center is a confounder with negative correlation (−0.75) between logit(γ_c) and f(c). The gray horizontal line represents AUC_c(β) as determined analytically.

When center was a confounder and center was ignored during construction (yielding $\widehat{\alpha }$), further ignoring center during evaluation tended to give a measure of performance that was higher than the actual performance in a new center (i.e., $AUC(\widehat{\mathit{\alpha }})$ tended to be larger than $AU{C}_c(\widehat{\mathit{\alpha }})$). On the other hand, if center was included in construction (yielding $\widehat{\mathit{\beta }}$), ignoring center during evaluation gave a measure of performance that could be higher or lower than the performance of the combination in a new center; that is, $AUC(\widehat{\mathit{\beta }})$ tended to be larger or smaller than $AU{C}_c(\widehat{\mathit{\beta }})$, depending on the correlation δ. As expected, ignoring center during construction generally resulted in a combination with worse performance in new centers $\left(AU{C}_c(\widehat{\mathit{\alpha }})\hspace{0.33em}\text{vs}.\hspace{0.33em}AU{C}_c(\widehat{\mathit{\beta }})\right)$.

Supplementary Material (S5.1) gives the full results.

6.2. Including Center

We conducted simulations to compare combinations constructed by RILR to those constructed by FILR. The set-up of these simulations is similar to those described in Section 6.1. When center was a case mix variable, f(c) = 0 and logit(γ_c) ∼ F with mean 0 and variance ${\sigma }_{{\gamma }_c}^2$. When center was a calibration variable, γ_c = 0.5 or 0.1 and f(c) = F with mean 0 and variance ${\sigma }_{f(c)}^2$. Finally When center was a confounder, logit(γ_c) ∼ F with mean 0 and variance ${\sigma }_{{\gamma }_c}^2$, f(c) = F with mean 0 and variance ${\sigma }_{f(c)}^2$, and Corr(logit(γ_c), f(c)) = δ. We varied F (Normal, Gumbel, Laplace, or Uniform), ${\sigma }_{{\gamma }_c}^2$ (0.5, 1, 3, (0.5, 1.5), or (1, 5)), ${\sigma }_{f(c)}^2$ (1, 5, or (2,8)) and δ (−0.5, 0, 0.5). We considered some settings where the variances of logit(γ_c) (or f(c)) were not constant (those pairs of values in parentheses); in these scenarios, half of the centers were assigned one value of ${\sigma }_{{\gamma }_c}^2$ (or ${\sigma }_{f(c)}^2$) and the remainder were assigned the other. When center was a calibration variable, γ_c = 0.5 in most simulations; however, to study the impact of concordance, we also considered simulations where γ_c = 0.1. This was also the motivation for including large values of ${\sigma }_{{\gamma }_c}^2$ when center was a case mix variable or a confounder.

Linear combinations were constructed in training data (which had either 6 or 200 centers) via logistic regression, where center was either (i) incorporated using RILR assuming $b_c\stackrel{iid}{~}N\left(0,{\sigma }^2\right)$ or (ii) incorporated using uFILR (in the case of 6 centers) or cFILR (in the case of 500 centers). The fitted biomarker combinations based on RILR and FILR were evaluated in test data, which consisted of a single new center with 10 000 observations. These simulations were repeated 500 times.

In Figure 3, we present the results for m = 500 centers in the training data with F = Normal, ${\sigma }_{{\gamma }_c}^2=1$, ${\sigma }_{f(c)}^2=5$, γ_c = 0.5 when center was a calibration variable, and δ = −0.5 when center was a confounder. In all scenarios, the results from FILR were close to the true values. The differences in the coefficient estimates when RILR was used are clear, particularly when center was a calibration variable or a confounder. This led to substantially different conditional AUCs for RILR compared to FILR, especially when center was a calibration variable. The differences in the AUC were small when center was a case mix variable; in this setting, the differences in the coefficient estimates were not as large, and the AUC, which is a rank-based measure, was able overcome these more modest perturbations. The differences in the coefficient estimates and the AUC were considerably larger when center was a calibration variable than when it was a confounder. The full results are given in the Supplementary Material (S5.2). In general, we see that the differences between RILR and FILR tended to be smaller when there were fewer centers (m = 6 vs. m = 500), ${\sigma }_{f(c)}^2$ was small, or ${\sigma }_{{\gamma }_c}^2$ was large.

Figure 3:

Simulation results comparing random and fixed intercept logistic regression for m = 500 in the training data, where F = Normal, ${\sigma }_{{\gamma }_c}^2=1$, ${\sigma }_{f(c)}^2=5$, γ_c = 0.5 when center was a calibration variable, and δ = 0.5 when center was a confounder. The median and interquartile ranges across the simulations are reported. The columns in each plot correspond to different roles for center. The results based on FILR are displayed as blue triangles and the results based on RILR are displayed as red circles. The results for the biomarker coefficients are shown in the first two plots, and the results for the AUC in a single new test center are shown in the third plot. In each plot, the dashed horizontal line indicates the true value.

The superior performance of FILR persisted even when we considered situations where there were 500 centers and, on average, 7–12% were concordant (Supplementary Material S5.2). In simulations not designed specifically to have high concordance, up to 2% of centers were concordant, on average.

Finally, we evaluated the estimate of the overall fixed intercept provided by RILR and found absolute biases of more than 20% in many scenarios (Supplementary Material S5.2).

We have provided the R functions used to conduct these simulations in the Supplementary Material (S6).

7. Application to the TRIBE-AKI Study

We applied the methods we have discussed to data from the TRIBE-AKI study. Recall that this is a study of 1219 adults undergoing cardiac surgery at six medical centers, and there is interest in using biomarkers to provide an earlier diagnosis of post-operative AKI. All participants provided written informed consent and details regarding subject recruitment and sample collection and storage have been previously reported.⁵ These data are used as illustration and not to report new findings of the TRIBE-AKI study. We considered three biomarkers, urine NGAL, plasma h-FABP, and plasma TNI, and used the measurements taken immediately after surgery. After removing observations with missing values for any of these biomarkers, 962 observations remained. The three biomarkers were log-transformed.

First we assess the role of center in this study. Since we are considering diagnostic biomarkers, we evaluated the distribution of the biomarkers in each center among AKI controls, and found variation in these distributions across centers (Figure 4). Additionally, the center-specific AKI prevalences were between 7.8% and 22.9%. These results strongly suggest that center is a confounder in this study.

Figure 4:

Distribution of log urine NGAL, log plasma h-FABP, and log plasma TNI in the TRIBE-AKI study among controls. The biomarker distributions are stratified by center.

We constructed linear biomarker combinations and evaluated their performance by estimating the center-adjusted AUC. We corrected this estimate for resubstitution bias by bootstrapping the individual observations.

The biomarker combination estimated by FILR was

$$0.025\ast \log (\text{NGAL})+1.103\ast \log (\text{h}-\text{FABP})-0.065\ast \log (\text{TNI}).$$

The optimism-corrected center-adjusted AUC for this combination was 0.6823 (95% confidence interval: 0.6387, 0.7344). The combination estimated by RILR was

$$0.054\ast \log (\text{NGAL})+1.096\ast \log (\text{h}-\text{FABP})-0.065*\log (\text{TNI}).$$

The optimism-corrected center-adjusted AUC for this combination was 0.6806 (0.6328, 0.7337). When center was ignored during construction, the estimated combination was

$$0.081*\log (\text{NGAL})+1.103\ast \log (\text{h}-\text{FABP})-0.094*\log (\text{TNI})$$

and the optimism-corrected center-adjusted AUC for this combination was 0.6811 (0.6321, 0.7349). Thus, in these data, the three fitted combinations were quite similar, and, correspondingly, the gains offered by FILR in terms of the center-adjusted AUC were very modest.

8. Discussion

We have created a unified framework for constructing and evaluating biomarker combinations in multicenter studies, including a taxonomy to differentiate the role center can play, tools for identifying the role of center, and methods for constructing a biomarker combination and evaluating its performance. Essentially, by conditioning on center in both the construction and evaluation of biomarker combinations, we obtain combinations and measures of performance that are unaffected by center differences. Given that such center differences are often not scientifically relevant and are expected to vary in magnitude from center to center, using conditional approaches for construction and evaluation of biomarker combinations is advised in order to avoid allowing center differences to influence either the combination itself or the assessment of its performance. The concepts and methods we describe apply to biomarker combinations and also to combinations of biomarkers and clinical or demographic variables.

The center-specific AUC may not be the same across centers; in this situation, it is generally informative to evaluate the variability in the center-specific AUCs across centers. This offers some indication of how the biomarker combination might be expected to perform in a new center, if the centers included in the evaluation are “similar” to the new centers. However, when assessing the center-specific AUCs, it is important to keep in mind that AUC estimates from centers with fewer observations are less reliable.

Different sampling schemes could affect the estimated weights ${\widehat{w}}_c$, which could in turn affect the estimated center-adjusted AUC. The center-specific AUC itself is unaffected by case-control sampling within each center⁴¹ and the center-adjusted AUC is unaffected by center-dependent sampling among controls,³⁵ though the asymptotic results we have provided may not hold under certain sampling schemes. If a multicenter study also involves matching, care must be taken to adjust the AUC for the matching in addition to center.³⁴

The issues discussed in this paper are to some degree applicable to continuous and survival outcomes. For continuous outcomes, within the linear model framework, when the random intercept model E(Y |X, C = c, b_c) = b_c + τ₀ + τ₁X holds, the fixed parameters (τ₀, τ₁) are equivalent to the parameters from a marginal model. However, when the assumptions of this random intercept model are violated, differences between within- and between-center effects may lead to some of the same issues described above. ¹¹ For survival outcomes, in the proportional hazards framework, the frailty model is the analogue of the RILR model, and the (stratified) Cox model is the analogue of the FILR model. The frailty model makes some of the same assumptions as the RILR model, including independence of the random intercept and the predictors. The issue of bias resulting from violations of this assumption are not well-explored in the literature, but the occurrence of such bias certainly seems plausible.

Future research will consider approaches that do not rely on empirical estimates of the center-specific AUC, perhaps by modeling the fitted combination parametrically (e.g., using a model to relate the combination to center among controls);³⁶ such an approach may be useful when there are a large number of very small centers, as might happen when the “centers” are clinicians. In these settings, the empirical center-specific AUC estimate may be unreliable, and an alternative estimate may be preferable.

An important contribution of this work is that it demonstrates that methods often applied to multicenter biomarker data are frequently not appropriate. Biomarkers hold great potential for use as diagnostic and prognostic tools, but have for the most part been relatively disappointing thus far. Much of the problem has been blamed on “validation failures”; that is, biomarkers that are found to be quite promising initially, but are never used in clinical practice due to disappointing results in follow-up studies.⁴⁸ Thus, to the extent possible, it is important to recognize aspects of study design, conduct, and analysis that require special attention when developing biomarker combinations. Carefully addressing these issues can increase the likelihood of identifying clinically useful combinations, ultimately leading to improvements in patient care.