Comparing and combining data from immune assays based on left-censored multivariate normal model assuming common assay differences across settings

Abstract

In vaccine research towards the prevention of infectious diseases, immune response biomarkers serve as an important tool for comparing and ranking vaccine candidates based on their immunogenicity and predicted protective effect. However, analyses of immune response outcomes can be complicated by differences across assays when immune response data are acquired from multiple groups/laboratories. Motivated by a real-world problem to accommodate the use of two different neutralization assays in COVID-19 vaccine trials, we propose methods based on left-censored multivariate normal model assuming common assay differences across settings, to adjust for differences between assays with respect to measurement error and the lower limit of detection. Our proposed methods integrate external paired-sample data with bridging assumptions to achieve two objectives, both using pooled data acquired from different assays: i) comparing immunogenicity between vaccine regimens, and ii) evaluating correlates of risk. In simulation studies, for the first objective, our method leads to unbiased calibrated assay mean with good coverage of bootstrap confidence interval, as well as valid test for immunogenicity comparison, while the alternative method assuming constant calibration model between assays leads to biased estimate of assay mean with undercoverage problem and invalid test with inflated type-I error; for the second objective, in the presence of noticeable left-censoring rate, our proposed method can drastically outperform the existing method that ignores left-censoring, in terms of reduced bias and improved precision. We apply the proposed methods to SARS-CoV-2 spike-pseudotyped virus neutralization assay data generated in vaccine and convalescent samples by two different laboratories.

1 |. INTRODUCTION

Immune response biomarkers play a key role in vaccine research towards infectious disease prevention. Vaccine efficacy trials are costly and challenging to conduct; multiple phase I/II studies are routinely conducted to assess the immunogenicity of candidate vaccines prior to launching an efficacy trial. An immune “correlate of protection”¹, or immune response biomarker associated with the protective effect of a vaccine, if identified and validated, can be used to screen, rank, and compare candidate vaccine regimens in immunogenicity studies as well as to guide the development of vaccine candidates. A recent immune correlates analysis of the Moderna COVE efficacy trial of the mRNA-1273 COVID-19 vaccine found that the evaluated binding antibody and neutralizing antibody markers correlated inversely with symptomatic COVID-19 risk and directly with mRNA-1273 efficacy against symptomatic COVID-19, providing evidence for each of these markers as such a correlate of protection².

In studies of immune response biomarkers, collaboration among multiple laboratories is desirable for expediting the process. A challenge that often occurs in these multi-lab collaborations is the need to compare and/or combine data acquired using different lab assays. The research described in this paper are critically needed to compare and combine neutralization assay data from two different labs in COVID-19 vaccine correlates analyses. In each of the United States Government (USG)-sponsored phase 3 COVID-19 vaccine efficacy trials, neutralizing antibodies (nAb) are assessed by one of two different pseudovirus-based SARS-CoV-2 neutralization assays, one performed at Duke University and one at Monogram Biosciences. While both assays have been rigorously validated in accordance with the ICH/FDA guideline^3,4, they differ in various aspects such as pseudovirus preparation, cell lines, plate layouts, dilution scheme, and measurement error⁵. Although nAb immune responses to the ancestral strain of SARS-CoV-2 were identified as a correlate of COVID-19 vaccine effect based on data generated by the Duke assay in the Moderna COVE phase 3 trial², the Monogram assay will be used in the correlates work for other USG-sponsored COVID-19 vaccine trials and the mixed of these two assays may be used in the evaluation of emerging SARS-CoV-2 variant-targeting vaccine candidates. When run on identical samples, the Duke assay yields generally lower titers than the Monogram assay⁵. Moreover, the two assays have different assay ranges or limits of detection (LOD), which is further complicated by the fact that the LOD of an assay (e.g. Duke nAb assay in this application) can change over time and thus differ between studies. These differences between assays, if not properly accounted for, might generate misleading results in immune correlates analysis and misguide regulatory decisions regarding vaccine candidates. We have previously detailed the application of three different statistical approaches to calibrate the two assays and enable cross-assay data comparison, yet a number of questions still remain⁵. Later in this paper we will present an example for analyzing SARS-CoV-2 pseudovirus neutralization assay data measured by both Duke and Monogram assays. Our data application consists of nAb titer readouts from both assays in 248 COVID-19 convalesent serum samples and 30 mRNA-1273 vaccine recipient serum samples collected at Day 57 (4 weeks post second vaccination) from Moderna phase 1 trials^6,7. From Figure 1, for both types of samples, the nAb titers are higher for the Monogram assay compared to the Duke assay, with both assays subject to left censoring, demonstrating the importance of developing appropriate statistical methods for calibration between the Duke and Monogram assays in order to put them on the same scale for comparison and combination. In this paper we describe new statistical methodology based on a left-censored multivariate normal model assuming common assay differences across settings, to adjust for differences between assays with respect to measurement error and the lower limit of detection of the assays. We propose methods to address two important questions. The first question is to compare immunogenicity between vaccine regimens when immune responses in different vaccine arms are measured by different assays, which can happen in cross-protocol studies that compare vaccine candidates from different immunogenicity trials. The second question is to assess immune response biomarkers as correlates of risk (CoR)¹ — i.e. immune responses associated with risk of a clinical outcome — based on efficacy and biomarker data combined from multiple studies that utilize different assays, such as the two different nAb assays used in the context of COVID-19 trials. Methods for handling LOD and measurement error have been widely studied in disease association analysis. For example, for handling left-censoring due to lower limit of detection (LLOD), various algorithms have been proposed such as replacing values below the LLOD by a low constant value⁸, or by random draws from the truncated part of the distribution estimated using a likelihood method^9,10,11 or quantile regression¹². Different methods for measurement error correction have also been developed utilizing internal or external studies that assume the existence of replicate measurements or measurement of the true covariate without measurement error¹³. Few investigations; however, have been conducted for the problem settings considered here that compare or combine study data utilizing different assays. Previously,¹⁴ used paired assay data to correct for measurement error when comparing different assays or combining assay data in an association study. A heuristic approach that calibrates one assay to the scale of the other assay was also proposed in⁵. However, none of the approaches has rigorously addressed the issue that the assay LOD can differ both between assays and between studies. Moreover, the performance of these approaches for comparing and combining assay data in the presence of the LOD has not been rigorously investigated. We aim to fill these gaps by developing new methods that combine paired assay data with bridging assumptions to address the two questions mentioned above accommodating LOD and measurement error differences across assays. We conduct extensive simulation studies to demonstrate the superior performance of our proposed methods compared to alternative methods and apply the proposed methods to the SARS-CoV-2 pseudovirus neutralization assays described above.

FIGURE 1.

Observed neutralizing antibody (nAb) titers by Monogram and Duke in COVID-19 convalescent samples (a: ID50)(b: ID80); observed and calibrated nAb titers by Monogram and Duke in Day 57 samples from Moderna mRNA-1273 vaccine recipients (c: ID50)(d: ID80).

While our methods were developed to address scientific questions related to immune responses, immunogenicity comparison and CoR meta-analysis represent two general questions in biomedical research: i) comparison of treatment effect on biomarker endpoints that are measured by heterogenenous assays, and ii) estimating true underlying biomarker-disease association based on pooled data with biomarkers measured using heterogeneous assays. That is, COVID outcome, vaccination, and nAb response in our problem setting can correspond to a general clinical endpoint, treatment or prevention strategy for this clinical endpoint, and a biomarker response that can be measured by heterogeneous assay, respectively. Our proposed methods thus have more general use in biomedical fields. There were many studies in existing literature on meta-analysis studying treatment effect combining data from heterogeneous studies, where heterogeneity across studies was either modeled by a fixed effect model with heterogeneity explained by some covariates or by a random effect model assuming random error in each study^15,16,17,18. These methods cannot be utilized to handle heterogeneity due to the use of different lab assays that can include both mean and measurement differences, since assay effect is completely confounded with treatment effect. To address the use of heterogeneous assay it requires paired study that models the relationship between assays and then bridges the results to the new setting of interest. One approach often used by lab researchers is to construct a between-assay calibration model using the paired data, based on observed assay value^19,5,20; the calibration curve can be used to calibrate assay in the new setting. We will show later in the simulation and data application the advantage of our proposed method over this comparative approach. For the problem of estimating association between underlying true covariate value and disease risk when covariate is measured with error, regression calibration²¹ has been a commonly used approach among others¹³. Development for this question was mainly in problem settings with one covariate measured with error. The development in¹⁴ represents an extension of the regression calibration idea to the meta-analysis setting with two different normally distributed covariate measures. Our proposed method in this paper will have more general practical applicability by allowing for left-censoring of covariate measurements across studies.

2 |. METHODS

In this section, we first describe the study setting/notation and then present the methodology framework to address the questions of immunogenicity comparison and CoR assessment with measures from different assays. Motivated by the SAS-CoV-2 pesudovirus nAb assay problem, here we consider the involvement of two different lab assays for measuring the same type of immune response (e.g. the nAb response) and consider assays with a lower limit of detection. The methods developed can be readily generalized to accommodate more than two assays as well as censoring by upper limit of detection. As will be presented in Supplementary Material, when answering questions regarding multiple assays, once one fixes a reference assay, the strategy to handle multiple assay essentially boils down to the calibration of each other assay to the scale of this reference assay so the strategy described below for dealing with two assays can be easily extended to handle numerous assays.

2.1 |. Settings and Studies

2.1.1 |. Paired study sample for modeling

Let $X_1^{*}$ and $X_2^{*}$ indicate measurement of two lab assays that characterize the same type of immune response. Suppose data are available from a paired sample of size n each with measurements using both assays, and suppose assay 1 and assay 2 have an LLOD a₁ and a₂, respectively, in the paired study. That is, $X_1^{*}\ge a_1$ and $X_2^{*}\ge a_2$. Based on the paired study sample, we will propose methods to extract information on the relationship between the two assays, which can be applied to new independent studies described next either for comparing immunogenicity between vaccines or for CoR meta-analyses assessing the association between an immune response and the risk of the clinical outcome.

2.1.2 |. Independent studies in the new setting for comparing and combining assay data

Now suppose $X_1^{*}$ and $X_2^{*}$ are measured from independent samples of size m₁ and m₂, respectively, in a new setting (e.g. new populations or vaccines), again subject to measurement error and left censoring. However the LLODs of assays 1 and 2 in the new setting do not necessarily equal those in the old setting (i.e. in the paired study). Here we use b₁ and b₂ to indicate the LLODs of assays 1 and 2, respectively, in the new setting. We call the sub-sample with assay 1 measurements sub-sample 1 (of size m₁) and the sub-sample with assay 2 measurements sub-sample 2 (of size m₂). We consider two types of studies in the new setting. The first type of study is the immunogenicity study. The goal of the study is to compare immunogenicity between two vaccine regimens based on immune response data measured by assay 1 and assay 2, respectively. The second type of study is a meta-analytical CoR study that includes data from efficacy trials with immune response data measured using either assay 1 or 2 in each trial, where the objective is to assess the association between the underlying true immune response and the risk of the clinical outcome.

2.2 |. Measurement Error Modeling

In the paired study, suppose ($X_1^{*}$, $X_2^{*}$) are left-censored versions of variables (X₁, X₂), where X₁, X₂ each equal some underlying true immune response U measured with independent error plus possible assay-specific systematic bias (or location shift) δ₁ and δ₂. We assume U and measurement error are normally distributed. That is:

$$\begin{gathered} X_1={\delta }_1+U+{ϵ}_1, \\ {X}_2={\delta }_2+U+{ϵ}_2, \end{gathered}$$ (1)

with $U~N\left({\mu }_0,{\sigma }_U^2\right)$, ${ϵ}_1~N\left(0,{\sigma }_{ϵ1}^2\right)$ and ${ϵ}_2~N\left(0,{\sigma }_{ϵ2}^2\right)$ independent of each other. In other words, $X_1^{*}=max\left(X_1,a_1\right)$, $X_2^{*}=max\left(X_2,a_2\right)$ for pre-specified LLOD a₁ and a₂.

Here the pre-censoring variables (X₁, X₂) are bivariate normal with mean μ = (μ₁, μ₂)^T = (μ₀ + δ₁, μ₀ + δ₂)^T and variance Σ. Let $\Sigma =\left(\begin{array}{cc}{\sigma }_1^2& \rho {\sigma }_1{\sigma }_2\\ \rho {\sigma }_1{\sigma }_2& {\sigma }_2^2\end{array}\right)$. The probability of observing $\left(X_1^{*},X_2^{*}\right)=\left(t_1,t_2\right)$ is

$${(2\pi )}^{-1}{(\text{det }\Sigma )}^{-1/2}\text{\hspace{0.17em}exp\hspace{0.17em}}\left\{-\frac{1}{2}{\left(\left(\begin{array}{l}{t}_1\hfill \\ t_2\hfill \end{array}\right)-\mu \right)}^{\top }{\Sigma }^{-1}\left(\left(\begin{array}{l}{t}_1\hfill \\ t_2\hfill \end{array}\right)-\mu \right)\right\}\text{ if }{t}_1>a_1,t_2>a_2$$

$$\Phi \left(a_1;{\mu }_1,{\sigma }_1^2\right)\varphi \left(t_2;{\mu }_2,{\sigma }_2^2\right)\text{ if }{t}_1=a_1,t_2>a_2$$

$$\varphi \left(t_1;{\mu }_1,{\sigma }_1^2\right)\Phi \left(a_2;{\mu }_2,{\sigma }_2^2\right)\text{ if }{t}_1>a_1,t_2=a_2$$

$$\Phi \left(a_1;{\mu }_1,{\sigma }_1^2\right)\Phi \left(a_2;{\mu }_2,{\sigma }_2^2\right)\text{ if }{t}_1=a_1,t_2=a_2,$$

where ϕ and Φ are the probability density function (pdf) and the cumulative density function (cdf), respectively, for a standard normal distribution. This allows us to derive the maximum likelihood estimator (MLE) of μ₁, μ₂ and Σ by maximizing the likelihood of observed $X_1^{*}$, $X_2^{*}$ in the paired sample $ℒ(\mu ,\Sigma )={\prod }_{i=1}^{n}Pr\left(X_{1i}^{*},X_{2i}^{*}\right)$. The estimated MLE ${\widehat{\mu }}_1$, ${\widehat{\mu }}_2$ and $\widehat{\Sigma }$ in the paired study, together with the bridging assumption we will make next regarding the relationship between assay differences in the paired study and in the new setting, will allow us to answer the assay comparison and combination questions in the new setting.

2.3 |. Bridging

To analyze data in the new setting based on measurements using both assays, we need to make bridging assumptions to borrow information from the paired study data to calibrate the assay measure in the new setting. Next we propose bridging assumptions and calibration methods for two types of analysis: immunogenicity comparison and CoR analysis.

2.3.1 |. Immunogenicity comparison

First, consider a new setting where we have immunogenicity data from two independent groups of individuals, with each group receiving a different vaccine regimen. The immune responses elicited by vaccine 1 and by vaccine 2 are measured with assay 1 and assay 2, respectively. The objective of the study is to compare average immunogenicity between the two vaccine regimens.

Here we consider calibrating assay 2 data to the scale of assay 1 with $E\left(X_1^{*}\mid X_2^{*}\right)$ and then compare the vaccine regimens on the scale of assay 1, with respect to the mean of measured assay 1 data (in sub-sample 1) and the mean of calibrated assay 1 values based on measured assay 2 data (in sub-sample 2). A similar comparison can be made by calibrating assay 1 to the scale of assay 2 measures with $E\left(X_2^{*}\mid X_1^{*}\right)$. To achieve this we make a transportability or bridging assumption that the difference between assay 1 and assay 2 prior to censoring is the same in the old (paired study) and the new (independent study) settings, i.e., the difference in distributions between the pre-censoring variables X₁ and X₂ is the same between the two settings, had a common set of samples been measured in the later setting. Estimation of this transportable relationship using the paired data together with estimation of the X₂ distribution in the new setting allows estimation of $E\left(X_1^{*}\mid X_2^{*}\right)$ in the new setting.

Without loss of generality, suppose the variability of (X₁, X₂) is the same between the paired study and sub-sample 2 (in the new setting) and assume δ = δ₁ − δ₂, the difference between E(X₁) and E(X₂) from the same individual, is also the same across settings. This can happen when pre-censoring immune response measurements X₁ and X₂ follow a location-shift model with the same location shift across settings. Let superscript ^† indicate the new setting and let (${\mu }_1^{†}$, ${\mu }_2^{†}$) and Σ^† indicate mean and variance of (X₁, X₂) among sub-sample 2. We also let ${\Sigma }^{†-1}=\left(\begin{array}{ll}{s}_{11}^{†}\hfill & s_{12}^{†}\hfill \\ s_{12}^{†}\hfill & s_{22}^{†}\hfill \end{array}\right)$. We make the bridging assumption

$$\delta ={\mu }_1-{\mu }_2={\mu }_1^{†}-{\mu }_2^{†}.$$

Let $\widehat{\delta }={\widehat{\mu }}_1-{\widehat{\mu }}_2$ be MLE of δ estimated from the paired data, and ${\widehat{\mu }}_2^{†}$ be MLE for the mean of X₂ in sub-sample 2 in the new setting, estimated based on observed $X_2^{*}$, assumed to be a left-censored normal random variable. We can then estimate ${\widehat{\mu }}_1^{†}=\widehat{\delta }+{\widehat{\mu }}_2^{†}$. We adopt ${\widehat{\Sigma }}^{†}=\widehat{\Sigma }$ based on the assumption of constant variance for (X₁, X₂) between paired sample and sub-sample 2. The mean of $X_1^{*}$ conditional on $X_2^{*}$, i.e., $E\left(X_1^{*}\mid X_2^{*}=t_2\right)$ in sub-sample 2 can be derived as a functional form of model parameters (${\mu }_1^{†}$, ${\mu }_2^{†}$, Σ^†), based on the assumption of left-censored bivariate-normal distribution of ($X_1^{*}$, $X_2^{*}$) in sub-sample 2 as below.

(1) when t₂ > b₂, $E\left(X_1^{*}\mid X_2^{*}=t_2\right)$ in sub-sample 2 equals

$$\begin{gathered} {\int }_{-\infty }^{b_1}{b}_1\times P\left(X_1=x_1\mid X_2=t_2\right)d{x}_1+{\int }_{b_1}^{\infty }P\left(X_1=x_1\mid X_2=t_2\right)x_{1}d{x}_1 \\ =b_1\times P\left(X_1\le b_1\mid X_2=t_2\right)+\frac{{\int }_{b_1}^{\infty }P\left(X_1=x_1\mid X_2=t_2\right)x_{1}d{x}_1}{{\int }_{b_1}^{\infty }P\left(X_1=x_1\mid X_2=t_2\right)d{x}_1}\times {\int }_{b_1}^{\infty }P\left(X_1=x_1\mid X_2=t_2\right)d{x}_1 \\ =b_1\times P\left(X_1\le b_1\mid X_2=t_2\right)+P\left(X_1>b_1\mid X_2=t_2\right)\times E\left(X_1\mid X_1>b_1,X_2=t_2\right) \\ =b_1\times \Phi \left(b_1;{\mu }_{1\mid t_2}^{†},{\sigma }_{1.2}^{†2}\right)+\left(1-\Phi \left(b_1;{\mu }_{1\mid t_2}^{†},{\sigma }_{1.2}^{†2}\right)\right)\times \left(v+\frac{\varphi \left(b_1-v\right)}{1-\left(b_1-v\right)\sqrt{s_{11}^{†}}}\right); \end{gathered}$$

(2) when t₂ = b₂, $E\left(X_1^{*}\mid X_2^{*}=t_2\right)$ in sub-sample 2 equals

$$\begin{gathered} b_1\times P\left(X_1\le b_1\mid X_2\le b_2\right)+\frac{{\int }_{-\infty }^{b_2}P\left(X_2=x_2\right)E\left(X_1\mid X_1>b_1,X_2=x_2\right)P\left(X_1>b_1\mid X_2=x_2\right)d{x}_2}{P\left(X_2\le b_2\right)} \\ =b_1\times \Phi \left(b_1;{\mu }_{1\mid b_2}^{†},{\sigma }_{1.2}^{†2}\right)+\frac{{\int }_{-\infty }^{b_2}\varphi \left(x_2;{\mu }_2^{†},{\sigma }_2^{†2}\right)\left(v+\frac{\varphi \left(b_1-v\right)}{1-\left(b_1-v\right)\sqrt{s_{11}^{†}}}\right)\left(1-\Phi \left(b_1;{\mu }_{1\mid x_2}^{†},{\sigma }_{1.2}^2\right)\right)d{x}_2}{\Phi \left(b_2;{\mu }_2^{†},{\sigma }_2^{†2}\right)}, \end{gathered}$$

where ${\mu }_{1\mid t_2}^{†}=E\left(X_1\mid X_2=t_2\right)$, ${\sigma }_{1.2}^{†2}=var\left(X_1\mid X_2\right)$ in sub-sample 2, and $v\equiv v\left(t_2\right)={\mu }_1^{†}-\left(t_2-{\mu }_2^{†}\right)s_{12}^{†}/s_{11}^{†}$.

The calibrated $X_2^{*}$ values in sub-sample 2 — based on $E\left(X_1^{*}\mid X_2^{*}\right)$ estimated using sub-sample 2 — can then be used to compare with the assay 1 measures in sub-sample 1.

More generally, the location-shift bridging assumption could be relaxed to allow for change in variability of X₁ and X₂ across settings. For example, one can assume that measurement error in X₁ and X₂ (i.e. ${\sigma }_{ϵ1}^2$ and ${\sigma }_{ϵ2}^2$) is constant across settings but allow between-subject variability (i.e. ${\sigma }_U^2$) to change across settings. Knowing ${\sigma }_{ϵ2}^2$ (e.g. from pilot lab data) would allow us to recover ${\sigma }_U^{†2}$ in sub-sample 2, which together with information about ${\sigma }_{ϵ1}^2$ will allow the identification of Σ^† in sub-sample 2. We call our proposed method for immunogenicity comparision based on left-censored normal distribution the Regression Calibration Proposed Immunogenicity Comparison (RCP.I) Method.

Comparative Methods for Immunogenicity Comparison

For the immunogenicity comparison problem, previously¹⁴ has taken a heuristic approach that computes the nonparametric mean difference between observed assay value in the pair sample and assumes the same difference holds between means of observed assay value in the new setting. It is easy to see that this is not a reasonable assumption when the observed assay value is left-censored, especially if an assay’s LLOD can differ between the old and new settings. In contrast, our proposed RCP.I method assumes a more feasible assumption of constant difference in underlying assay means that is not affected by LLOD. Another heuristic bridging approach that has oftentimes been adopted by lab researchers in practice is to build a calibration model for mean of assay 1 value $X_1^{*}$ conditional on assay 2 value $X_2^{*}$, i.e. $E\left(X_1^{*}\mid X_2^{*}\right)$, and assumes this conditional mean model is transportable between the old and new settings [e.g.¹⁹ and Approach 3 in⁵]. Based on this assumption, one can estimate $E\left(X_1^{*}\mid X_2^{*}\right)$ from the paired data and directly apply it in the new setting to calibrate $X_2^{*}$ in sub-sample 2. A heuristic way is to model $E\left(X_1^{*}\mid X_2^{*}\right)$ as a linear function of $X_2^{*}$. We call this approach the Regression Calibration Linear Immunogenicity Comparison (RCL.I) approach. However, as we will show below, for bivariate normal (X₁, X₂), even under a simple location-shift model without LLOD (i.e. $X_1^{*}=X_1$ and $X_2^{*}=X_2$) such that the linear conditional mean model holds, the transportable linear model assumption can be violated. Specifically, one can show that E(X₁|X₂) in the paired study equals to (μ₁ − μ₂ρσ₁∕σ₂) + (ρσ₁∕σ₂)X₂. Suppose there is a location-shift η for X₁ and X₂ from the old to the new setting, i.e. ${\mu }_1^{†}={\mu }_1+\eta$ and ${\mu }_2^{†}={\mu }_2+\eta$. In this case, E(X₁|X₂) in the new setting would have the same slope as in the old setting but would have the same intercept only if ${\mu }_1-{\mu }_2\rho {\sigma }_1/{\sigma }_2={\mu }_1^{†}-{\mu }_2^{†}\rho {\sigma }_1/{\sigma }_2$, i.e. $\rho {\sigma }_1/{\sigma }_2={\sigma }_U^2/\left({\sigma }_U^2+{\sigma }_{ϵ2}^2\right)$ if η ≠ 0. In other words, unless the old and new settings are exactly the same, the intercept term for E(X₁|X₂) would differ between the two settings if there is measurement error in assay 2 (i.e. σ_ϵ2 > 0). If in addition, the between-individual variability σ_U differs between the old and new settings while measurement error for X₂ remains constant, then the slope term would differ between the old and new settings as well. The existence of left censoring due to the LLOD would also lead to slope difference, as will be shown later in the simulation study. For immunogenicity comparison, we will conduct simulation study next to demonstrate the advantage of the proposed RCP.I method relative to the alternative RCL.I approach for estimating calibrated mean response and testing for difference between immunogenicity.

2.3.2 |. CoR meta-analysis

In this section we consider the meta-analytical question of CoR analysis combining data measured using two different assays. Let Y indicate the binary clinical outcome for CoR analysis, e.g. the symptomatic COVID-19 event. Suppose two independent samples, sub-sample 1 and sub-sample 2, are available with information on Y as well as on the immune response measured by assay 1 and assay 2, respectively; each sample could be a random sample from the target population or a case control sample with case enrichment for cost efficiency.

Suppose the risk of Y conditional on the true underlying immune response U follows a linear logistic model

$$\text{logit}(P(Y=1\mid U))={\beta }_0+{\beta }_{1}U.$$ (2)

The objective is to estimate the association between the true underlying immune response U and disease risk, i.e. the log odds ratio (OR) parameter β₁ in risk model (2) combining the two independent sub-samples. We adopt the regression calibration idea that has been widely used to correct for measurement error in the rare disease setting²¹. Specifically, we propose to substitute U with estimated $E\left(U\mid X_1^{*}\right)$ in sub-sample 1 and estimated $E\left(U\mid X_2^{*}\right)$ in sub-sample 2 into the logistic regression model (2).

Following derivation similar to that in Section 2.3.1, we can show that $E\left(U\mid X_1^{*}=t_1\right)$ in sub-sample 1 equals

$$\{\begin{array}{cc}{\int }_{-\infty }^{\infty }P\left(U=u\mid X_1=t_1\right)udu=E(U)+\left(t_1-{\mu }_1^{†}\right){\rho }^{†}{\sigma }_U^{†}/{\sigma }_1^{†}& \text{when }{t}_1>b_1\\ \frac{{\int }_{-\infty }^{b_1}P\left(X_1=x_1\right)E\left(U\mid X_1=x_1\right)d{x}_1}{P\left(X_1\le b_1\right)}=\frac{{\int }_{-\infty }^c\varphi \left(x_1;{\mu }_1^{†},{\sigma }_1^{†2}\right)\left(E(U)+\left(x_1-{\mu }_1^{†}\right){\rho }^{†}{\sigma }_U^{†}/{\sigma }_1^{†}\right)d{x}_1}{\Phi \left(b_1;{\mu }_1^{†},{\sigma }_1^{†2}\right)}& \text{when }{t}_1=b_1,\end{array}$$

and similarly $E\left(U\mid X_2^{*}=t_2\right)$ in sub-sample 2 equals

$$\{\begin{array}{cc}{\int }_{-\infty }^{\infty }P\left(U=u\mid X_2=t_2\right)udu=E(U)+\left(t_2-{\mu }_2^{†}\right){\rho }^{†}{\sigma }_U^{†}/{\sigma }_2^{†}& \text{when }{t}_2>b_2\\ \frac{{\int }_{-\infty }^{b_2}P\left(X_2=x_2\right)E\left(U\mid X_2=x_2\right)d{x}_2}{P\left(X_2\le b_2\right)}=\frac{{\int }_{-\infty }^{b_2}\varphi \left(x_2;{\mu }_2^{†},{\sigma }_2^{†2}\right)\left(E(U)+\left(x_2-{\mu }_2^{†}\right){\rho }^{†}{\sigma }_U^{†}/{\sigma }_2^{†}\right)d{x}_2}{\Phi \left(b_2;{\mu }_2^{†},{\sigma }_2^{†2}\right)}& \text{when }{t}_2=b_2.\end{array}$$

Note that a location shift in calibrated immune response does not affect the OR estimate. Therefore we assume δ₁ = 0 in (1) such at E(U) = μ₁.

MLE modeling from the paired data and information about measurement error in assays 1 and 2 allow us to compute ${\sigma }_U^{†2}$. Replicate data from pilot study for $X_1^{*}$ and $X_2^{*}$ can be used to derive the measurement error variance ${\sigma }_{ϵ1}^2$ and ${\sigma }_{ϵ2}^2$. For example, let (X_1i, X_1j) indicate the replicate data for X₁. We have $X_{1i}^{*}$, $X_{1j}^{*}$ as being left-censored bivariate normal each with mean μ₁, variance ${\sigma }_U^2+{\sigma }_{ϵ1}^2$ and correlation ${\sigma }_U^2/\left({\sigma }_U^2+{\sigma }_{ϵ1}^2\right)$. The MLE estimator described in Section 2.2 can be used to estimate various parameters including μ₁, ${\sigma }_U^2$, and ${\sigma }_{ϵ1}^2$. Similarly, we can use replicate data for $X_2^{*}$ to estimate μ₂, ${\sigma }_U^2$, ${\sigma }_{ϵ2}^2$. We name this proposed approach for CoR analysis the Regression Calibration Proposed CoR (RCP.C) method.

Comparative Methods for Meta-Analytical CoR Analysis

For the meta-analytical CoR analysis,¹⁴ adopted a regression calibration approach assuming observed assay value follows a bivariate normal distribution, which would be problematic when there exists LLOD for the assay measurement. We name the existing method¹⁴ the Regression Calibration Normality CoR (RCN.C) method. Our proposed approach extends RCN.C by incorporating left-censoring into the modeling. Later in simulation study we will demonstrate the importance of this development when censoring rate in the assay measurement is not ignorable.

Asymptotic normality of the proposed estimators for mean immunogenicity comparison and odds ratio estimation follow standard likelihood based theory. We utilize nonparametric bootstrap procedure for standard error estimation where resampling of individuals is performed for both the paired data and the data in the new setting.

3 |. SIMULATION STUDIES

In this section, we conduct numerical studies to assess the performance of our proposed methods for comparing the immunogenicity of two vaccines measured by two different assays and for CoR analysis combining assay data.

Paired study

Assume there is a paired study sample of size n = 30 with immune responses from each participant measured by both assay 1 and assay 2, with correlation 0.91 between the pre-censoring variables X₁ and X₂. We consider various combinations of mean (μ₁, μ₂), standard deviation (σ₁, σ₂), and LLOD (a₁, a₂) for X₁ and X₂ as presented in Table 1. The left-censoring rate is low for both assays 1 and 2 in settings 1–4; settings 5 and 6 have a low left-censoring rate for assay 2 and assay 1, respectively, but a substantial left-censoring rate for assay 1 and assay 2, respectively; setting 7 has a substantial left-censoring rate for both assays.

TABLE 1.

Characteristics of simulation scenarios and parameter estimates in a linear model of $X_1^{*}$ vs $X_2^{*}$ (based on 5,000 Monte Carlo simulations): β₀ and β₁ are intercept and slope in the paired study and ${\beta }_0^{†}$ and ${\beta }_1^{†}$ are intercept and slope in the new independent study, estimated using the RCL.I method. Results in parentheses are based on replacing assay values measured at LLOD with LLOD/2. F₁(a₁) = Φ(a₁; μ₁, σ₁), F₂(a₂) = Φ(a₂; μ₂, σ₂).

Scenario	μ 1	μ 2	σ 1	σ 2	ρ	${10}^{a_1}$	${10}^{a_2}$	F1(a1)	F2(a2)	ρσ 1/ σ 2		β 0	β 1	${\beta }_0^{†}$	${\beta }_1^{†}$
1	3.2	3.6	0.46	0.42	0.91	10	40	9e-7	le-6	0.997	Mean	−0.391 (−0.391)	0.997 (0.997)	−0.390 (−0.390)	0.998 (0.998)
											SD	0.323 (0.323)	0.089 (0.089)	0.215 (0.215)	0.047 (0.047)
2	3.2	3.6	0.46	0.42	0.91	80	40	0.002	le-6	0.997	Mean	−0.385 (−0.425)	0.996 (1.006)	−0.390 (−0.390)	0.998 (0.998)
											SD	0.324 (0.362)	0.089 (0.099)	0.215 (0.215)	0.047 (0.047)
3	3.6	3.2	0.42	0.46	0.91	40	10	le-6	9e-7	0.831	Mean	0.941 (0.941)	0.831 (0.831)	1.107 (1.107)	0.832 (0.832)
											SD	0.239 (0.239)	0.074 (0.074)	0.166 (0.166)	0.039 (0.039)
4	3.6	3.2	0.42	0.46	0.91	40	80	le-6	0.002	0.831	Mean	0.937 (0.969)	0.832 (0.823)	1.107 (1.107)	0.832 (0.832)
											SD	0.240 (0.266)	0.075 (0.082)	0.166 (0.166)	0.039 (0.039)
5	2.2	3.6	0.46	0.42	0.91	80	40	0.259	le-6	0.997	Mean	−0.407 (−3.094)	0.743 (1.422)	−1.340 (−1.447)	0.996 (1.010)
											SD	0.386 (0.615)	0.108 (0.161)	0.212 (0.261)	0.046 (0.056)
6	3.6	2.2	0.42	0.46	0.91	40	80	le-6	0.259	0.831	Mean	1.306 (2.566)	1.011 (0.510)	1.934 (1.985)	0.833 (0.818)
											SD	0.290 (0.151)	0.123 (0.069)	0.128 (0.154)	0.039 (0.047)
7	1.8	2.2	0.42	0.46	0.91	40	80	0.319	0.259	0.831	Mean	0.152 (0.130)	0.764 (0.740)	0.138 (0.149)	0.832 (0.828)
											SD	0.168 (0.178)	0.079 (0.077)	0.120 (0.158)	0.037 (0.048)

Immunogenicity study

Now suppose we have immunogenicity data from two different vaccines in a new population, with assay 1 and assay 2 used for measuring immune responses from vaccine 1 and vaccine 2, respectively. Assume ${\mu }_1^{†}={\mu }_1+1+s$ for s ∈ {0, 0.2, 0.5} and ${\mu }_2^{†}={\mu }_2+1$ but Σ and assay LLOD are the same as those in the paired study. Here s = 0 and s > 0 indicate equal immunogenicity between the two vaccines and higher immunogenicity of vaccine 1 relative to vaccine 2, respectively. Samples of size 100 are available from each vaccine arm.

CoR meta-analysis

For CoR meta-analysis suppose we have clinical outcome Y and immunogenicity data from vaccine arms in two efficacy trials. Suppose the two vaccine arms each include 10,000 participants with average outcome rate 0.5%. All cases and 3 times as many controls are selected for measuring immune responses. One vaccine arm has the $X_1^{*}$ measure, the other has the $X_2^{*}$ measure.

For immunogenicity comparison, we present results based on the proposed RCP.I method that estimates $E\left(X_1^{*}\mid X_2^{*}\right)$ based on a left-censored bivariate normal model. We also present the comparative regression calibration method that fits a linear model of $X_1^{*}$ on $X_2^{*}$ using paired samples and applies that to calibrate $X_2^{*}$ in sub-sample 2 of the new setting (i.e. the RCL.I approach). We present results for predicting the (unmeasured) assay 1 value elicited by vaccine 2 in sub-sample 2 using the actual assay 2 measure, and results for comparing immunogenicity between vaccine 1 and vaccine 2 in the immunogenicity study. For both RCP.I and RCL.I, for $X_1^{*}$ and $X_2^{*}$ values measured at LLOD, we consider estimation and testing based on using the corresponding LLOD values; we also present results for an alternative way of replacing $X_1^{*}$ and $X_2^{*}$ measured at LLOD with LLOD/2. Note that for RCL.I, this corresponds to replacing $X_1^{*}$ and $X_2^{*}$ measured at LLOD with LLOD/2 value when estimating the conditional mean function of $X_1^{*}$ given $X_2^{*}$. For RCP.I, estimation of model parameters in pre-censored bivariate normal distribution is not impacted; only in subsequent estimation of $E\left(X_1^{*}\mid X_2^{*}\right)$ based on left-censored bivariate normal, the LLOD value of $X_1^{*}$ is replaced with LLOD/2.

For CoR meta-analysis, we present results based on the proposed RCP.C method that calibrates $E\left(U\mid X_1^{*}\right)$ and $E\left(U\mid X_2^{*}\right)$ based on a left-censored bivariate normal model. In simulation studies, we assume measurement errors σ_ϵ1 and σ_ϵ2 are available from a pilot study. For the comparative methods, we consider a naive estimator that utilizes observed immune response values directly and the RCN.C method that assumes a bivariate-normal distribution for ($X_1^{*}$, $X_2^{*}$) ignoring the existence of LLODs.

3.0.1 |. Results

For immunogenicity comparison, Table 2 and Table 3 present the performance of the RCL.I and the RCP.I methods in predicting assay 1 response in vaccine arm 2 (sub-sample 2) (for s = 0) and the corresponding test comparing immunogenicity between the two vaccines (for s ∈ {0, 0.2, 0.5}). From Table 2, in all settings, RCP.I leads to unbiased estimates of the mean assay 1 response with bootstrap confidence interval (CI) coverage close to the nominal level, whether the $X_1^{*}$, $X_2^{*}$ observed at the LLOD is set as LLOD or LLOD/2. In contrast, the RCL.I method can lead to biased estimates of the mean assay 1 response and substantial bootstrap CI under-coverage in various scenarios where the linear conditional mean model in the paired study is not transportable to the new study (i.e. scenarios 3–7 as shown in Table 1). Violation of the transportable linear model assumption is commonly observed for scenarios with substantial left-censoring (i.e. scenarios 5–7) but can also happen for scenarios with very low rates of left-censoring (e.g. scenarios 3 and 4, where ρσ₁∕σ₂ deviates from unity, so the intercept of the linear model differs between old and new settings as shown in Table 1). The problem of RCL.I persists whether or not the assay value at the LLOD is replaced with LLOD/2: bias of the calibrated assay can be in the same (scenarios 3,4,6,7) or opposite direction (scenario 5) (Table 2). The bias in calibrated assay values using RCL.I also leads to inflated type-I error when comparing immunogenicity between the two vaccines in the new setting (Table 3). The inflation is apparent even in scenarios with very small calibration bias (e.g. scenarios 1 and 2) although with a small magnitude. In contrast, the test of immunogenicity based on the RCP.I method has well-controlled type-I error in all scenarios. Compared to the ideal situation where vaccine responses for both vaccines were measured using the same assay, RCP.I has a small decrease in power when the effect size is small (s = 0.2), but fairly comparable power for a larger effect size (s = 0.5).

TABLE 2.

Estimation of $E\left(X_1^{*}\right)$ based on $X_2^{*}$ measurement derived by RCL.I or RCP.I. The ‘ideal’ method is a reference method assuming $X_1^{*}$ is observable. Results in parentheses are based on replacing assay value measured at the LLOD with LLOD/2.

Scenario	μ 1	μ 2	σ 1	σ 2	ρ	${10}^{a_1}$	${10}^{a_2}$		Ideal	RCL.I	RCP.I
1	3.2	3.6	0.46	0.42	0.91	10	40	Mean	4.198 (4.200)	4.195 (4.195)	4.198 (4.198)
								SD	0.046 (0.046)	0.104 (0.104)	0.054 (0.054)
								Coverage	95.1 (95.1)	93.1 (93.1)	94.2 (94.2)
2	3.2	3.6	0.46	0.42	0.91	80	40	Mean	4.198 (4.198)	4.193 (4.203)	4.198 (4.198)
								SD	0.046 (0.045)	0.104 (0.109)	0.054 (0.054)
								Coverage	95.1 (95.1)	93.0 (93.1)	94.2 (94.2)
3	3.6	3.2	0.42	0.46	0.91	40	10	Mean	4.602 (4.602)	4.434 (4.434)	4.603 (4.603)
								SD	0.042 (0.042)	0.090 (0.090)	0.057 (0.057)
								Coverage	94.9 (94.9)	49.1 (49.1)	94.5 (94.5)
4	3.6	3.2	0.42	0.46	0.91	40	80	Mean	4.602 (4.602)	4.435 (4.426)	4.603 (4.603)
								SD	0.042 (0.042)	0.091 (0.096)	0.057 (0.057)
								Coverage	94.9 (94.9)	49.8 (47.2)	94.6 (94.6)
5	2.2	3.6	0.46	0.42	0.91	80	40	Mean	3.198 (3.196)	3.009 (3.445)	3.197 (3.194)
								SD	0.045 (0.046)	0.121 (0.155)	0.058 (0.060)
								Coverage	95.1 (95.3)	58.7 (64.2)	95.4 (95.6)
6	3.6	2.2	0.42	0.46	0.91	40	80	Mean	4.602 (4.602)	4.543 (4.197)	4.603 (4.603)
								SD	0.042 (0.042)	0.122 (0.088)	0.062 (0.062)
								Coverage	94.9 (94.9)	87.4 (0.3)	95.7 (95.7)
7	1.8	2.2	0.42	0.46	0.91	40	80	Mean	4.602 (4.602)	4.543 (4.197)	4.603 (4.603)
								SD	0.042 (0.042)	0.122 (0.088)	0.062 (0.062)
								Coverage	94.9 (94.9)	87.4 (0.3)	95.7 (95.7)

TABLE 3.

Type-I error (s = 0) and power for comparing immunogenicity based on RCL.I and RCP.I. The ‘ideal’ method is a reference method assuming $X_1^{*}$ is observable. Results in parentheses are based on replacing assay values measured at LLOD with LLOD/2.

Scenario	μ 1	μ 2	σ 1	σ 2	ρ	${10}^{a_1}$	${10}^{a_2}$		s = 0	s = 0.2	s = 0.5
1	3.2	3.6	0.46	0.42	0.91	10	40	Ideal	4.9 (4.9)	88.2 (88.2)	100 (100)
								RCL.I	6.9 (6.9)	46.1 (46.1)	98.5 (98.5)
								RCP.I	4.8 (4.8)	82.0 (82.0)	100 (100)
2	3.2	3.6	0.46	0.42	0.91	80	40	Ideal	4.9 (4.9)	88.2 (88.2)	100 (100)
								RCL.I	7.1 (7.1)	46.5 (43.9)	98.5 (96.0)
								RCP.I	4.8 (4.8)	82.0 (82.0)	100 (100)
3	3.6	3.2	0.42	0.46	0.91	40	10	Ideal	5.0 (5.0)	92.3 (92.3)	100 (100)
								RCL.I	42.4 (42.4)	95.2 (95.2)	100 (100)
								RCP.I	4.9 (4.9)	82.1 (82.1)	100 (100)
4	3.6	3.2	0.42	0.46	0.91	40	80	Ideal	5.0 (5.0)	92.3 (92.3)	100 (100)
								RCL.I	41.8 (44.1)	94.9 (95.3)	100 (100)
								RCP.I	4.9 (4.9)	82.0 (82.0)	100 (100)
5	2.2	3.6	0.46	0.42	0.91	80	40	Ideal	5.0 (5.0)	88.1 (88.1)	100 (100)
								RCL.I	37.6 (32.4)	90.0 (6.0)	99.9 (41.2)
								RCP.I	4.5 (4.0)	79.3 (78.5)	100 (100)
6	3.6	2.2	0.42	0.46	0.91	40	80	Ideal	5.0 (5.0)	92.3 (92.3)	100 (100)
								RCL.I	10.6 (98.0)	57.0 (100)	96.2 (100)
								RCP.I	3.7 (3.7)	69.1 (69.1)	94.2 (94.2)
7	1.8	2.2	0.42	0.46	0.91	40	80	Ideal	5.1 (4.9)	92.2 (92.3)	100 (100)
								RCL.I	53.1 (78.7)	96.1 (99.7)	100 (100)
								RCP.I	3.6 (3.3)	69.1 (68.0)	94.9 (94.9)

In CoR meta-analysis (Table 4), the naive estimator that ignores assay differences and measurement error can be severely biased in estimating the odds ratio association between the true underlying immune response and the risk of the clinical outcome, with severe under-coverage in bootstrap CI. Calibration methods accounting for measurement error with (RCP.C) or without (RCN.C) adjusting for the LLOD always improve over the naive method. The RCP.C method leads to unbiased odds ratio estimators with good bootstrap CI coverage; it achieves 80–90% efficiency compared to the ideal estimator that assumes the underlying true immune responses are available. In scenarios with low left-censoring rates, the performance of RCN.C is very similar to that of RCP.C (scenarios 1–4). When the level of censoring is noticeable, RCN.C ignoring the LLOD could lead to a biased odds ratio estimate with a substantial increase in variability compared to RCP.C (25% - 100% increase in scenarios 5–7), which can lead to under-coverage of bootstrap CIs when bias is severe (e.g. scenario 7).

TABLE 4.

Estimation of OR based on true U from everyone (Ideal), just using $X_1^{*}$ and $X_2^{*}$ as available (Naive), or based on RCN.C and RCP.C.

Scenario	μ 1	μ 2	σ 1	σ 2	ρ	${10}^{a_1}$	${10}^{a_2}$		Ideal	Naive	RCN.C	RCP.C
1	3.2	3.6	0.46	0.42	0.91	10	40	Mean	−0.504	−0.380	−0.506	−0.506
								SD	0.330	0.279	0.343	0.343
								Coverage	94.7	89.9	95.8	95.8
2	3.2	3.6	0.46	0.42	0.91	80	40	Mean	−0.504	−0.380	−0.507	−0.506
								SD	0.330	0.280	0.343	0.343
								Coverage	94.7	90.0	95.8	95.8
3	3.6	3.2	0.42	0.46	0.91	40	10	Mean	−0.509	−0.380	−0.514	−0.514
								SD	0.356	0.304	0.367	0.367
								Coverage	96.9	91.5	96.2	96.2
4	3.6	3.2	0.42	0.46	0.91	40	80	Mean	−0.509	−0.380	−0.515	−0.514
								SD	0.356	0.304	0.367	0.367
								Coverage	96.9	91.6	96.4	96.3
5	2.2	3.6	0.46	0.42	0.91	80	40	Mean	−0.492	−0.126	−0.547	−0.491
								SD	0.257	0.134	0.304	0.271
								Coverage	95.6	3.1	95.4	96.0
6	3.6	2.2	0.42	0.46	0.91	40	80	Mean	−0.509	−0.128	−0.573	−0.515
								SD	0.356	0.197	0.427	0.377
								Coverage	96.9	18.1	96.2	96.6
7	1.8	2.2	0.42	0.46	0.91	40	80	Mean	−0.495	−0.422	−0.676	−0.490
								SD	0.226	0.256	0.353	0.249
								Coverage	95.0	90.6	94.2	96.1

4 |. SARS-COV-2 NEUTRALIZATION ASSAY DATA EXAMPLE

In this section we apply the proposed calibration methods to analyze SARS-CoV-2 pseudovirus neutralization assay data from COVID-19 vaccine trials. The nAb responses have been of substantial interest in COVID-19 vaccine research as a potential immune correlate for predicting a vaccine’s protective effect^22,2. In each of the five US Government-sponsored phase 3 COVID trials, nAb titers are measured by one of the two different pseudovirus-based SARS-CoV-2 neutralization assays performed at Duke University and Monogram Bioscience, respectively⁵. It is important to assess calibration between the two assays in order to facilitate the research that integrates SARS-CoV-2 nAb data across trials. Here we compare the nAb titer readouts from the two assays, develop calibration models using a set of paired COVID-19 convalesent serum samples, and assess performance of the calibration using samples from Moderna mRNA-1273 vaccine recipients in a phase 1 trial^6,7.

Our training data for developing the calibration model consists of 50% inhibitory dilution (ID50) and 80% inhibitory dilution (ID80) nAb titers measured by both Duke and Monogram from 248 COVID-19 convalescent serum samples. Our test data consists of ID50 and ID80 nAb titers measured by both Duke and Monogram in 30 mRNA-1273 vaccine recipient serum samples collected at Day 57 (4 weeks post second vaccination). The LLOD of the Duke assay is 20 and 10 in the training and test set respectively, while the LLOD of the Monogram assay is 40 in both datasets. In the training set, the left censoring rate is 0% and 0.4% for ID50 measured by the Duke and Monogram assays, respectively, and 16.5% and 0.4% for ID80. In the test set, the left censoring rate is 0% for ID50 and 3.3% for ID80 for both assays. As shown in Figure 1, both ID50 and ID80 titers are higher for the Monogram assay compared to the Duke assay, in both datasets. The Wilcoxon signed-rank test for comparing paired assay samples has a p-value< 0.001 for ID50 and for ID80, in both the training or test data. The concordance correlation coefficient (CCC)²³ between the two assay measures on the log10 transformed scale is 0.602 and 0.440 for ID50 and ID80, respectively, in the training data and 0.602 and 0.378 in the test data.

We applied the RCL.I and RCP.I methods to the training data to develop calibration models for predicting Monogram assay nAb measurements based on Duke nAb measurements, for log10 transformed ID50 and ID80 titer separately. The estimated models were applied to the test data. Nonparametric bootstrap resampling of individuals in the test data was used to construct percentile confidence intervals for CCC estimation. For both approaches, the calibrated Day 57 vaccine recipient nAb titers obtained by the Duke assay are much closer to the observed Monogram assay titers compared to the raw Duke assay titers (Figure 1 (c)(d)): CCC between predicted and observed Monogram ID50 is 0.812 (95% bootstrap CI: 0.549, 0.942) and 0.901 (95% bootstrap CI: 0.618, 0.964) using RCL.I and RCP.I respectively; CCC between predicted and observed Monogram ID80 is 0.856 (95% bootstrap CI: 0.600, 0.917) and 0.918 (95% bootstrap CI: 0.731, 0.949) using RCL.I and RCP.I respectively.

5 |. CONCLUDING REMARKS

We developed new methodology frameworks for harmonizing different lab assays in immunogenicity comparison and in CoR analysis. Our modeling approaches utilize paired study samples to reconcile heterogeneity between assays with respect to the limit of detection and measurement error. Through maximization of likelihood based on left-censored bivariate-normal measures, our proposed methods can achieve appreciable improvement over existing approaches with respect to estimation and testing. In particular, for immunogenicity comparison, our proposed method leads to unbiased assay mean estimation and valid test for treatment group comparision while existing method assuming constant assay calibration model across studies^19,5 can result in biased estimation and test with inflated type-I error; for the CoR analysis, in the presence of noticeable left-censoring rate, our proposed method can drastically outperform the existing method that ignores left-censoring¹⁴, in terms of reduced bias and improved precision.

Of note, the bridging/transportability assumption between the paired study and the new independent study can be made flexibly in practice, to allow for evolution of lab techniques over time in terms of changing LOD and measurement error. While we considered two assays in this paper with lower LODs of major interest, motivated by the COVID-19 nAb assay application, the methods can be readily extended to handle an upper LOD and allow for data from multiple assays. The CoR meta-analysis framework can be readily generalized to involve data from multiple efficacy trials and to allow for adjustment of additional covariates that characterize the difference across trial populations, by including covariates in a logistic regression model and in a calibration model for the underlying true immune response. Another potential future direction is to develop an alternative type of CoR meta-analytic approach that models the relationship between an observed immune response using a particular assay, instead of the underlying true immune response, and the risk of the clinical outcome. The proposed methods will play an important role in the near future for integrating immunogenicity data from multiple COVID-19 vaccine trials towards further vaccine development. They will also be useful to clinical and biological researchers as a tool to facilitate collaboration between multiple studies and labs/institutes in treatment comparison and disease-association assessment.

In practice, the appropriate methods to deal with below detection limit data need to be specific to different lab assays. For example, as one of the reviewer pointed out, luminex assay could have quantitative value below detection limit based on a calibration curve and in other applications one can use original optical values instead of absolute concentrations to avoid the censoring issue. For the ID50 and ID80 assays nAb titer (the inhibitory dilutions at which 50% (or 80%) neutralization is attained) involved in our motivating application, inbibitory level is measured at serial dilutions of antibodies, and the LLOD corresponds to the lowest dilution level that 50% or 80% neutralization is obtained; unlike luminex assay there is no quantitative value associated with values at LLOD for ID50 and ID80. In the presence of measurement at LLOD, replacing below limit data by the LLOD or half the LLOD during calibration (e.g. in RCL.I) is sub-optimal since it messes up variance estimates (which are then systematically too low) if there are more than a few below limit observations. In contrast, our proposed method is a likelihood based method explicitly modeling left-censoring due to LLOD and so the model parameters for the pre-censoring bivariate-normal distribution can be consistently estimated in the presence of LLOD.

The method in this paper builds upon an approximate normal distribution for pre-censoring assay value measured with error, which is deemed a reasonable approximation for log-transformed nAb titer. Alternative error models have been proposed by other researchers for dealing with different lab assays, An interesting future research direction is to incorporate alternative error modeling approach into the assay comparison and combination problem. For example, a useful two-component model for measurement error modeling was proposed by²⁴. This method assumes availability of quantitative values associated with below detection limit data and availability of true concentration data together with observed concentration for model fitting and is not directly applicable to our application. However it will be an interesting research question to develop and evaluate new methodology for assay comparison and combination under the framework of two-component measurement error structure, in problem settings with suitable data for measurement error modeling such as replicate data and quantitative values associated with data below the limit of detection.

DATA AVAILABILITY STATEMENT

All source codes can be requested from the corresponding author.