Calibration of Two Validated SARS-CoV-2 Pseudovirus Neutralization Assays for COVID-19 Vaccine Evaluation

Abstract

Vaccine-induced neutralizing antibodies (nAbs) are key biomarkers considered to be associated with vaccine efficacy. In United States Government-sponsored phase 3 efficacy trials of COVID-19 vaccines, nAbs are measured by two different validated pseudovirus-based SARS-CoV-2 neutralization assays, with each trial using one of the two assays. Here we describe and compare the nAb titers obtained in the two assays. We observe that one assay consistently yielded higher nAb titers than the other when both assays were performed on the World Health Organization’s anti-SARS-CoV-2 immunoglobulin International Standard, COVID-19 convalescent sera, and mRNA-1273 vaccinee sera. To overcome the challenge this difference in readout poses in comparing/combining data from the two assays, we evaluate three calibration approaches and show that readouts from the two assays can be calibrated to a common scale. These results may aid decision-making based on data from these assays for the evaluation and licensure of new or adapted COVID-19 vaccines.

Introduction

Multiple vaccines have shown short-term efficacy against coronavirus disease-19 (COVID-19) in phase 3 randomized placebo-controlled trials^1–7; one has been fully approved⁸ and three have been issued an Emergency Use Authorization⁹ by the US Food and Drug Administration (FDA), and six an Emergency Use Listing from the World Health Organization (WHO).¹⁰ To expediently meet the worldwide demand for sufficient doses (including booster doses) of safe and efficacious vaccines, and for the licensure of new or modified vaccines, the FDA and the WHO have recommended that immunobridging trials may be used to assess the effectiveness of booster vaccines, vaccines in new populations (e.g., pediatric) or against emerging SARS-CoV-2 variants,^11,12 thus sidestepping the need for additional large-scale placebo-controlled randomized trials (which are becoming increasingly difficult to conduct, especially in high-income countries¹³) or active-control randomized trials.¹⁴ In such immunobridging trials, vaccine-induced immune responses of an established immune marker surrogate endpoint^15–17 are typically assessed to ensure that levels deemed sufficient for clinical efficacy are achieved. Neutralizing antibodies (nAbs) are correlates of protection for many licensed vaccines,¹⁷ and an accumulating body of evidence^18–26 supports their evaluation in immunobridging trials for COVID-19 vaccines.

In each of the five US Government (USG)-sponsored phase 3 trials, nAb responses are assessed by one of two different pseudovirus-based SARS-CoV-2 neutralization assays, one performed at Duke University and one at Monogram Biosciences. Each pseudovirus neutralization assay has been rigorously validated in accordance with the ICH/FDA guideline^27,28 and these assays are planned for use in additional efficacy trials or immunobridging trials of COVID-19 vaccine candidates for regulatory approvals. If the Duke and Monogram assays do not render comparable readouts or are not calibrated (cross-validated) to a common scale, nAb response data from these assays may misguide regulatory decisions on these COVID-19 vaccine candidates.

To facilitate the harmonization of SARS-CoV-2 nAb data across multiple laboratories and assay types, the WHO developed an International Standard (WHO IS) for anti-SARS-CoV-2 immunoglobulin consisting of a pool of 11 COVID-19 convalescent plasma samples and provided as lyophilized material that, after reconstitution, was assigned a unitage of 1,000 international units per mL (IU/mL) for neutralization assays.²⁹ Here we describe the Duke and Monogram nAb assays, and compare titer readouts between the two assays using a diverse set of samples that include the WHO IS, COVID-19 convalescent serum samples, and serum samples from Moderna mRNA-1273 vaccine recipients. Because the ultimate goal is to ensure comparability of nAb readouts in vaccine recipients, we also present and compare statistical approaches for calibrating readouts from the two assays to support future vaccine evaluation and development.

Results

Table 1 summarizes and compares the Duke and Monogram assays. Similarities of both assays include utilizing lentiviral vector that 1) is pseudotyped with full-length G614 Spike protein (the SARS-CoV-2 surface protein responsible for virus entry via interaction with the human ACE2 receptor^30,31) and 2) contains a firefly luciferase (Luc) reporter gene. Thus, in both assays a quantitative luminescence readout is directly proportional to the amount of pseudovirus entry into target cells. In both assays, neutralizing antibodies in the assayed sample inhibit pseudovirus entry and reduce luminescence signal in a dose-dependent manner, providing the basis for quantifying the neutralizing activity of the sample. Both assays are established to measure ID50 and ID80 neutralizing titers and are fully validated in terms of all required assay performance parameters²⁷ for convalescent serum and partially validated in terms of specificity, linearity, precision and limits of quantitation of the assay for vaccine incurred samples, in order to extend validation for the purpose of testing clinical trial specimens.

Table 1.

Overview comparison of the Duke vs. Monogram assays.

Component	Duke	Monogram
Purpose	Measures ability of antibody in a serum/plasma/monoclonal or purified antibody sample to neutralize SARS-CoV-2 pseudovirus
Principal	SARS-CoV-2 pseudotyped virus expresses firefly luciferase upon infecting HEK 293T cells, with relative light units (RLU) of luminescence directly proportional to the infectivity of the virus inoculum Neutralizing antibodies inhibit pseudovirus infection of cells, resulting in lower RLUs Serial dilutions of antibodies are used to construct a dose-response curve to quantify potency for each sample, recorded as 50% inhibitory dose (ID50) and 80% inhibitory dose (ID80)
Format and capacity	Performed in 96-well plate format and includes controls
	Up to 5 samples in duplicates per plate	Up to 6 samples per plate
Virus	Lentivirus:
	Pseudotyped with SARS-CoV-2 G614 full-length Spike protein and packaged with TMPRSS2 serine protease and Phr’ CMV Luc luciferase reporter genes	Pseudotyped with SARS-CoV-2 G614 full-length Spike protein and packaged with HIV genomic vector, pRTVl.FlucP.CNDO-ΔU3, containing a luciferase reporter gene in place of HIV envelope
Cell Line	HEK 293T cell line:
	Stably overexpressing the human ACE2 cell surface receptor protein (defined as 293T/ACE2), Cells may be used up to passage 46 or 4 months in culture	Transiently transfected to express human ACE2 cell surface receptor protein along with a serine airway protease, TMPRSS2, for processing the S protein. Cells may be used up to passage 30 or as long as data maintains high quality
Controls	Assay Plate Positive Control – anti-RBD of the Spike proteins mAbs Assay Run Positive Controls – convalescent serum samples with high, medium, and low titers Assay Run Negative Control – normal human serum (NHS) Positive control with known titer included on each plate, negative control included at least on one plate in each assay run	SARS-CoV-2 Positive Control (CPC#) - pool of COVID-19 convalescent plasma with positive SARS-CoV-2 neutralization titer. Included on every test plate SARS-CoV-2 Negative Control (CNC#) – pool of COVID-19 convalescent plasma with negative SARS-CoV-2 neutralization titer. Included on every test plate Daily Positive Controls – convalescent serum samples with high, medium, and low titers tested once per day Specificity Control Pseudovirus – a lentiviral pseudovirus with Influenza H10N3 envelope tested with every sample and control to detect non-specific inhibition
Equipment	96-well Poly-L-Lysine or regular plates GloMax® Navigator Microplate Luminometer (Promega) Manual liquid handling	96 well opaque Luminometer plates Luminoskan Luminometers Semi-automated liquid handling
Assay Range	1:10 to 1:781,250 or 1:20 to 1:1,562,500	1:40 to 1:787,320
Sample types	Heat-inactivated (56°C, 30 minutes) serum (preferred), plasma, convalescent and vaccine incurred samples plus monoclonal antibodies and purified antibodies	Heat-inactivated (56°C, 60 minutes) serum, plasma, convalescent and vaccine incurred samples plus monoclonal and purified antibodies
Basic steps	Each sample is diluted serially 5-fold in duplicate for a total of 8 dilutions starting at a 1:10 or 1:20 dilution (dilution after adding virus but before adding cells). Virus is added to the plate (except cell control wells) and incubated for 45–90 minutes in humidified 37°C and 5% CO2 incubator 293T/ACE2 cells are harvested and 100 μl added to the plate at a density of 10,000 cells/well Plates are incubated for 71 – 73 hours in humidified 37°C and 5% CO2 incubator Add Bright-Glo luciferase reagent to lysed cells and luminescence measured with a luminometer	Each sample is diluted initially 1:20 and then diluted serially 3-fold to yield a total of 10 concentrations. The starting dilution of 1:20 becomes the final, reported as 1:40 after addition of the virus. Virus is added to the plate and incubated for 1–2 hours in humidified 37°C and 7% CO2 incubator Transfected 293/ACE2/TMPRSS2 cells are harvested and diluted to 10,000 cells/well and added to the plates Plates are incubated for 3 days in humidified 37°C and 7% CO2 incubator Steady Glo luciferase reagent is added to plate and luminescence measured with a luminometer
Calculations	Percent neutralization calculated as difference between RLUs of virus- and cell-only wells and test sample wells:
	$\begin{array}{l}\%\mathit{\text{Neutralization}}\text{=}\\ \text{=100\%}\times \left(1-\frac{Avg.RLU\left(\mathit{\text{Virus}}+\mathit{\text{Sample}}+\mathit{\text{Cells}}\right)}{Avg.RLU\left(\mathit{\text{Virus}}+\mathit{\text{Diluent}}+\mathit{\text{Cells}}\right)}\right)\end{array}$ Average RLU of cell control wells (without virus) are subtracted from RLUs of all other wells on a plate	$\begin{array}{l}\%\mathit{\text{Neutralization}}\text{=}\\ \text{=100\%}\times \left(1-\frac{RLU\left(\mathit{\text{Virus}}+\mathit{\text{Sample}}+\mathit{\text{Cells}}\right)}{Avg.RLU\left(\mathit{\text{Virus}}+\mathit{\text{Diluent}}+\mathit{\text{Cells}}\right)}\right)\end{array}$
	ID50 and ID80 neutralizing antibody titers are expressed as the reciprocal of the sample dilution required to reduce RLU by 50% and 80%, respectively
Acceptance Criteria/Plate level	Positive Controls within acceptance range for ID50 or ID80 titers Signal of virus-only wells is in range of RLU >48,000 to < 1,000,000 %CV ≤35% for virus only control wells	Positive Control, CPC#, titer falls within expected range for the assay (+/−3 SD of the mean) and within 30% of all CPC#s titers generated for the day RLUs fall between ~10,000 and ~500,000
Acceptance Criteria/Sample level	Relative Percent Difference ≤ 30% between duplicate wells for each sample with ≥ 40% neutralization Neutralization curves must cross 50% neutralization 0–1 time	Percent inhibition values of serial dilutions must follow a logical progression, e.g. the 1:40 serum dilution should exhibit equal or more inhibition of viral infection than the 1:120 serum dilution. Once the data points decrease below maximum inhibition and are on the linear part of the curve, each succeeding data point (higher dilution) should be less inhibiting than the preceding point (lower dilution). Graphs are rejected when antibody inhibition profiles (curves) have all or most (> 80%) of the points not falling on or within 20% of the extrapolated linear portion of the curve.
Acceptance Criteria/each Assay Run	ID50 or ID80 titers for Assay Run Positive and Negative Controls must be within acceptance range	Positive Control, CPC#, titer falls within expected range for the assay (+/−3 SD of the mean) and within 30% of all CPC#s titers generated for the day
Validation Status	Full validation for convalescent samples Partial validation for vaccine incurred samples

Differences in pseudovirus preparation, cell lines, plate layouts, and dilution schemes are also noted in Table 1. Among the notable differences is the way that Transmembrane Protein Serine 2 (TMPRSS2) is introduced into the assay. SARS-CoV-2 fuses with host cells by binding to the ACE2 receptor; the host cell TMPRSS2 protease cleaves the Spike protein at the S1/S2 sites to facilitate fusion.³² In the Duke assay, TMPRSS2 is expressed during pseudovirus production, whereas in the Monogram assay it is expressed on the surface of HEK 293T/ACE2 target cells. This difference can impact relative neutralization titers measured in these assays, with higher titers seen in HEK 293T/ACE2/TMPRSS2 cells (Montefiori, unpublished).

For assay comparison and establishment of the calibration algorithms, nAbs were measured in the first WHO IS for anti-SARS-CoV-2 immunoglobulin (20/136), as well as in 248 convalescent serum samples by both the Duke and Monogram labs following each assay format. For demonstration of the performance of the proposed calibration methods, nAbs were measured by both labs in 90 mRNA-1273 vaccine recipient serum samples [30 participants at three time points each: Day 1 (baseline), Day 29 (4 weeks post first vaccination), and Day 57 (4 weeks post second vaccination)].

Among the 90 samples from vaccine recipients, all Day 1 (pre-vaccination) samples had non-detectable nAbs by both labs, indicating a high level of specificity of the assays. Moderate nAb titers were detected after the first vaccination at Day 29 among almost all of the participants for ID50 (30 out of 30, 100% by Duke, 27 out of 30, 90% by Monogram) and the majority of participants for ID80 (80% by Duke, 70% by Monogram). nAb titers at Day 57 were detected among all participants who received the second dose (100% for ID50 and ID80 by both labs), and Day 57 titers were on average more than 20-fold higher than Day 29 nAb responses in both assays (Figure 1, Table 2). Specifically, geometric mean ID50 titers (standard deviation of log_e-transformed titers, SD) were 65.7 (0.8) at Day 29 and 1463.5 (1.1) at Day 57 as measured by the Duke lab and 178.3 (1.0) at Day 29 and 4120.7 (1.0) at Day 57 as measured by the Monogram lab. Geometric mean ID80 titers were 16.2 (0.8) at Day 29, and 330.5 (0.9) at Day 57 as measured by the Duke lab; 60.9 (0.8) at Day 29 and 1522.5 (1.0) at Day 57 as measured by the Monogram lab. Moderate correlations were seen between the Day 29 and Day 57 nAb titers (Figure S1). Day 29 and Day 57 nAb titers were treated separately in the calibration process, in order to cover different dynamic ranges of possible assay readouts.

Figure 1: Distributions of ID50 (Panel A) and ID80 (Panel B) titers measured by the Duke and Monogram labs of convalescent patient samples, vaccine recipient samples, and the WHO International Standard sample.

For the convalescent samples, the assay lower limit of detection (LLoD) is 1:20 for Duke and 1:40 for Monogram. For the vaccine samples collected at Day 29 and Day 57, 4 weeks post the first and the second Moderna doses, respectively, and the WHO IS sample, the assay LLoD is 1:10 for Duke and 1:40 for Monogram. Circle-shape points indicate titers > LLoD; triangle-shape points indicate titers ≤ LLoD.

Table 2:

Summary of ID50 and ID80 titers from each lab and their fold-differences for convalescent patient sera, Day 29 vaccinee sera, Day 57 vaccinee sera, and the WHO International Standard sample.

		Convalescent (n=248)	Vaccine Day 29 (n=30)	Vaccine Day 57 (n=30)	Vaccine Day 57 vs. Day 29	WHO International Standard
ID50
Duke	Mean (SD)*	607.3 (1.6)	65.7 (0.8)	1463.5 (1.1)	22.3 (1.1)	3046.7 (0.8)
	Median	581.0	72.5	1547.0	22.4	2421.5
	Range	23.0 – 32362.0	13.0 – 456.0	15.0 – 4714.0	0.5 – 158.4	957.0 – 11101.0
	Response Rate**	99%	100%	100%	NA	100%
Monogram	Mean (SD)*	1692.0 (1.4)	178.3 (1.0)	4120.7 (1.0)	23.1 (1.1)	14997.5 (0.2)
	Median	1604.0	219.5	5274.0	23.5	16133.0
	Range	72.0 – 100563.0	20.0 – 1222.0	62.0 – 16280.0	0.4 – 163.2	10044.0 – 21785.0
	Response Rate**	100%	90%	100%	NA	100%
Monogram vs. Duke fold difference	Mean (SD)*	2.8 (0.7)	2.7 (0.4)	2.8 (0.4)		4.9 (NA)
	Median	3.0	2.8	2.7	NA	6.7
	Range	0.3 – 26.0	1.0 – 7.1	1.2 – 6.3		2.0 – 10.5
ID80
Duke	Mean (SD)*	103.7 (1.6)	16.2 (0.8)	330.5 (0.9)	20.3 (0.9)	566.7 (0.6)
	Median	94.5	19.5	421.5	22.7	489.0
	Range	10.0 – 5078.0	5.0 – 91.0	5.0 – 1255.0	1.0 – 112.0	250.0 – 1391.0
	Response Rate**	83%	77%	97%	NA	100%
Monogram	Mean (SD)*	536.8 (1.4)	60.9 (0.8)	1522.5 (1.0)	25.0 (1.1)	4259.7 (0.2)
	Median	504.5	82.0	1758.5	27.5	4062.5
	Range	20.0 – 32557.0	20.0 – 393.0	20.0 – 5984.0	0.5 – 130.5	2950.0 – 6892.0
	Response Rate**	98%	70%	97%	NA	100%
Monogram vs. Duke fold difference	Mean (SD)*	5.2 (0.7)	3.7 (0.5)	4.6 (0.2)		7.5 (NA)
	Median	5.1	4.0	4.5	NA	8.3
	Range	0.4 – 75.2	1.3 – 15.0	2.9 – 7.8		5.0 – 11.8

^*geometric mean of ID50 or ID80 titers and standard deviation of the log_e-transformed ID50 or ID80 titers.

^**proportion of samples with an ID50 or ID80 titer above the assay LLoD.

Prior to calibration, we observed that ID50 and ID80 titer readouts from the Monogram lab were consistently higher than those from the Duke lab measured in convalescent sera, vaccine recipient sera, and the WHO IS sample (Figure 1, Table 2). Specifically, the fold difference of individual-level ID50 titers from Monogram vs. Duke across individuals had a geometric mean (SD) of 2.8 (0.7) in convalescent samples, 2.7 (0.4) and 2.8 (0.4) in Day 29 and Day 57 vaccine recipient samples, respectively, and the fold difference of the geometric (arithmetic) mean ID50 titers across replicates of the WHO IS sample was 4.9 (3.7); the fold difference of individual-level ID80 titers Monogram vs. Duke across individuals had a geometric mean (SD) of 5.2 (0.7) in convalescent samples, 3.7 (0.5) and 4.6 (0.2) in Day 29 and Day 57 vaccine recipient samples, respectively, and the fold difference of the geometric (arithmetic) mean ID80 titers across replicates of the WHO IS sample was 7.5 (6.6). These differences in readouts between the two assays are an example of the need for calibration of the responses to the same scale prior to cross-assay comparisons or cross-assay merging of results. The fact that fold differences between the two assays are largely consistent across convalescent samples and vaccine recipient samples suggests that the differences between the assays are not specific to naturally acquired or vaccine-elicited nAbs. In addition, the range of readouts in convalescent samples generally covers that in vaccine recipient samples; this supports the use of those convalescent samples to develop the algorithms for the calibration of titers in the vaccine recipient samples. On the other hand, the WHO IS, which is a single sample comprised of a pool of convalescent plasma samples, does not cover the entire range of titers and hence resulted in an expected narrower range of replicate values.

We evaluated three different approaches for the calibration of vaccine-induced neutralizing antibody response titers measured by the Duke and Monogram labs. Approach 1 uses tiers of the WHO IS sample to `standardize’ nAb titers measured by each lab to a common international unit (IU/ml),³³ whereas Approaches 2 and 3, via two different statistical algorithms (details in Methods), use titers of an independent pool of clinical (e.g. convalescent) samples measured by both labs to calibrate titers measured by each lab to a common scale still in titer units.

For Approach 1, calibration factors were calculated using the arithmetic mean, geometric mean or median titers of the WHO IS sample measured by both labs (Table S1). Using the arithmetic mean-based calibration factor, calibrated ID50 (cID50) and ID80 (cID80) readouts of the Day 29 and Day 57 vaccine-induced responses from the two labs rendered the highest agreement, compared to the geometric- and median-based factors, with a concordance correlation coefficient (CCC) of 0.75 (95% CI: 0.60, 0.85) for cID50, and 0.72 (0.56, 0.82) for cID80 (Figure S2). Therefore, Approach 1 using the arithmetic mean-based calibration factor is used to compare with Approaches 2 and 3 (Figure 2).

Figure 2: Scatterplots of calibrated neutralizing antibody readouts demonstrating performance of the three calibration approaches using vaccine recipient samples collected at Day 29 (turquoise) and Day 57 (orange), 4 weeks post the first and the second mRNA-1273 vaccine doses, respectively.

The arithmetic mean-based calibration factor was used in Approach 1. The concordance correlation coefficient (CCC) and 95% confidence intervals indicate the level of agreement between the x- and y-axis values.

• cID50, cID80: ID50, ID80 titers calibrated to the WHO International Standard, expressed in International Units per ml (IU/ml) (Approach 1).
• cBIVN-ID50, cBIVN-ID80: ID50, ID80 titers from each lab calibrated to a common scale using an independent pool of clinical samples, assuming a bivariate normal distribution for the readouts from the two labs (Approach 2).
• cREG-ID50, cREG-ID80: ID50, ID80 titers from the non-reference lab calibrated to the reference lab using an independent pool of clinical samples, based on regressing titers from the reference lab on the non-reference lab (Approach 3).

For Approach 2, readouts from the two labs were calibrated to the same scale (with the Monogram readouts set to the reference) via an algorithm developed using titers of the convalescent samples. This approach assumes a bivariate normal (BIVN) distribution of the two assay readouts and incorporates the reported measurement error of each assay. Calibrated vaccine-induced neutralizing antibody ID50 and ID80 titers (cBIVN-ID50 and cBIVN-ID80, respectively) using Approach 2 achieved the highest agreement among the three approaches with a CCC of 0.87 (0.85, 0.87) for cBIVN-ID50 titers, and 0.85 (0.81, 0.88) for cBIVN-ID80 titers.

Similar to Approach 2, readouts from the Duke lab in Approach 3 were calibrated to the scale of the Monogram lab via an algorithm also developed using titers of the convalescent samples. Unlike Approach 2, Approach 3 is based on regressing (REG) the log-transformed titers of the Monogram assay (reference lab) on the Duke assay without incorporating assay measurement errors (Figure S3). Calibrated ID50 and ID80 titers (cREG-ID50 and cREG-ID80) from the Duke lab and observed readouts from the Monogram lab also showed reasonable agreement with a CCC of 0.77 (0.71, 0.82) for ID50 titers, and 0.68 (0.61, 0.75) for ID80 titers, comparable to the performance of Approach 1.

Discussion

Neutralizing antibody data obtained from multiple trials of different vaccines can support the establishment of nAbs as a correlate of protection, and if demonstrated, may facilitate approval of future vaccines through immunobridging. It is not often possible for samples from different vaccine trials to be evaluated in a single assay and laboratory. In the case of SARS-CoV-2, nAb data were generated at one of two independent laboratories for each of the five USG-sponsored vaccine efficacy trials; this approach is planned for future trials, as well. Because assay readouts in vaccine recipient sera from these two laboratories are on average 3–5 fold different, our study addresses this imperative need via calibrating data from the two assays to the same scale.

Our first main conclusion is that the WHO IS sample (Approach 1) demonstrated satisfactory performance comparable to that of the panels of convalescent sera in calibrating COVID-19 vaccine-induced nAb titers measured by these laboratories. This finding has important implications for future development and evaluation of COVID-19 vaccines because the WHO IS is more widely available and hence provides a straightforward way to compare and combine data across different platforms, and/or different developers and laboratories. Another main conclusion is that any of the three different calibration approaches can be used to bring readouts from different assays to the same scale for further analyses across studies, but that, due to the distinct advantages and disadvantages of each approach (summarized in Table S2), the specific approach that will be best suited for a given analysis depends on the availability of data to establish the calibration algorithms. For our motivating application – cross-protocol immunogenicity and correlates analyses of the 5 USG COVID-19 trials, where all nAb data are obtained by either the Duke or the Monogram assay – Approach 2 is recommended. However, for analyses including nAb data not measured by either the Duke or Monogram assay, then Approach 1 (using International Units) is likely the most feasible option, because not all labs have access to the convalescent samples that are necessary for Approach 2. We discuss other implementation considerations for the three Approaches below.

Among the three approaches, Approach 1 is the easiest to implement and likely requires the fewest resources as long as data on the WHO IS sample are available from the lab from a sufficient number of vials (e.g., > 20–30) and replicates. One requirement is to choose an appropriate summary statistic to calculate the calibration factor. In the dataset presented in this study, the calibration factor based on the arithmetic mean of ID50 or ID80 titers rendered the best performance. This is largely due to the fact that the corresponding WHO standard-based calibration factor for each lab translates to conversion factors, i.e., the ratio of readouts in international units of 3.7 for ID50 and 6.6 for ID80 titers between Monogram vs. Duke labs, that are in line with the fold-differences observed in the convalescent patient samples and in the vaccine recipient samples. This explains why this simple multiplicative-factor approach rendered reasonable calibration performance. Meanwhile it also highlights that external neutralizing antibody titers from convalescent or vaccine sera generated by both labs on a common set of samples can inform decisions on which summary statistic to use in Approach 1 by evaluating the conversation factor against the additional data sources. Approach 1 is currently being used in the analyses of nAb data generated from two of the USG trials, one set of data from the Duke lab and the other by the Monogram lab. These analyses provide an important validation that assay-specific differences are minimized in the resulting calibrated data based on Approach 1 across the two USG trials, so that potential differences or similarity between the vaccine candidates in terms of their immune responses are not masked by differences between the two assays.

For both Approaches 2 and 3, a common pool of convalescent or vaccine samples will need to be assayed by both labs in order to establish the calibration algorithm, where Approach 2 requires readouts from both labs to be calibrated to a common scale, and Approach 3 calibrates the readouts from one lab to the other reference lab. We find that Approach 2 rendered the best calibration performance, followed by comparable performance between Approaches 1 and 3 for both ID50 and ID80 titers. These results suggest that precision in calibration can be improved by accounting for assay measurement error and the differences between the two assays across a range of response magnitudes (i.e., as done in Approach 2). In our case, measurement errors were characterized in assay validation experiments using both convalescent samples and vaccine samples by both labs, and assay differences in magnitude were characterized using convalescent samples with dynamic ranges of titers that were comparable to the ranges of titers of vaccine trial samples requiring calibration. Therefore, Approach 2 is recommended when paired-sample data are available from the two assays/labs (as is the case for the analyses of assay data from the 5 USG COVID-19 vaccine trials, where both the Duke and Monogram labs assayed a common set of convalescent samples for the calibration of their readouts in vaccine samples).

There are several limitations of this study. First, for evaluating the performance of the calibration approaches, we only considered a limited number of samples from trials of the mRNA-1273 vaccine. This potentially limits our ability to generalize findings from the calibration study to other COVID-19 vaccine platforms. We plan to assay a subset of samples from phase 3 trials of other COVID-19 vaccines to further validate the calibration approaches when those data become available. Second, to develop the calibration algorithms, both labs must have their assays performed on a common pool of independent samples that are expected to cover a similar dynamic range as the readouts requiring calibration. Access to such a common pool may be challenging for many labs. In that case, Approach 1 might be the most feasible option as long as the WHO IS sample is available in sufficient quantity. Third, for all three approaches, we focused the calibration only for titers above the lower limit of detection (LLoD) of the reference assays. Methods that account for calibration of responses below the LLoD, especially in the context of two assays having different LLoDs, are currently under development. Fourth, this study only concerns data from two specific labs measuring nAbs. Further validation is needed to investigate the performance of the three approaches applied to the calibration of other assays and/or immune responses. Although the described calibration approaches are only applied to the calibration of assays measuring anti-SARS-CoV-2 nAb responses induced by candidate vaccines, a more general use is amenable when assay data from multiple sources are of interest for comparison or combination. Fifth, the two nAb assays used in this study were both pseudovirus-based using lentiviral packaging. Future studies could also compare nAb responses obtained by the pseudovirus assay to those obtained by other SARS-CoV-2 neutralization platforms, e.g., pseudovirus-based neutralization using other viral vectors, or neutralization of live SARS-CoV-2. Lastly, this study focused on a single variant of SARS-CoV-2 (G614) that dominated the pandemic during the earliest phase 3 trials (e.g., Pfizer and Moderna trials), but has since been replaced by other variants, most recently the Delta variant.³⁴ Although the G614 form of Spike remains an important comparator, additional efforts are needed to establish new standards for current and future SARS-CoV-2 variants of concern.

Our study has a number of strengths. First, three distinct statistical approaches for calibration, each with different data requirements and/or data assumptions, are discussed to provide researchers versatile tools to help guide decisions for which approach may be optimal for calibrating a given data set. Second, each assay was formally validated for the assessment of COVID-19 vaccine-induced or naturally acquired neutralizing antibody immune responses, according to current regulatory guideline and standards. Third, to our knowledge, this is the first study that uses convalescent sera and the WHO IS sample to establish calibration algorithms and evaluate the performance of different calibration approaches for samples from COVID-19 vaccine trials. When more data become available, each approach could be readily updated to improve calibration accuracy and precision. Lastly, although the described calibration approaches were only illustrated for the analysis of data from two labs, all three approaches can be easily generalized to the calibration of data from more than two assays or labs.

In summary, correlates and immuno-bridging studies that could accelerate the development and evaluation of new and modified vaccines are at the heart of COVID-19 prevention efforts. We presented calibration approaches to support valid comparisons and combination of key immune response data from these studies to minimize the influence of assay-specific differences and optimize the characterization of future vaccine candidates.

Online Methods

Pseudovirus neutralization assays

Table 1 provides a comparison of the assays’ components and major steps. Further details are provided for each assay below.

Duke

Neutralization was measured in a formally validated assay that utilized lentiviral particles pseudotyped with full-length SARS-CoV-2 Spike protein and containing a firefly luciferase (Luc) reporter gene for quantitative measurements of infection by relative luminescence units (RLU). The assay was performed in 293T/ACE2.MF provided by Drs. Michael Farzan and Huihui Mu. Pseudoviruses were prepared and titrated for infectivity essentially as described previously.³⁵ Briefly, an expression plasmid encoding codon-optimized full-length spike of the Wuhan-1 strain (VRC7480) was provided by Drs. Barney Graham and Kizzmekia Corbett at the Vaccine Research Center, National Institutes of Health (USA). Mutations, including the D614G mutation, were introduced into VRC7480 by site-directed mutagenesis³⁵ using the QuikChange Lightning Site-Directed Mutagenesis Kit from Agilent Technologies (Catalog # 210518). All mutations were confirmed by full-length spike gene sequencing by Sanger Sequencing, using Sequencher and SnapGene for sequence analyses. Pseudovirions were produced in HEK293T/17 cells (ATCC cat. no. CRL-11268) by transfection using Fugene 6 (Promega Cat#E2692) and a combination of spike plasmid, lentiviral backbone plasmid (pCMV ΔR8.2) and firefly Luc reporter gene plasmid (pHR’ CMV Luc)³⁶ in a 1:17:17 ratio in Opti-MEM (Life Technologies).

Transfection mixtures were added to pre-seeded HEK 293T/17 cells in T-75 flasks containing 12 ml of growth medium and incubated for 16–20 h at 37°C. Medium was removed and 15 ml of fresh growth medium added. Pseudovirus-containing culture medium was collected after an additional 2 days of incubation and clarified of cells by low-speed centrifugation and 0.45 μm micron filtration. TCID50 assays were performed as described previously.³⁵ For measurements of neutralization, a pre-titrated dose of virus was incubated with 8 serial 5-fold dilutions of serum samples (1:10 or 1:20 starting dilution) in duplicate in a total volume of 150 μl for 1 h at 37°C in 96-well flat-bottom poly-L-lysine-coated culture plates. 293T/ACE2-MF cells were detached from T75 culture flasks using TrypLE Select Enzyme solution, suspended in growth medium (100,000 cells/ml) and immediately added to all wells (10,000 cells in 100 μL of growth medium per well). One set of 8 wells received cells + virus (virus control) and another set of 8 wells received cells only (background control). After 66–72 h of incubation, medium was removed by gentle aspiration and 30 μl of Promega 1X lysis buffer was added to all wells. After a 10-minute incubation at room temperature, 100 μl of Bright-Glo luciferase reagent was added to all wells. After 1–2 minutes, 110 μl of the cell lysate was transferred to a black/white plate. Luminescence was measured using a GloMax Navigator luminometer (Promega). Serum samples were heat-inactivated for 30 minutes at 56°C prior to assay. Neutralization titers are the inhibitory dilution (ID) of serum samples at which RLUs were reduced by either 50% (ID50) or 80% (ID80) compared to virus control wells after subtraction of background RLUs.

Monogram

Neutralizing antibody activity was measured in a formally validated assay that utilized lentiviral particles pseudotyped with full-length SARS-CoV-2 Spike protein and containing a firefly luciferase (Luc) reporter gene for quantitative measurements of infection by relative luminescence units (RLU). The backbone vector used in pseudovirus creation, F-lucP.CNDOΔU3, encodes the HIV genome with firefly luciferase replacing the HIV env gene. A codon-optimized version of the full-length spike gene of the Wuhan-1 SARS-CoV-2 strain (MN908947.3) (GenScript) was cloned into the Monogram proprietary env expression vector, pCXAS-PXMX, for use in the assay. The D614G spike mutation was introduced into the original Wuhan sequence by site-directed mutagenesis. Sequences of the spike gene and expression vector were confirmed by full-length sequencing using Illumina MiSeq NGS.

Pseudovirus stock was produced in HEK 293 cells via a calcium phosphate transfection using a combination of spike plasmid (pCXAS-SARS-CoV-2-D614G) and lentiviral backbone plasmid (F-lucP.CNDOΔU3). Transfected 10 cm² plates were re-fed the next day and harvested on Day 2 post transfection. The pseudovirus stock (supernatant) was collected, filtered and frozen at ≤70°C in single-use aliquots. Pseudovirus infectivity was screened at multiple dilutions using HEK293 cells transiently transfected with ACE2 and TMPRSS2 expression vectors. RLUs were adjusted to ~50,000 for use in the neutralization assay. Neutralization was performed in white 96-well plates by incubating pseudovirus with 10 serial 3-fold dilutions of serum samples for one hour at 37°C. Serum samples were heat-inactivated for 60 minutes at 56°C prior to assay.

The dilution series was based on a 1:20 starting dilution which was reported as 1:40 after addition of virus. HEK293 target cells, which had been transfected the previous day with ACE2 and TMPRSS2 expression plasmids, were detached from 10 cm² plates using trypsin/EDTA and re-suspended in culture medium to a final concentration that accommodated the addition of 10,000 cells per well. Cell suspension was added to the serum-virus mixtures and assay plates were incubated at 37°C in 7% CO₂ for 3 days. On the day of assay read, Steady Glo (Promega) was added to each well. Reactions were incubated briefly and luciferase signal (RLU) was measured using a luminometer. Neutralization titers represent the inhibitory dilution (ID) of serum samples at which RLUs were reduced by either 50% (ID50) or 80% (ID80) compared to virus control wells (no serum wells).

The Monogram assay employs a specificity control which is created using the same HIV backbone/Luc sequence used in the SARS-CoV-2 pseudovirus. The envelope is 1949 Influenza A H10N3. It is unlikely for human sera to have antibodies against this rare avian influenza virus. The specificity control is designed to detect non-antibody factors (e.g., ART therapy) that could inhibit SARS-CoV-2 pseudovirus and result in false positive measurements of antibody neutralization. Positive anti-SARS-CoV-2 nAb activity was defined as an anti-SARS-CoV-2 nAb titer >3 times greater than the titer of the same serum tested with the specificity control.

Serum Samples

mRNA-1273 vaccine recipients:

Neutralization activity was assayed in a total of 90 serum samples collected in a phase 1 trial^37,38 (NCT04283461) that used the same mRNA-1273 vaccine, dose, and schedule as that used in the Moderna phase 3 trial³ (NCT04470427). Sera were collected from 30 vaccine recipients at day 0, day 29 (4 weeks post-first dose), and day 57 (4 weeks post-second dose). Samples were stored at −80°C, thawed, and heat-inactivated for 30 minutes (Duke) or 60 minutes (Monogram) at 56°C. Heat-inactivated samples were stored at 4°C (Duke) or at −80°C (Monogram) until assayed.

Convalescent patients:

Neutralization activity was also assayed in a total of 248 serum samples collected in HVTN 405/HPTN 1901 (NCT04403880), an observational cohort study that enrolled individuals 18 years or older who had a positive SARS-CoV-2 test and recovered from a spectrum of infection and COVID-19 severity, from asymptomatic to requiring advance medical care in the intensive care unit, within 1–8 weeks of enrollment (if symptomatic). The laboratory was blinded to the clinical status of the donors but not to HIV-1 infection status. Only convalescent serum samples from HIV-1-uninfected individuals were used in the present analyses because antiretroviral drugs, which individuals with HIV may be taking, can inhibit replication of the HIV backbone of the pseudovirus used in the assay. Samples were stored at −80°C, thawed, and heat-inactivated for 30 minutes (Duke) or 60 minutes (Monogram) at 56°C. Heat-inactivated samples were stored at 4°C (Duke) or at −80°C (Monogram) until assayed.

WHO anti SARS-CoV-2 immunoglobulin International Standard (IS) sample

In December 2020, the WHO released a well-characterized international standard for the purpose of improving comparability of results among different assays in different laboratories and reducing interlaboratory variability of anti-SARS-CoV-2 antibody assays.³³ As described in the user instructions provided by the National Institute for Biological Standards and Controls (NIBSC): “The First WHO International Standard for anti-SARS-CoV-2 immunoglobulin is the freeze-dried equivalent of 0.25 mL of pooled plasma obtained from eleven individuals recovered from SARS-CoV-2 infection. The preparation was evaluated in a WHO International Collaborative study. The intended use of the International Standard is for the calibration and harmonization of serological assays detecting anti-SARS-CoV-2 neutralizing antibodies. The preparation can also be used as an internal reference reagent for the harmonization of binding antibody assays. The preparation has been solvent-detergent treated to minimize the risk of the presence of enveloped viruses”.³⁹

Calibration approaches and evaluation of concordance

We discuss and compare three different approaches for the calibration of vaccine-induced neutralizing antibody responses measured by the Duke lab and the Monogram lab. The Monogram lab is used as the reference lab in Approaches 2 and 3 because vaccine-induced neutralizing antibody responses from 4 out of 5 of the USG-sponsored COVID-19 phase 3 vaccine trials are being assayed by, or are planned to be assayed by, the Monogram lab for correlates of protection assessments.

In Approach 1, the Duke and Monogram responses are calibrated to a common scale based on the WHO IS sample. For measurements from both labs the calibrated responses cID50 and cID80 in international units (IUs) per microliter (ml) are calculated as the observed ID50 or ID80 titers multiplied by the calibration factor, which is calculated as 1,000 (IU/ml) divided by the geometric mean, median or arithmetic mean ID50 or ID80 titers of the WHO IS sample that each lab measured in multiple vials. During the validation step, the calibration factor based on one of the above summary statistics that renders the highest agreement between calibrated responses from the two labs for the same set of samples is selected.

In Approach 2, both the Duke and Monogram responses are calibrated to a common scale using the Monogram lab as the reference based on methods described in Huang et al.⁴⁰ Briefly, both readouts are assumed to share the same underlying true values but carry lab-specific measurement errors, following a bivariate normal distribution. Based on the assay validation experiments from both labs, a 30% coefficient of variation is assumed for the measurement error for ID50 and ID80 titers in both assays. The calibrated values are then calculated as the expected conditional means derived from the joint bivariate distribution as ${\mu }_1+\frac{{\sigma }_{x^{\left(1\right)}}^2}{{\sigma }_{x^{\left(1\right)}}^2+{\tau }_1^2}\left({\omega }^{\left(1\right)}-\left({\mu }_1-{\delta }_1\right)\right)$ for the Monogram assay, where μ₁ and ${{\sigma }^2}_{x^{\left(1\right)}}$ are estimated by the sample mean and variance of the log-transformed ID50 and ID80 titers of the convalescent samples measured by the Monogram assay, ${\tau }_1^2$ is the estimated measure error of the assay (= 0.3^2 = 0.09), δ₁ is zero because the Monogram assay is considered as the reference, and ω⁽¹⁾ is the log-transformed pre-calibrated titer of the vaccine sample measured by the Monogram assay. Similarly, the calibrated values for the Duke assay are calculated as ${\mu }_2+\frac{{\sigma }_{x^{\left(2\right)}}^2}{{\sigma }_{x^{\left(2\right)}}^2+{\tau }_2^2}\left({\omega }^{\left(2\right)}-\left({\mu }_2-{\delta }_2\right)\right)$, where all parameters are estimated using data from the Duke assay as is done for the Monogram assay, except that δ₂ is estimated by the average of the log-transformed difference between the Duke and Monogram assay readouts of the convalescent samples.

In Approach 3, the Duke responses are calibrated using the relationship between Duke and Monogram readouts of the convalescent samples. Specifically, a linear regression model is used to regress the Monogram readouts on the Duke readouts; the calibrated Monogram scale value of a Duke sample is calculated by plugging in the observed Duke vaccine readout as the predictor in the regression model.

The concordance correlation coefficient (CCC)⁴¹ is used to quantify the level of agreement between the calibrated values of the two labs using samples measured by both labs. The overall CCC is then calculated as the average time-specific CCC using an inverse hyperbolic tangent transformation for stability. For both Approaches 2 and 3, to account for the statistical variability in the calibrated responses, 1000 bootstrap samples of the convalescent paired sample dataset are drawn with replacement to estimate the precision of the estimated CCC.