Abstract
A total of 1080 individual patient samples (158 positive serology samples from confirmed, predominantly mildly symptomatic COVID-19 patients and 922 serology negative including 496 collected pre-COVID) from four states in Australia were analysed on four commercial SARS-CoV-2 serological assays targeting antibodies to different antigens (Roche Elecsys and Abbott Architect: nucleocapsid; Diasorin Liaison and Euroimmun: spike). A subset was compared to immunofluorescent antibody (IFA) and micro-neutralisation. Sensitivity and specificity of the Roche (n = 1033), Abbott (n = 806), Diasorin (n = 1034) and Euroimmun (n = 175) were 93.7 %/99.5 %, 90.2 %/99.4 %, 88.6 %/98.6 % and 91.3 %/98.8 %, respectively. ROC analysis with specificity held at 99 % increased the sensitivity for the Roche and Abbott assays from 93.7% to 98.7% (cut-off 0.21) and 90.2 % to 94.0 % (cut-off 0.91), respectively. Overall seropositivity of samples increased from a maximum of 23 % for samples 0−7 days-post-onset of symptoms (dpos), to 61 % from samples 8−14dpos and 93 % from those >14dpos. IFA and microneutralisation values correlated best with assays targeting antibodies to spike protein with values >80 AU/mL on the Diasorin assay associated with neutralising antibody. Detectable antibody was present in 22/23 (96 %), 20/23 (87 %), 15/23 (65 %) and 9/22 (41 %) patients with samples >180dpos on the Roche, Diasorin, Abbott and microneutralisation assays respectively. Given the low prevalence in this community, two-step algorithms on initial positive results saw an increase in the positive predictive value (PPV) of positive samples (39 %–65 % to ≥98 %) for all combinations. Similarly accuracy increased from a range of 98.5 %–99.4 % to ≥99.8 % assuming a 1 % seroprevalence. Negative predictive value (NPV) was high (≥99.8 %) regardless of which assay was used initially.
Keywords: SARS-CoV-2 IgG, Commercial immunoassay, Neutralising antibody, Immunofluorescent antibody assay
1. Introduction
As of December 15, 2020, more than 73 million cases of COVID-19 have been diagnosed causing over 1.6 million deaths worldwide. While diagnosis has relied largely on SARS-CoV-2 nucleic acid amplification testing (NAAT), for a small number of patients with equivocal or negative NAAT results, serology has been instrumental in clarifying the true infection status of a case. Serology has a greater role in retrospectively diagnosing COVID-19 especially in asymptomatic cases [1,2]. This in turn improves estimates of attack rate, case fatality rate and reproduction number (R0) in a population [[3], [4], [5]]. Serology may also have prognostic value, with antibody titres found to correlate with severity of infection [4,[6], [7], [8], [9]], and can assist public health investigations of outbreaks [5].
In the longer term, studies may be able to assess whether herd immunity against SARS-CoV-2 has been achieved [3]. If high-throughput commercial Enzyme Linked Immunosorbent Assays (ELISA) or Chemiluminescent Microparticle Immunoassays (CMIA/CLIA) correlate with neutralising antibody (Nab) titres, an immune population may be determined who are lower risk for returning to frontline work [3,5]. At present, serological correlates of immunity post-vaccination are yet to be determined and published studies of Nab following infection with SARS-CoV-2 remain limited [6,[10], [11], [12], [13], [14]].
Commercial assays generally target one of two proteins: the spike or the nucleocapsid protein with uncertainty about which targets are the most sensitive or correlate best with Nab [15,16].
While published studies to date have compared commercial assays head-to-head [[17], [18], [19], [20], [21]] most have been in higher prevalence areas. Australia and New Zealand have been fortunate in largely eliminating transmission in the community but this has presented a unique diagnostic challenge to understand the relative performance of commercial test platforms for use in a very low incidence community. We sought to investigate the reliability of four commercial assays for use within our laboratory network in Australia.
2. Methodology
Commercial assay testing (Roche Elecsys Anti-SARS-CoV-2 assay targeting IgM/IgA/IgG to nucleocapsid protein, Abbott Architect SARS-CoV-2 IgG targeting nucleocapsid protein, Diasorin Liaison SARS-CoV-2 S1/S2 IgG targeting the S1 and S2 domains of the spike protein, Euroimmun Anti-SARS CoV-2 IgG targeting the S1 domain of the spike protein) was performed at 4 laboratories across 4 states of Australia according to the assays’ instructions for use (IFU). Due to limited availability of the Euroimmun assay and residual stored sera, samples from COVID-19 patients were prioritised over specificity samples for testing using this assay. The Institute of Clinical Pathology and Medical Research, NSW performed an in-house immunofluorescent antibody (IFA) assay [22] and a microneutralisation assay [23] as previously described. All data was de-identified and verbal consent of COVID-19 patients was obtained and approval for this study obtained from the Sullivan Nicolaides Pathology Low Risk Ethics Committee.
Sensitivity analysis was performed on stored sera from confirmed patients with COVID-19 diagnosed by NAAT as defined by local guidelines and also household contacts seropositive for IgG by IFA in the absence of NAAT being performed [24,25]. The majority of cases were diagnosed by the Seegene Allplex 2019-nCoV Assay (targeting E, N and RdRP genes) or an in-house developed Taqman assay targeting the E and N genes.
Only the latest sample for each patient was selected. Seropositive samples from confirmed COVID-19 cases had to be positive by at least IFA if tested or in the absence of IFA, at least one commercial assay. A two-step algorithm analysing the different possible combinations of commercial assays to optimise positive predictive value (PPV) was evaluated [26].
The COVID-19 cases and three cohorts used to assess the specificity of the assays under evaluation are depicted in Table 1 .
Table 1.
Sensitivity and Specificity samples.
| Sensitivity cohort | n (%) | Specificity cohort | n (%) | |
|---|---|---|---|---|
| COVID-19 cases | Qld 77 (45) | Cohort 1 pre-COVID | 411 | |
| NSW 71 (41) | -antenatal | 324 (79) | ||
| Victoria 14 (8) | -healthy adults 2019 | 57 (14) | ||
| WA 11 (6) | -children 2019 | 30 (7) | ||
| Seropositive COVID-19 cases | Qld 67 (42) | Cohort 2* | 378 | |
| NSW 66 (42) | Cross-reactivity (pre-COVID) | 235 (62) | ||
| Victoria 14 (9) | Cross-reactivity (COVID) | 143 (38) | ||
| WA 11 (7) | ||||
| Cohort 3 | 118 | |||
| ARI; COVID-19PCR negative | ||||
| Total COVID-19 cases | 173 (16) | Total specificity samples | 907 (84) | Total overall samples 1080 (100) |
| Total seropositive (% of total overall) | 158 (15) | pre-COVID | 646 (71) |
NSW New South Wales, Qld Queensland, WA Western Australia.
ARI: Acute respiratory infection.
Including but not limited to patients diagnosed with the following respiratory pathogens: influenza A/B n = 55, parainfluenza n = 16, respiratory syncytial virus n = 12, adenovirus n = 10, rhinovirus n = 3, Mycoplasma pneumoniae n = 16, Bordetella pertussis n = 14, Chlamydia pneumoniae n = 10, Legionella longbeachae n = 8; non-respiratory pathogens positive for EBV n = 53, CMV n = 43, Parvovirus B19 n = 19, Hepatitis A/B/C n = 26, HIV 5, Ross River virus n = 17; or highly elevated autoantibodies: Rheumatoid factor n = 19, Antinuclear antibody n = 28.
A novel illness severity score (ISS) assessing 19 symptoms and need for hospitalisation, intensive care support and mechanical ventilation (HIM) was developed and evaluated on a subset of patients as depicted in Table 2 . Duration of symptoms (DS) was also assessed and capped at 30 days.
Table 2.
Illness Severity Score.
| 1 point | Score | 2 points | Score | 3 points | Score | |
|---|---|---|---|---|---|---|
| Symptoms | Chills/sweats | □ | Fever | □ | Shortness of breath | □ |
| Cough | □ | Rigors | □ | |||
| Sore throat | □ | Chest pain | □ | |||
| Rhinorrhoea | □ | Myalgia/ arthralgia | □ | |||
| Headache | □ | |||||
| Dizziness | □ | |||||
| Tiredness | □ | |||||
| Conjunctivitis | □ | |||||
| Loss of smell | □ | |||||
| Loss of taste | □ | |||||
| Rash | □ | |||||
| Anorexia | □ | |||||
| Nausea/ | □ | |||||
| vomiting | ||||||
| Diarrhoea | □ | |||||
| Hospital features | Non-ICU 3pts | □ | ||||
| ICU 4 pts | □ | |||||
| Mechanical ventilation 5pts | □ | |||||
| Score | /14 | /8 | /8 | |||
| Total score | /30 |
Equivocal results for Euroimmun and Diasorin were considered positive for statistical analysis. Sub-analysis was performed with serology values obtained from IFA, ‘nucleocapsid assays’(NA), ie Abbott and Roche assays and ‘spike assays’ (SA), ie Diasorin and Euroimmun.
Receiver operator characteristic (ROC) curve analysis was undertaken to determine the area under the curve (AUC), and sensitivity, specificity and accuracy, along with 95 % confidence intervals (CI), were calculated. Assay cut-off values were optimised by interrogation of ROC results.
Pairwise comparisons of sensitivity and specificity between assays were made. As many of the same samples were assessed by each assay, the diagnostic measures were correlated with each other, and to account for this a procedure using 2000 bootstrap replicates was adopted [27]. Sensitivity was tested while holding specificity at 99 %, then specificity was tested while holding sensitivity at 95 %. A significance threshold of 0.05 was used, with a Bonferroni correction for multiple testing giving an adjusted threshold for 6 tests of α' = 0.0083.
We evaluated the relationship between IFA and assay values using linear regression analysis, with adjustment for the covariates age and sex. As the IFA distribution was skewed, a normalising square root transformation was applied before subsequent analysis. Associations between the outcomes IFA, NA and SA and the predictors ISS, DS and HIM were similarly explored, with adjustment for the covariates age, sex and age by sex interaction. To correct for the effect of multiple testing on Type I error in this setting, the significance threshold was set at α' = 0.05/9 = 0.0056. The same associations were also examined in sex-stratified analyses.
Data were analysed in the R statistical computing environment, version 4.0.0, including the package pROC [27].
3. Results
There were 173 patients with confirmed COVID-19 by NAAT (n = 169) or IFA (n = 4) who had serology samples evaluated. Of these, 90 (52 %) were male and the mean age of patients was 50.7 years (range 7−85years). Severity of disease data for 163 patients indicated that 5 (3 %) were asymptomatic, 131 (80 %) were symptomatic but non-hospitalised, 18 (11 %) were hospitalised/non-ICU patients and 9 (6 %) were ICU patients. Thirty day mortality was 0 %.
Of the 173 individual samples from confirmed cases and 907 negative samples, there were 158/1080 (14.6 %) seropositive patients. Time since illness onset available for 147 patients found 139/147 (95 %) were collected ≥10 days-post-symptom onset (dpos). Sensitivity, specificity and accuracy results are presented in Table 3 . In the subset of 175 samples common to all assays, there were no differences in AUC among the 4 assays (all p > 0.042), nor in specificity when sensitivity was held at 95 % (all p > 0.15). When specificity was held at 99 %, there was a significant difference in sensitivity only between the Roche and Diasorin assays (p = 0.003).
Table 3.
Diagnostic performance of 4 assays at the published cut-offs for detection of COVID-19 disease (a) in all 1080 samples, of which 158 (14.6 %) tested positive; and (b) in subset of 175 subjects with measurements on all 4 assays, of which 92 (52.6 %) tested positive. Values are expressed as percentages (95 % CI), and as a decimal fraction for AUC.
| (a) All samples | ||||
|---|---|---|---|---|
| Roche | Abbott | Diasorin | Euroimmun | |
| N | 1033 | 806 | 1034 | 175 |
| Cut-off | 1 | 1.4 | 12* | 0.8* |
| AUC | 0.997 (0.994, 1) | 0.994 (0.986, 1) | 0.977 (0.961, 0.992) | 0.990 (0.980, 1) |
| Sensitivity | 93.7 (89.9, 97.5) | 90.2 (85.0, 95.5) | 88.6 (83.5, 93.0) | 91.3 (84.8, 96.7) |
| Specificity | 99.5 (99.1, 99.9) | 99.4 (98.8, 99.9) | 98.6 (97.8, 99.3) | 98.8 (96.4, 100) |
| Accuracy | 98.6 (98.0, 99.3) | 97.9 (96.9, 98.8) | 97.1 (96.1, 98.1) | 94.9 (91.4, 97.7) |
| (b) Subset of 175, with values on all 4 assays | ||||
|---|---|---|---|---|
| Roche | Abbott | Diasorin | Euroimmun | |
| N | 175 | 175 | 175 | 175 |
| Cut-off | 1 | 1.4 | 12* | 0.8* |
| AUC | 0.999 (0.996, 1) | 0.989 (0.976, 1) | 0.984 (0.967, 1) | 0.990 (0.980, 1) |
| Sensitivity | 93.5 (88.0, 97.8) | 89.1 (82.6, 94.6) | 87.0 (79.4, 93.5) | 91.3 (84.8, 96.7) |
| Specificity | 100 (100, 100) | 97.6 (94.0, 100) | 97.6 (94.0, 100) | 98.8 (96.4, 100) |
| Accuracy | 96.6 (93.7, 98.9) | 93.1 (89.1, 96.6) | 92.0 (87.4, 95.4) | 94.9 (91.4, 97.7) |
Cut-offs presented for Diasorin and Euroimmun are the equivocal and borderline ranges respectively; positive for Diasorin and Euroimmun is considered ≥15 and ≥1.1 respectively.
When assay cutoff values for the Roche and Abbott assays were optimised (1.0 to 0.21 and 1.4 to 0.91, respectively) to achieve 99 % specificity, sensitivity increased for both assays (Table 4 ).
Table 4.
Diagnostic performance of 4 assays for detection of COVID19 disease, when specificity is set at 99 %. Values are expressed as percentages (95 % CI).
| Roche | Abbott | Diasorin | Euroimmun | |
|---|---|---|---|---|
| N | 1033 | 806 | 1034 | 175 |
| Published Cut-off | 1 | 1.4 | 12* | 0.8* |
| 99 % Specificity Cut-off | 0.210 | 0.91 | 16.7 | 1.55 |
| Sensitivity | 98.7 (93.7, 1) | 94.0 (83.5, 98.5) | 86.1 (67.7, 92.4) | 83.7 (72.8, 97.8) |
| Accuracy | 99.0 (98.2, 99.2) | 98.2 (96.4, 98.9) | 97.0 (94.2, 98.0) | 91.0 (85.2, 98.4) |
Cut-offs presented for Diasorin and Euroimmun are the equivocal and borderline ranges respectively; positive for Diasorin and Euroimmun is considered ≥15 and ≥1.1 respectively.
Twenty-nine samples from 22/158 (14 %) COVID-19 patients were found to be discrepant on ≥1 commercial assays (Table 5 ). For a family of three COVID-19 cases, one had a positive result only on SA (16−32 dpos), one only on NA (12 dpos) and one with positive results from at least one of SA and NA (17dpos). Falsely negative results were found in 9/29 (31 %), 14/29 (48 %), 14/29 (48 %), and 17/29 (59 %) Euroimmun, Roche, Abbott and Diasorin results respectively. Two patients were only positive by IFA but negative/equivocal on commercial assays (although values for these two patients were above the mean negative value of pre-COVID samples).
Table 5.
Discrepant results in 22 COVID-19 positive patients.
 |
dpos: days post onset of symptoms; Cat: Categorisation; P: Positive, N: Negative, EQ: Equivocal; Results highlighted in blue indicate samples that were only positive on nucleocapsid-specific assays while those highlighted in yellow indicate samples that were only equivocal/positive on spike-specific assays.
Sensitivities by time period are presented in Table 6 . Seroconversion occurred by at least 12 dpos on commercial assays (range 5−12dpos). Samples remained positive beyond 180 days in the majority of patients by the Roche (96 %), Diasorin (87 %) and Abbott assays (65 %) (Fig. 1 ).
Table 6.
Sensitivity of various assays for detection of SARS-CoV-2 antibodies by increasing time intervals; All Positive: positive by at least one assay; dpos: days post onset of symptoms.
| dpos | n (%) | All Positive n (%) |
Roche n (%) |
Abbott n (%) |
Euroimmun n (%) |
Diasorin n (%) |
IFA n (%) |
|---|---|---|---|---|---|---|---|
| 1−7 | 32 (13) | 12/32 (38) | 3/32 (9) | 4/27 (15) | 5/22 (23) | 6/32 (19) | – |
| 8−14 | 23 (9) | 15/23 (65) | 14/23 (61) | 8/19 (42) | 5/16 (31) | 10/23 (44) | 1/1 |
| >14 | 190 (78) | 187/190 (99) | 171/190 (90) | 154/182 (85) | 103/111 (93) | 171/190 (90) | – |
| >28 | 148 (60) | 147/148 (99) | 139/148 (94) | 122/144 (85) | 69/74 (93) | 133/148 (90) | 72/75 (96) |
| >60 | 40 (16) | 40/40(100) | 39/40 (98) | 28/40 (70) | – | 36/40 (90) | 22/25 (88) |
| >180 | 23 (9) | 23/23 (100) | 22/23 (96) | 15/23 (65) | – | 20/23 (87) | 20/23(87) |
| Total | 245 | 214/245 (87) | 188/245 (77) | 168/228 (74) | 113/149 (76) | 187/245 (76) | – |
Fig. 1.
Trajectory of serology values in patients with samples collected at approximately 42 dpos (31-48) and after 180 dpos (183-219) in 23 patients. The dashed blue line represents the assay threshold for positivity.
The mean cutoff value from the 3 specificity cohorts ranged between 0.08−0.09, 0.06−0.11, 4.3–4.5 and 0.3 (positive cutoff of 1.0, 1.4, 15.0 and 1.1) for Roche, Abbott, Diasorin and Euroimmun assays respectively.
Commercial assays were also compared with 51 positive and 47 negative IFA results with high positive agreement (Diasorin 43/51 (84 %), Abbott 45/51 (88 %), Euroimmun 46/51 (90 %) and Roche 48/51 (94 %)) and negative agreement (Diasorin and Roche 47/47 (100 %) and Abbott and Euroimmun 46/47 (98 %)). From regression analysis after adjustment for age, there was a stronger association between IFA and SA values (Diasorin: R2 = 59 %; Euroimmun: R2 = 61 %;) than between IFA and NA values (Roche: R2 = 10 %; Abbott: R2 = 23 %).
Thirty microneutralisation results from 27 patients were available (range 32−203 dpos). Fourteen patients (52 %) had detectable Nab (6/6 samples collected <180 dpos, 9/22 (41 %) samples ≥180 dpos). Significantly higher titres were found in those who were hospitalised, while samples collected ≥180 dpos had lower titres (p < 0.003). A minority of patients (1/14, by Roche; 2/14 by Diasorin and 3/14 by Abbott) were negative by all commercial assays but had detectable Nab. Increasing assay values only on the SA correlated with presence of Nab. All 8 patients with an assay value >80 AU/mL by Diasorin were found to have detectable Nab with higher values in males than females (Fig. 2 ).
Fig. 2.
Correlation of Diasorin assay values with Neutralising antibody titres in 23 patients.
* One was added to all Nab values to enable the use of a logarithmic scale, however the vertical axis is marked to reflect the original Nab values.
Eighteen possible two-step algorithms (including optimised sensitivities for the Roche and Abbott assays) were analysed assuming a population prevalence of 1 % based on recent data indicating low seroprevalence in our community [28]. PPV after the second assay improved from a range of 0.39−0.65 to ≥0.98 for all combinations. The highest PPV was obtained from the combination of the Roche and Abbott assays (0.996). Accuracy increased from a range of 0.985−0.994 to ≥0.998. The most accurate combination was the optimised Roche assay followed by the Euroimmun assay (0.999). Calculations for the Roche followed by Diasorin combination are illustrated in Fig. 3 .
Fig. 3.
An example of a 2-step SARS-CoV-2 serology algorithm.
Twenty pairs of household contacts (HHC) (spouses/partners/family members) providing 54 samples collected within 4 dpos of each other, with range of collection : 7-219 dpos, were matched with 18 pairs of unrelated patients (URP). No significant difference was found between the HHC and URP within-pair assay differences for IFA or the 4 commercial assays, even when this was limited to the subset of patients with probable same exposure (Table 7 ).
Table 7.
Comparison of mean assay difference (diff) between household contact (HHC) pairs and matched unrelated patient pairs (URP); values for continuous variables are presented as mean (standard deviation).
| HHC | HHC (same exposure) | URP | P-value |
||
|---|---|---|---|---|---|
| HHC v URP | HHC (same exposure) v URP |
||||
| Number of patients | 40 | 22 | 36 | ||
| Number of patient pairs | 20 | 11 | 18 | ||
| Number of samples | 54 | 30 | 36 | ||
| % Male | 48 | 41 | 47 | ||
| Age mean | 51.6 (17.1) | 50.9 (17.0) | 51.4 (15.5) | 0.952 | 0.912 |
| dpos diff mean | 1.8 (1.6) | 1.5 (1.4) | 8.0 (6.1) | 0.005 | 0.061 |
| Roche pairs | 27 | 15 | 18 | ||
| Roche mean diff | 22.4 (19.6) | 22.4 (19.8) | 32.9 (28.8) | 0.285 | 0.360 |
| Abbott pairs | 27 | 15 | 18 | ||
| Abbott mean diff | 2.3 (2.1) | 2.9 (2.4) | 2.2 (2.1) | 0.839 | 0.343 |
| Diasorin pairs | 27 | 15 | 18 | ||
| Diasorin mean diff | 52.2 (60.1) | 50.2 (51.8) | 63.7 (63.1) | 0.616 | 0.514 |
| Euroimmun pairs | 17 | 8 | 9 | ||
| Euroimmun mean diff | 2.9 (2.0) | 2.8 (1.8) | 2.6 (2.6) | 0.689 | 0.817 |
The constructed ISS, evaluated in 62 seropositive patients, ranged from 0 to 17/30 with a mean value of 9. The presence of HIM was more commonly demonstrated in males than females (13/35 vs 2/27). Mean DS was 20 days (range 1 to ≥30 days). ISS was not found to be significantly associated with any serology outcomes. DS showed a significant positive association only with SA (p = 0.001) and HIM was significantly positively associated only with IFA and SA (both p ≤ 0.001). There was a divergence between male and female groups for Diasorin, Euroimmun, SA and IFA values (Fig. 4 ). In females, age was found to be significantly positively associated with IFA and all assay values (all p ≤ 0.004); while HIM, DS and ISS were not. By contrast, in males, age was not associated with any outcome (all p > 0.05); HIM showed positive associations only with SA and IFA (both p ≤ 0.004); and DS and ISS were not significantly associated with any assay outcomes.
Fig. 4.
Boxplot of assay values by sex for individual assays and grouped by target antigen (SA and NA) and IFA in the subset of subjects who have an IFA value. 0= female, 1=male.
Mean/median ratios of SA:NA values for Diasorin:Roche, Diasorin:Abbott, Euroimmun:Roche and Euroimmun:Abbott were significantly higher in hospitalised compared with non-hospitalised patients and highest in those in ICU (Fig. 5 ).
Fig. 5.
Boxplots of spike:nucleocapsid median assay values for the different combinations in non-hospitalised vs hospitalised patients; D:A Diasorin:Abbott, D:R Diasorin:Roche, E:A Euroimmun:Abbott, E:R Euroimmun:Roche.
4. Discussion
This evaluation found satisfactory performance and comparability of the tested commercial assays. Using all available samples, specificity was >98.5 % (range 98.6–99.5 %), and sensitivity/accuracy was high (range 88.6–93.7 %/ 94.9–98.6 % respectively). Sensitivity/specificity varied by 0.2–1.1 %/0.5−1.8% respectively when analysis was confined to samples that were measured by all 4 assays. The area under the curve (AUC) was >97 % for all assays, indicating high discriminatory power. When analysed by time period, sensitivity increased from a maximum of 61 %<14 dpos to 93 %>14 dpos and 98 %>60 dpos. The positive cases in our dataset experienced predominantly mild to moderate COVID-19 and may explain the reduced sensitivities when compared with those in the assay IFUs.
Discordant results by commercial assays were found in a minority (14 %) of our patients as has been found in other studies [12,17] and may relate to assay design or immune response. Also of note was the difference in persisting seropositivity between the Abbott and Roche assays on samples ≥180 dpso despite targeting the same nucleocapsid protein in keeping with the indirect ‘sandwich assay’ design of the Roche assay that preferentially detects more mature antibody. This finding supports the use of more than one assay with differing target antigens or design as has been recommended locally and by others [8,29] to allow serological testing to address a variety of clinical and public health questions.
This is also the only currently available study providing large numbers in the Oceania region where COVID-19 case numbers are relatively limited. Reassuringly, the high specificities of the assays found in this study corresponded closely to those reported by the manufacturers (≤0.5 % difference). We found that initial false positive results can be further reduced by adopting a two-step approach utilising a second assay as found by others [21,30]. Only NA (Roche and Abbott) gained sensitivity by lowering of the positive cutoff without a significant reduction in specificity. A two-step algorithm may also increase the detection of low level antibody only detectable by one type of assay. In one of our laboratories, the use of the Roche assay as the initial screening assay has been enhanced by reflexing any results >0.2 cutoff index to the Diasorin assay. In the other laboratory in our network performing serology, positive results on the Diasorin were reflex tested on the Abbott assay. While these two combinations were not the most accurate or provided the highest PPV, they resulted in the most efficient laboratory workflow using assays targeting two different antigens and resulted in a PPV/accuracy of 98.4 %/99.9 % and 99.0 %/99.8 %respectively. In the near future, with the advent of vaccines that largely target the spike protein, there may be a need to differentiate the development of antibodies and immunity following infection (NA + SA positive) or vaccination (SA positive only) providing further reason for utilising assays that target both antigens. However, careful evaluation of the performance of available assays is required to maximise PPV as was found by Ripperger et al. where their in-house nucleocapsid-specific assay was not suitably discriminatory for use in their two-step algorithm [30].
This is also one of only a few studies that have correlated commercial assay results with Nab and the only one we are aware of comparing it with reference laboratory IFA results [11]. Overall concordance of negative and positive samples with IFA was high (92–97 %) and the high sensitivity of IFA in samples ≥28 dpos (96 %) was comparable to that reported recently by members of our group (91 % for samples ≥14 dpos) [22]. Nab, currently considered the best potential correlate with protective immunity, was still detectable in 9/22 (41 %) of those with samples collected more than 6 months post-illness. Nab also corresponded to IFA results and SA values. Taking into account the higher proportion (67 %) of late collected samples (≥180 dpos) of predominantly mildly infected patients in our study, we also found a similar proportion of samples with Diasorin values >80 AU/mL associated with Nab titres of ≥1:160 (5/9, 56 % vs 87 % in the Diasorin IFU). A negative commercial assay result does not exclude the possibility of detectable Nab and possible protective immunity and caution should be taken when interpreting negative results >6 months following a compatible illness due to the possibility of false negative serology particularly in young females where antibody levels were found to be lowest in both SA and IFA.
There are clear differences in mortality between males and females, despite similar incidence of infection, and this may relate to a differing immune response which is reflected in lower antibody levels [31]. While our novel ISS alone did not correlate with assay values and was likely limited by the retrospective design and associated recall and reporting bias, HIM was found to be significantly associated with higher assay values in the IFA and SA in males. Similar findings were found in one large study where higher antibody levels were seen in older and sicker patients while lower levels were seen in females suggesting gender-specific immune responses towards SARS-CoV-2 [32]. In contrast to other studies that showed lower spike:nucleocapsid antibody ratios in hospitalised and deceased patients [33,34], our study found the higher/highest ratios in the hospitalised/intensive care patients compared with those managed in the community. This may relate to both sampling later in the illness and ultimate recovery of patients in our study, with strong spike-specific antibody responses having been shown to correlate with survival [33].
Since greater inoculum of SARS-CoV-2 has been shown in hamster models to lead to more severe disease [35], by comparing assay values from COVID-19 patients of the same household with matched unrelated patients, we postulated that the household group were more likely to have been exposed to a similar inoculum of SARS-CoV-2 from the same source and therefore might return similar assay values. However, this was not found to be the case and may relate to probable varying inoculum despite the same source/setting in addition to differing host factors amongst household members especially those who were genetically unrelated.
Limitations of this study include its restricted demography relating only to Australian patients and the small sample size of patients with long term results (≥180 dpos).
In conclusion, we demonstrate that currently available commercial assays perform well with the majority of samples taken more than 6 months following infection still positive with associated neutralising antibody detected in almost half. In low prevalence areas such as Australia, the use of a two-step algorithm increases the PPV of a positive result while maintaining high NPV/accuracy.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Transparency document
Declaration of Competing Interest
The authors report no declarations of interest.
Acknowledgements
The authors would like to thank the patients who consented to providing blood samples, their collectors and laboratory staff involved and to Dr Ian Chambers, Dr Miriam Paul and Dr Rodney Givney for review of the manuscript.
References
- 1.Buitrago-Garcia D., Egli-Gany D., Counotte M.J., Hossmann S., Imeri H., Ipekci A.M., Salanti G., Low N. Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: a living systematic review and meta-analysis. PLoS Med. 2020;17 doi: 10.1371/journal.pmed.1003346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Mizumoto K., Kagaya K., Zarebski A., Chowell G. Estimating the asymptomatic proportion of coronavirus disease 2019 (COVID-19) cases on board the Diamond Princess cruise ship, Yokohama, Japan, 2020. Euro Surveill. 2020:25. doi: 10.2807/1560-7917.ES.2020.25.10.2000180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Theel E.S., Slev P., Wheeler S., Couturier M.R., Wong S.J., Kadkhoda K. The role of antibody testing for SARS-CoV-2: Is there one? J. Clin. Microbiol. 2020:58. doi: 10.1128/JCM.00797-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Long Q.X., Liu B.Z., Deng H.J., Wu G.C., Deng K., Chen Y.K., Liao P., Qiu J.F., Lin Y., Cai X.F., Wang D.Q., Hu Y., Ren J.H., Tang N., Xu Y.Y., Yu LH Mo Z., Gong F., Zhang X.L., Tian W.G., Hu L., Zhang X.L., Xiang J.L., Du HX Liu H.W., Lang C.H., Luo X.H., Wu S.B., Cui X.P., Zhou Z., Zhu M.M., Wang J., Xue C.J., Li X.F., Wang L., Li Z.J., Wang K., Niu C.C., Yang Q.J., Tang X.J., Zhang Y., Liu X.M., Li Z.J., Zhang D.C., Zhang F., Liu P., Yuan J., Li Q., Hu J.L., Chen J., et al. Antibody responses to SARS-CoV-2 in patients with COVID-19. Nat. Med. 2020;26:845–848. doi: 10.1038/s41591-020-0897-1. [DOI] [PubMed] [Google Scholar]
- 5.Winter A.K., Hegde S.T. The important role of serology for COVID-19 control. Lancet Infect. Dis. 2020;20:758–759. doi: 10.1016/S1473-3099(20)30322-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Perera R.A., Mok C.K., Tsang O.T., Lv H., Ko R.L., Wu N.C., Yuan M., Leung W.S., Chan J.M., Chik T.S., Choi C.Y., Leung K., Chan K.H., Chan K.C., Li K.C., Wu J.T., Wilson I.A., Monto A.S., Poon L.L., Peiris M. Serological assays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), March 2020. Euro Surveill. 2020:25. doi: 10.2807/1560-7917.ES.2020.25.16.2000421. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Qu J., Wu C., Li X., Zhang G., Jiang Z., Li X., Zhu Q., Liu L. Profile of immunoglobulin g and IgM antibodies against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) Clin. Infect. Dis. 2020;71:2255–2258. doi: 10.1093/cid/ciaa489. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Sun B., Feng Y., Mo X., Zheng P., Wang Q., Li P., Peng P., Liu X., Chen Z., Huang H., Zhang F., Luo W., Niu X., Hu P., Wang L., Peng H., Huang Z., Feng L., Li F., Zhang F., Li F., Zhong N., Chen L. Kinetics of SARS-CoV-2 specific IgM and IgG responses in COVID-19 patients. Emerg. Microbes Infect. 2020;9:940–948. doi: 10.1080/22221751.2020.1762515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Zhao J., Yuan Q., Wang H., Liu W., Liao X., Su Y., Wang X., Yuan J., Li T., Li J., Qian S., Hong C., Wang F., Liu Y., Wang Z., He Q., Li Z., He B., Zhang T., Fu Y., Ge S., Liu L., Zhang J., Xia N., Zhang Z. Antibody responses to SARS-CoV-2 in patients with novel coronavirus disease 2019. Clin. Infect. Dis. 2020;71:2027–2034. doi: 10.1093/cid/ciaa344. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Chandrashekar A., Liu J., Martinot A.J., McMahan K., Mercado N.B., Peter L., Tostanoski L.H., Yu J., Maliga Z., Nekorchuk M., Busman-Sahay K., Terry M., Wrijil L.M., Ducat S., Martinez D.R., Atyeo C., Fischinger S., Burke J.S., Slein M.D., Pessaint L., Van Ry A., Greenhouse J., Taylor T., Blade K., Cook A., Finneyfrock B., Brown R., Teow E., Velasco J., Zahn R., Wegmann F., Abbink P., Bondzie E.A., Dagotto G., Gebre M.S., He X., Jacob-Dolan C., Kordana N., Li Z., Lifton M.A., Mahrokhian S.H., Maxfield L.F., Nityanandam R., Nkolola J.P., Schmidt A.G., Miller A.D., Baric R.S., Alter G., Sorger P.K., Estes J.D., et al. SARS-CoV-2 infection protects against rechallenge in rhesus macaques. Science. 2020;369:812–817. doi: 10.1126/science.abc4776. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wölfel R., Corman V.M., Guggemos W., Seilmaier M., Zange S., Müller M.A., Niemeyer D., Jones T.C., Vollmar P., Rothe C., Hoelscher M., Bleicker T., Brünink S., Schneider J., Ehmann R., Zwirglmaier K., Drosten C., Wendtner C. Virological assessment of hospitalized patients with COVID-2019. Nature. 2020;581:465–469. doi: 10.1038/s41586-020-2196-x. [DOI] [PubMed] [Google Scholar]
- 12.Seow J., Graham C., Merrick B., Acors S., Pickering S., Steel K.J.A., Hemmings O., O’Byrne A., Kouphou N., Galao R.P., Betancor G., Wilson H.D., Signell A.W., Winstone H., Kerridge C., Huettner I., Jimenez-Guardeño J.M., Lista M.J., Temperton N., Snell L.B., Bisnauthsing K., Moore A., Green A., Martinez L., Stokes B., Honey J., Izquierdo-Barras A., Arbane G., Patel A., Tan M.K.I., O’Connell L., O’Hara G., MacMahon E., Douthwaite S., Nebbia G., Batra R., Martinez-Nunez R., Shankar-Hari M., Edgeworth J.D., Neil S.J.D., Malim M.H., Doores K.J. Longitudinal observation and decline of neutralizing antibody responses in the three months following SARS-CoV-2 infection in humans. Nat. Microbiol. 2020;5:1598–1607. doi: 10.1038/s41564-020-00813-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Wajnberg A., Amanat F., Firpo A., Altman D.R., Bailey M.J., Mansour M., McMahon M., Meade P., Mendu D.R., Muellers K., Stadlbauer D., Stone K., Strohmeier S., Simon V., Aberg J., Reich D.L., Krammer F., Cordon-Cardo C. Robust neutralizing antibodies to SARS-CoV-2 infection persist for months. Science. 2020;370:1227–1230. doi: 10.1126/science.abd7728. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Reynolds C.J., Swadling L., Gibbons J.M., Pade C., Jensen M.P., Diniz M.O., Schmidt N.M., Butler D.K., Amin O.E., Bailey S.N.L., Murray S.M., Pieper F.P., Taylor S., Jones J., Jones M., Lee W.J., Rosenheim J., Chandran A., Joy G., Di Genova C., Temperton N., Lambourne J., Cutino-Moguel T., Andiapen M., Fontana M., Smit A., Semper A., O’Brien B., Chain B., Brooks T., Manisty C., Treibel T., Moon J.C., Noursadeghi M., Altmann D.M., Maini M.K., McKnight Á, Boyton R.J. Discordant neutralizing antibody and T cell responses in asymptomatic and mild SARS-CoV-2 infection. Sci. Immunol. 2020:5. doi: 10.1126/sciimmunol.abf3698. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Liu W., Liu L., Kou G., Zheng Y., Ding Y., Ni W., Wang Q., Tan L., Wu W., Tang S., Xiong Z., Zheng S. Evaluation of nucleocapsid and spike protein-based enzyme-linked immunosorbent assays for detecting antibodies against SARS-CoV-2. J. Clin. Microbiol. 2020:58. doi: 10.1128/JCM.00461-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Okba N.M.A., Müller M.A., Li W., Wang C., GeurtsvanKessel C.H., Corman V.M., Lamers M.M., Sikkema R.S., de Bruin E., Chandler F.D., Yazdanpanah Y., Le Hingrat Q., Descamps D., Houhou-Fidouh N., Reusken C., Bosch B.J., Drosten C., Koopmans M.P.G., Haagmans B.L. Severe acute respiratory syndrome coronavirus 2-specific antibody responses in coronavirus disease patients. Emerg. Infect. Dis. 2020;26:1478–1488. doi: 10.3201/eid2607.200841. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Kohmer N., Westhaus S., Ruhl C., Ciesek S., Rabenau H.F. Brief clinical evaluation of six high-throughput SARS-CoV-2 IgG antibody assays. J. Clin. Virol. 2020;129 doi: 10.1016/j.jcv.2020.104480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Perkmann T., Perkmann-Nagele N., Breyer M.K., Breyer-Kohansal R., Burghuber O.C., Hartl S., Aletaha D., Sieghart D., Quehenberger P., Marculescu R., Mucher P., Strassl R., Wagner O.F., Binder C.J., Haslacher H. Side-by-side comparison of three fully automated SARS-CoV-2 antibody assays with a focus on specificity. Clin. Chem. 2020;66:1405–1413. doi: 10.1093/clinchem/hvaa198. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Wolff F., Dahma H., Duterme C., Van den Wijngaert S., Vandenberg O., Cotton F., Montesinos I. Monitoring antibody response following SARS-CoV-2 infection: diagnostic efficiency of 4 automated immunoassays. Diagn. Microbiol. Infect. Dis. 2020;98 doi: 10.1016/j.diagmicrobio.2020.115140. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Krüttgen A., Cornelissen C.G., Dreher M., Hornef M., Imöhl M., Kleines M. Comparison of four new commercial serologic assays for determination of SARS-CoV-2 IgG. J. Clin. Virol. 2020;128 doi: 10.1016/j.jcv.2020.104394. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Turbett S.E., Anahtar M., Dighe A.S., Garcia Beltran W., Miller T., Scott H., Durbin S.M., Bharadwaj M., Thomas J., Gogakos T.S., Astudillo M., Lennerz J., Rosenberg E.S., Branda J.A. Evaluation of three commercial SARS-CoV-2 serologic assays and their performance in two-test algorithms. J. Clin. Microbiol. 2020 doi: 10.1128/jcm.01892-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Hueston L., Kok J., Guibone A., McDonald D., Hone G., Goodwin J., Carter I., Basile K., Sandaradura I., Maddocks S., Sintchenko V., Gilroy N., Chen S., Dwyer D.E., O’Sullivan M.V.N. The antibody response to SARS-CoV-2 infection. Open Forum Infect. Dis. 2020;7 doi: 10.1093/ofid/ofaa387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Beaty B.J., Calisher C.H., Shope R.E. In: Diagnostic Procedures for Viral, Rickettsial and Chlamydial Infections. Lynette E.H., Lynette D.A., Lynette E.T., Schmidt N.J., Emmons R.W., editors. American Public Health Association; Washington DC: 1995. Arboviruses; pp. 189–212. [Google Scholar]
- 24.CDNA . 2020. Coronavirus Disease 2019 (COVID-19)https://www1.health.gov.au/internet/main/publishing.nsf/Content/cdna-song-novel-coronavirus.htm (Accessed 10 December 2020) [Google Scholar]
- 25.PHLN . 2020. PHLN Guidance on Laboratory Testing for SARS-CoV-2 (the Virus That Causes COVID-19)https://www.health.gov.au/sites/default/files/documents/2020/11/phln-guidance-on-laboratory-testing-for-sars-cov-2-the-virus-that-causes-covid-19.pdf [Google Scholar]
- 26.van Hal S.J., Wehrhahn M.C., Barbagiannakos T., Mercer J., Chen D., Paterson D.L., Gosbell I.B. Performance of various testing methodologies for detection of heteroresistant vancomycin-intermediate Staphylococcus aureus in bloodstream isolates. J. Clin. Microbiol. 2011;49:1489–1494. doi: 10.1128/JCM.02302-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Robin X., Turck N., Hainard A., Tiberti N., Lisacek F., Sanchez J.C., Müller M. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Gidding H.F.M.D., Hendry A.J., Quinn H.E., Vette K., Beard F.H., Shilling H., Hirani R., Gosbell I.B., Irving D.O., Hueston L., Downes M., Carlin J.B., O’Sullivan M.N.V., Dwyer D.E., Kaldor J.M., Macartney K. Seroprevalence of SARS-CoV-2-specific antibodies in Sydney, Australia following the first epidemic wave in 2020. Med. J. Aust. 2020 doi: 10.5694/mja2.50940. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.PHLN . 2020. Public Health Laboratory Network (PHLN) Guidance on Serological Testing in COVID-19.https://www.health.gov.au/resources/publications/phln-guidance-for-serological-testing-in-covid-19 [Google Scholar]
- 30.Ripperger T.J., Uhrlaub J.L., Watanabe M., Wong R., Castaneda Y., Pizzato H.A., Thompson M.R., Bradshaw C., Weinkauf C.C., Bime C., Erickson H.L., Knox K., Bixby B., Parthasarathy S., Chaudhary S., Natt B., Cristan E., El Aini T., Rischard F., Campion J., Chopra M., Insel M., Sam A., Knepler J.L., Capaldi A.P., Spier C.M., Dake M.D., Edwards T., Kaplan M.E., Scott S.J., Hypes C., Mosier J., Harris D.T., LaFleur B.J., Sprissler R., Nikolich-Žugich J., Bhattacharya D. Orthogonal SARS-CoV-2 serological assays enable surveillance of low-prevalence communities and reveal durable humoral immunity. Immunity. 2020;53 doi: 10.1016/j.immuni.2020.10.004. 925-933.e4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Jin J.M., Bai P., He W., Wu F., Liu X.F., Han D.M., Liu S., Yang J.K. Gender differences in patients with COVID-19: focus on severity and mortality. Front. Public Health. 2020;8:152. doi: 10.3389/fpubh.2020.00152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Gudbjartsson D.F., Helgason A., Jonsson H., Magnusson O.T., Melsted P., Norddahl G.L., Saemundsdottir J., Sigurdsson A., Sulem P., Agustsdottir A.B., Eiriksdottir B., Fridriksdottir R., Gardarsdottir E.E., Georgsson G., Gretarsdottir O.S., Gudmundsson K.R., Gunnarsdottir T.R., Gylfason A., Holm H., Jensson B.O., Jonasdottir A., Jonsson F., Josefsdottir K.S., Kristjansson T., Magnusdottir D.N., le Roux L., Sigmundsdottir G., Sveinbjornsson G., Sveinsdottir K.E., Sveinsdottir M., Thorarensen E.A., Thorbjornsson B., Löve A., Masson G., Jonsdottir I., Möller A.D., Gudnason T., Kristinsson K.G., Thorsteinsdottir U., Stefansson K. Spread of SARS-CoV-2 in the icelandic population. N. Engl. J. Med. 2020;382:2302–2315. doi: 10.1056/NEJMoa2006100. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Atyeo C., Fischinger S., Zohar T., Slein M.D., Burke J., Loos C., McCulloch D.J., Newman K.L., Wolf C., Yu J., Shuey K., Feldman J., Hauser B.M., Caradonna T., Schmidt A.G., Suscovich T.J., Linde C., Cai Y., Barouch D., Ryan E.T., Charles R.C., Lauffenburger D., Chu H., Alter G. Distinct early serological signatures track with SARS-CoV-2 survival. Immunity. 2020;53 doi: 10.1016/j.immuni.2020.07.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Röltgen K., Powell A.E., Wirz O.F., Stevens B.A., Hogan C.A., Najeeb J., Hunter M., Wang H., Sahoo M.K., Huang C., Yamamoto F., Manohar M., Manalac J., Otrelo-Cardoso A.R., Pham T.D., Rustagi A., Rogers A.J., Shah N.H., Blish C.A., Cochran J.R., Jardetzky T.S., Zehnder J.L., Wang T.T., Narasimhan B., Gombar S., Tibshirani R., Nadeau K.C., Kim P.S., Pinsky B.A., Boyd S.D. Defining the features and duration of antibody responses to SARS-CoV-2 infection associated with disease severity and outcome. Sci. Immunol. 2020:5. doi: 10.1126/sciimmunol.abe0240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Imai M., Iwatsuki-Horimoto K., Hatta M., Loeber S., Halfmann P.J., Nakajima N., Watanabe T., Ujie M., Takahashi K., Ito M., Yamada S., Fan S., Chiba S., Kuroda M., Guan L., Takada K., Armbrust T., Balogh A., Furusawa Y., Okuda M., Ueki H., Yasuhara A., Sakai-Tagawa Y., Lopes T.J.S., Kiso M., Yamayoshi S., Kinoshita N., Ohmagari N., Hattori S.I., Takeda M., Mitsuya H., Krammer F., Suzuki T., Kawaoka Y. Syrian hamsters as a small animal model for SARS-CoV-2 infection and countermeasure development. Proc. Natl. Acad. Sci. U. S. A. 2020;117:16587–16595. doi: 10.1073/pnas.2009799117. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.