Abstract
Context.—
Severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) is a recently emerged, currently pandemic virus, and the etiological agent of coronavirus disease 2019 (COVID-19). Clinical testing for antibodies to SARS-CoV-2 has rapidly become widespread, but data regarding the inter-laboratory performance of these serologic assays are limited.
Objective.—
To describe the development and initial results of the College of American Pathologists (CAP) SARS-CoV-2 Serology Survey.
Design.—
Members from the CAP Microbiology and Diagnostic Immunology and Flow Cytometry Committees formed a working group to support development of a new proficiency testing survey for anti-SARS-CoV-2 antibody assays. Supplemental questions in the survey assessed the state of SARS-CoV-2 serologic testing among participating laboratories as of July 2020. Results were analyzed for agreement by immunoglobulin isotype tested, assay manufacturer, and methodology.
Results.—
A total of 4,125 qualitative results were received from 1,110 laboratories participating in the first survey. Qualitative agreement for assays measuring anti-SARS-CoV-2 total antibodies or IgG was greater than 90% for all three samples in the survey. Qualitative agreement for IgM and IgA for the negative sample was greater than 95%, but lacked consensus for the other two samples.
Conclusions.—
These initial data suggest overall excellent agreement and comparable performance for most qualitative anti-SARS-CoV-2 IgG and total antibody assays across all participating clinical laboratories, regardless of specific target antigen or assay methodology.
Introduction
The emergence of severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) was met by a rapid response from clinical laboratories and manufacturers of in vitro diagnostic assays. Detection of antibodies against SARS-CoV-2 antigens has become an important component in the fight against coronavirus disease 2019 (COVID-19), playing a role in seroprevalence studies, identifying therapeutic plasma units, assessing multisystem inflammatory syndrome in children and, in the future, potentially for monitoring vaccine responses1, 2. Many commercial assays and laboratory-developed tests have been designed that use different detection methodologies (e.g., lateral flow immunoassays, enzyme-linked immunosorbent assays [ELISAs], chemiluminescent immunoassays [CIAs], etc.) to measure antibody isotypes (i.e., discrete IgG, IgM, and/or IgA assays) or combinations of isotypes (i.e., total antibody assays). In addition, there is variation in the viral epitope used for antibody detection, with most assays targeting some portion of the SARS-CoV-2 spike (S) envelope glycoprotein or nucleocapsid (N) protein.
These variables in assay design suggest that there could be widespread discrepancies in test results between clinical laboratories. Although an abundance of published studies have compared performance characteristics of small numbers of individual assays3–7, there are limited data on the overall agreement of clinical SARS-CoV-2 serologic tests. In addition, the indications for use and performance practices of clinical laboratories offering SARS-CoV-2 serologic tests are not well defined.
Proficiency testing is a valuable component of clinical laboratories’ quality assurance programs and promotes reliability of patient test results. In proficiency testing programs, samples are blind-tested by participating laboratories and individual laboratory performance is compared to the collective performance of peer groups or all participants. Proficiency testing programs can reveal differences in result reporting between methods and manufacturers and support the ultimate goal of promoting standardization and harmonization efforts over time. As such, proficiency testing can play an important role in revealing variability in assay performance8.
In response to the growth in SARS-CoV-2 serologic testing, the College of American Pathologists (CAP) rapidly developed a proficiency testing program to support external quality assurance for clinical laboratories. Here, we report the overall agreement of results from laboratories participating in the initial CAP SARS-CoV-2 Serology Proficiency Testing Survey.
MATERIALS AND METHODS
Data were collected from the initial College of American Pathologists (CAP) SARS-CoV-2 Serology Survey (COVS-A 2020). Three individual samples, each from single donors, (two that pre-tested positive and one negative) were sent to 1,195 subscribing laboratories on the 22nd of June, 2020, along with kit instructions and the result reporting form. Each laboratory received a 0.5 mL aliquot for each sample, sent in an insulated container with a cool pack and instructions to store samples at 2 – 8°C until testing could be performed. Laboratories were instructed to perform serology testing using the methodology routinely performed on clinical specimens and report the results to the CAP by the 14th of July, 2020. Reporting fields for qualitative and quantitative (ie, numeric values such as signal-to-cutoff ratio or index value) results were available for total antibody, IgG, IgM and IgA. A reporting field for titer results was also available for each antibody class and instructions were given to use this reporting field only for neutralization assays. The result reporting form also included fields for method and manufacturer codes.
A supplemental questionnaire developed by the working group was also distributed with the COVS-A 2020 Survey. This questionnaire consisted of four questions (Supplemental Table 1) and was designed to assess the state of SARS-CoV-2 serology testing in participating laboratories.
Results were compiled and analyzed in Microsoft Excel (Redmond, WA). For results indicating the Manufacturer Code of “Other,” manufacturer was manually assigned using write-in responses and answers to supplemental questions. Reponses from 17 participants were recorded in the fields for “Total Antibody” but used manufacturers that only had commercially available assays for individual isotypes when the survey was distributed. For example, the only Abbott Diagnostics antibody assay available for SARS-CoV-2 in June 2020 was an IgG test, yet 7 laboratories listed Abbott for a “Total” antibody result. For our analysis, we manually assigned these responses to IgG. Chi-squared tests of association (SAS, Cary, NC) were performed to compare differences in the reported results for each sample across lateral/nonlateral flow methods and across manufacturer. Fisher’s Exact Test was used if any table cell had less than 5 results. Only those manufacturers with greater than 15 results per sample were tested.
Results
Proficiency Testing Survey Development and Responses
Development of a new proficiency testing survey is often a lengthy process (averaging about 18 months from conception to market availability) involving assessment of need, feasibility of obtaining appropriate material and anticipating and addressing potential issues surrounding testing, reporting and return of results. Early in the COVID-19 pandemic, clinical laboratories in the United States and worldwide quickly evaluated and validated SARS-CoV-2 serology assays and began reporting patient results. CAP responded by rapidly developing a proficiency testing program, and the product was available to participants within three months of initial conception. The proficiency testing program was developed with input from an ad hoc working group comprised of members of the CAP Microbiology and Diagnostic Immunology and Flow Cytometry Committees. These committees provide expert scientific support in diagnostic microbiology, immunology and flow cytometry, and define and monitor the state of the art and emerging technologies in these fields. These committees also support CAP proficiency testing surveys in their respective areas of expertise, including identifying opportunities for new surveys or analytes.
Candidate SARS-CoV-2 serology proficiency testing materials from two commercial reference material vendors were identified and distributed to CAP members’ laboratories for pilot testing using anti-SARS-CoV-2 antibody assays from the following manufacturers: Abbott, Beckman, Diasorin, Epitope Diagnostics, Euroimmun, Ortho and Roche, as well as a laboratory-developed test. Three samples (two that pre-tested positive, and one negative) from unique individuals were sent to subscribing laboratories in the COVS-A 2020 survey.
Responses from 1,110 laboratories were returned by the reporting deadline. The majority of the laboratories were located in the United States, but 115 (10.4%) were from other countries, including Canada (n=16, 1.4%), Japan (n=16, 1.4%), Singapore (n=14, 1.3%), the United Arab Emirates (n=8, 0.7%), and 31 other countries (2.8%). For each of the three samples, participants had the option to provide qualitative, quantitative, and titer results for Total Antibody, IgG, IgM and IgA. A total of 4,125 discrete qualitative results were received.
The kit instructions specified to report titer results only when using a neutralization assay. Only four of the 1,110 participants reported results for antibody titer; two reported titer results for total antibody and two for IgG. The limited number of laboratories reporting titer information precludes further analysis and indicates that at the time this survey was performed, very few clinical labs were reporting results from neutralization assays.
Laboratories Reporting Results for anti-SARS-CoV-2 Total Antibody Assays
Total antibody qualitative results from 387 laboratories were analyzed. Across all manufacturers, sample COVS-01 was reported as “detected” (positive) by 383 participants (99.0%), sample COVS-02 was reported as “detected” (positive) by 386 participants (99.7%) and sample COVS-03 was reported as “not detected” (negative) by 381 of 386 responding participants (98.7%). No laboratories reported “indeterminate / borderline” for any of the samples. These results suggest that overall agreement of total anti-SARS-CoV-2 antibody assays for these proficiency testing samples was excellent.
Results from individual manufacturers are summarized in Table 1. Roche was the most commonly used manufacturer, followed by Ortho and Siemens. A smaller number of laboratories reporting using assays from Bio-Rad, other manufacturers or laboratory developed tests. Manufacturers reported by less than 5 respondents and included in the “Other” category were Beckman, GenScript, NOWDiagnostics, PHASE Scientific, Wantai and Wondfo.
Table 1.
Qualitative anti-SARS-CoV-2 Total Antibody Results
| COVS-01 (n=387) | COVS-02 (n=387) | COVS-03 (n=386) | ||||
|---|---|---|---|---|---|---|
| Manufacturer | Detected (Positive) | Not Detected (Negative) | Detected (Positive) | Not Detected (Negative) | Detected (Positive) | Not Detected (Negative) |
| BIO-RAD | 10 [100%] | 10 [100%] | 10 [100%] | |||
| ORTHO | 87 [100%] | 86 [98.9%] | 1 [1.1%] | 2 [2.3%] | 84 [96.6%] | |
| ROCHE | 176 [100%] | 176 [100%] | 176 [100%] | |||
| SIEMENS | 83 [95.4%] | 4 [4.6%] | 87 [100%] | 2 [2.3%] | 85 [97.7%] | |
| LDT | 10 [100%] | 10 [100%] | 10 [100%] | |||
| OTHER* | 17 [100%] | 17 [100%] | 1 [5.9%] | 16 [94.1%] | ||
| TOTAL | 383 [99.0%] | 4 [1.0%] | 386 [99.7%] | 1 [0.3%] | 5 [1.3%] | 381 [98.7%] |
Other includes responses for manufacturers used by less than 5 respondents or when manufacturer was not specified. Abbreviation: LDT, Laboratory Developed Test
Laboratories Reporting Results for anti-SARS-CoV-2 IgG Assays
A total of 772 laboratories reported qualitative results for anti-SARS-CoV-2 IgG antibody assays. Across all manufacturers, sample COVS-01 was reported as “detected” (positive) by 701 participants (90.8%), “indeterminate / borderline” by 14 participants (1.8%), and “not detected” (negative) by 57 participants (7.4%). Sample COVS-02 was reported as “detected” (positive) by 769 participants (99.6%), “indeterminate / borderline” by one additional participant (0.1%) and “not detected” (negative) by only two laboratories (0.3% of participants). Sample COVS-03 was reported as “not detected” (negative) by 764 of 768 responding participants (99.5%), with only four laboratories (0.5%) reporting “detected” (positive). Together, these results suggest that overall agreement of anti-SARS-CoV-2 IgG antibody assays for these proficiency testing samples was also very good.
Anti-SARS-CoV-2 IgG antibody results from individual manufacturers are summarized in Table 2. Abbott was the most commonly used manufacturer, followed by Ortho, Diasorin, and Beckman. A smaller number of laboratories reported using assays from other manufacturers or laboratory developed tests. Manufacturers reported by less than 5 respondents and included in the “Other” category were ACROBiosystems, Ansh Labs, Artron Laboratories, Bio-Rad, BioMedomics, bioMérieux, Boston Molecules, Clarity, ET Healthcare, Meridian Bioscience, PerkinElmer, PHASE Scientific, Premier Biotech, Quansys Biosciences, Savyon Diagnostics, Ringbio, SD Biosensor, Siemens, SNIBE, TheraTest, UCP Biosciences, Viramed, Vircell and ZEUS Scientific.
Table 2.
Qualitative anti-SARS-CoV-2 IgG Results
| COVS-01 (n=772) | COVS-02 (n=772) | COVS-03 (n=768) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Manufacturer | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) |
| ABBOTT | 361 [99.2%] | 1 [0.3%] | 2 [0.5%] | 363 [99.7%] | 1 [0.3%] | 3 [0.8%] | 361 [99.2%] | ||
| AUTOBIO | 5 [45.5%] | 6 [54.5%] | 11 [100%] | 11 [100%] | |||||
| AYTU | 6 [100%] | 6 [100%] | 6 [100%] | ||||||
| BECKMAN | 54 [100%] | 53 [98.1%] | 1 [1.9%] | 1 [1.9%] | 53 [98.1%] | ||||
| CELLEX | 6 [50%] | 1 [8.3%] | 5 [41.7%] | 12 [100%] | 12 [100%] | ||||
| DIASORIN | 55 [100%] | 55 [100%] | 54 [100%] | ||||||
| DIAZYME | 17 [100%] | 17 [100%] | 17 [100%] | ||||||
| EPITOPE | 2 [13.3%] | 1 [6.7%] | 12 [80%] | 15 [100%] | 14 [100%] | ||||
| EUROIMMUN | 29 [96.7%] | 1 [3.3%] | 30 [100%] | 30 [100%] | |||||
| GOLD STANDARD | 3 [37.5%] | 4 [50%] | 1 [12.5%] | 8 [100%] | 8 [100%] | ||||
| HANGZHOU | 8 [100%] | 8 [100%] | 8 [100%] | ||||||
| HEALGEN | 29 [96.7%] | 1 [3.3%] | 29 [96.7%] | 1 [3.3%] | 30 [100%] | ||||
| ORIENT GENE | 5 [83.3%] | 1 [16.7%] | 6 [100%] | 6 [100%] | |||||
| ORTHO | 76 [100%] | 76 [100%] | 75 [100%] | ||||||
| SYNTRON | 5 [83.3%] | 1 [16.7%] | 6 [100%] | 6 [100%] | |||||
| LDT | 10 [71.4%] | 1 [7.1%] | 3 [21.4%] | 14 [100%] | 14 [100%] | ||||
| OTHER* | 47 [78.3%] | 4 [6.7%] | 9 [15%] | 60 [100%] | 59 [100%] | ||||
| TOTAL | 701 [90.8%] | 14 [1.8%] | 57 [7.4%] | 769 [99.6%] | 1 [0.1%] | 2 [0.3%] | 4 [0.5%] | 764 [99.5%] | |
Other includes responses for manufacturers used by less than 5 respondents or when manufacturer was not specified. Abbreviation: LDT, Laboratory Developed Test
Quantitative results (i.e., numerical values, such as signal-to-cutoff ratio or index value) for anti-SARS-CoV-2 IgG antibodies were reported by 113 laboratories. The mean and median numerical result for COVS-01 was consistently lower than that reported for COVS-02 for each manufacturer. This suggests that sample COVS-01 contained a lower level of anti-SARS-CoV-2 IgG antibodies than COVS-02, and the decreased performance in qualitative reporting for COVS-01 may be related to interpretive cutoff and / or analytic sensitivity. Although COVS-01 was reported as “detected” (positive) by over 90% of participants overall, laboratories using assays from Autobio, Cellex, Diazyme, Epitope, and laboratory developed tests were less likely to have detected anti-SARS-CoV-2 IgG antibodies in this sample (P < .001) and had more than one participant report this sample as “not detected” (negative).
Laboratories Reporting Results for anti-SARS-CoV-2 IgM and IgA Assays
Fewer participants reported results for anti-SARS-CoV-2 IgM and IgA assays compared to IgG and total antibody assays. Sample COVS-03 was reported as “not detected” (negative) for IgM by 188 of 190 participants (98.9%) and “not detected” (negative) for IgA by 26 of 27 participants (96.3%). This suggests that there were few false positive results with this proficiency testing sample in assays for anti-SARS-CoV-2 IgM and IgA.
In contrast, there was less consensus for the detection of anti-SARS-CoV-2 IgM and IgA antibodies. Of 191 responding participants, sample COVS-01 was reported as “detected” (positive) for IgM by 88 (46.1%) and “not detected” (negative) for IgM by 95 (49.7%). Sample COVS-02 was reported as “detected” (positive) for IgM by 39 participants (20.4%) and “not detected” (negative) for IgM by 144 participants (75.4%). Thus, IgM detection for these two samples was highly variable. Table 3 summarizes the anti-SARS-CoV-2 IgM antibody results from individual manufacturers. Manufacturers reported by less than 5 respondents and included in the “Other” category were ACROBiosystems, Ansh Labs, Artron Laboratories, BioMedomics, bioMérieux, Boston Molecules, Clarity, ET Healthcare, Meridian Bioscience, NBGS, PHASE Scientific, Premier Biotech, Savyon Diagnostics, Ringbio, SD Biosensor, SNIBE, TheraTest, UCP Biosciences, Viramed, Vircell and ZEUS Scientific.
Table 3.
Qualitative anti-SARS-CoV-2 IgM Results
| COVS-01 (n=191) | COVS-02 (n=191) | COVS-03 (n=190) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Manufacturer | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) |
| AUTOBIO | 11 [100%] | 11 [100%] | 11 [100%] | ||||||
| AYTU | 6 [100%] | 1 [16.7%] | 1 [16.7%] | 4 [66.7%] | 6 [100%] | ||||
| CELLEX | 3 [25%] | 1 [8.3%] | 8 [66.7%] | 1 [8.3%] | 11 [91.7%] | 12 [100%] | |||
| DIAZYME | 3 [17.6%] | 14 [82.4%] | 17 [100%] | 17 [100%] | |||||
| EPITOPE | 21 [100%] | 21 [100%] | 21 [100%] | ||||||
| GOLD STANDARD | 8 [100%] | 8 [100%] | 8 [100%] | ||||||
| HANGZHOU | 6 [75%] | 1 [12.5%] | 1 [12.5%] | 1 [12.5%] | 7 [87.5%] | 8 [100%] | |||
| HEALGEN | 23 [76.7%] | 2 [6.7%] | 5 [16.7%] | 12 [40.0%] | 2 [6.7%] | 16 [53.3%] | 30 [100%] | ||
| ORIENT GENE | 6 [100%] | 3 [50.0%] | 1 [16.7%] | 2 [33.3%] | 6 [100%] | ||||
| SYNTRON | 6 [100%] | 1 [16.7%] | 1 [16.7%] | 4 [66.7%] | 6 [100%] | ||||
| LDT | 9 [81.8%] | 2 [18.2%] | 2 [18.2%] | 1 [9.1%] | 8 [72.7%] | 11 [100%] | |||
| OTHER* | 26 [47.3%] | 4 [7.3%] | 25 [45.5%] | 19 [34.5%] | 1 [1.8%] | 35 [63.6%] | 1 [1.9%] | 1 [1.9%] | 52 [96.3%] |
| TOTAL | 88 [46.1%] | 8 [4.2%] | 95 [49.7%] | 39 [20.4%] | 8 [4.2%] | 144 [75.4%] | 1 [0.5%] | 1 [0.5%] | 188 [98.9%] |
Other includes responses for manufacturers used by less than 5 respondents or when manufacturer was not specified. Abbreviation: LDT, Laboratory Developed Test
Although the overall consensus between manufacturers for COVS-01 and COVS-02 was poor, users of some individual assays provided consistent results. For example, all participants using assays from Autobio, Epitope, and Gold Standard reported COVS-01 as “not detected” (negative), while all laboratories using assays from Aytu, Orient Gene, and Syntron reported COVS-01 as “detected” (positive). The differences between manufacturers was statistically significant for both COVS-01 and COVS-02 (P < .001). Thus, the lack of consensus for IgM detection most likely results from assay differences and not individual laboratory performance.
Similarly, sample COVS-01 was reported as “detected” (positive) for IgA by 17 of 27 participants (63.0%) and “not detected” (negative) for IgA by 8 participants (29.6%). Sample COVS-02 was reported as “detected” (positive) for IgA by 17 of 27 participants (63.0%) and “not detected” (negative) for IgA by 7 participants (25.9%). Table 4 illustrates that this lack of consensus results from differences between manufacturers, as all laboratories using Euroimmun assays reported “detected” (positive) results for COVS-01 and COVS-02 for anti-SARS-CoV-2 IgA antibodies, while none of the laboratories using the assay from Gold Standard reported these samples as positive. Together, these results suggest variability in clinical assays for detection of IgM and IgA antibodies to SARS-CoV-2.
Table 4.
Qualitative anti-SARS-CoV-2 IgA Results
| COVS-01 (n=27) | COVS-02 (n=27) | COVS-03 (n=27) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Manufacturer | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) |
| EUROIMMUN | 16 [100%] | 16 [100%] | 16 [100%] | ||||||
| GOLD STANDARD | 7 [100%] | 1 [14.3%] | 6 [85.7%] | 7 [100%] | |||||
| LDT | 1 [50.0%] | 1 [50.0%] | 1 [50.0%] | 1 [50.0%] | 2 [100%] | ||||
| OTHER* | 1 [50.0%] | 1 [50.0%] | 1 [50.0%] | 1 [50.0%] | 1 [50.0%] | 1 [50.0%] | |||
| TOTAL | 17 [63%] | 2 [7.4%] | 8 [29.6%] | 17 [63%] | 3 [11.1%] | 7 [25.9%] | 1 [3.7%] | 26 [96.3%] | |
Other includes responses for manufacturers used by less than 5 respondents or when manufacturer was not specified. Abbreviation: LDT, Laboratory Developed Test
Results from Lateral Flow Assays
Recent data have called into question the accuracy of anti-SARS-CoV-2 lateral flow antibody assays9, 10; thus, we sought to assess performance of lateral flow assays compared to other methodologies. Results were reported from 107 and 109 laboratories using lateral flow assays for anti-SARS-CoV-2 IgG and IgM, respectively. Although the overall consensus for COVS-01 IgG was greater than 90%, a reduced percentage of participants (82 of 107, 76.6%, P < .001) using lateral flow assays detected anti-SARS-CoV-2 IgG antibodies in this sample (Table 5). Detection of anti-SARS-CoV-2 IgG antibodies in the strongly positive COVS-02 sample was similar between laboratories using lateral flow (106 of 107, 99.1%) versus other methodologies (663 of 665, 99.7%). Detection of anti-SARS-CoV-2 IgM antibodies was more frequent using lateral flow assays than other methodologies for samples COVS-01 and COVS-02 (P < .001 for both). There were no false positive results for IgG or IgM antibodies reported for COVS-03 from participants using lateral flow devices. Only three participants provided results for total antibody assays using lateral flow methodology; the limited number of results precluded further comparison to non-lateral flow methodologies. Together, these data suggest that the lateral flow assays employed by participating laboratories may be less sensitive for the detection of anti-SARS-CoV-2 IgG antibodies compared to other methodologies, but are potentially more sensitive for detecting IgM.
Table 5.
Results from Lateral Flow Assays for anti-SARS-CoV-2 IgG and IgM
| COVS-01 | COVS-02 | COVS-03 | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Manufacturer | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) | Detected (Positive) | Intermediate / Borderline | Not Detected (Negative) |
| LATERAL FLOW IgG | 82 [76.6%] | 5 [4.7%] | 20 [18.7%] | 106 [99.1%] | 1 [0.9%] | 107 [100%] | |||
| NOT LATERAL FLOW IgG | 619 [93.1%] | 9 [1.4%] | 37 [5.6%] | 663 [99.7%] | 2 [0.3%] | 4 [0.6%] | 657 [99.4%] | ||
| LATERAL FLOW IgM | 68 [62.4%] | 8 [7.3%] | 33 [30.3%] | 33 [30.3%] | 6 [5.5%] | 70 [64.2%] | 109 [100%] | ||
| NOT LATERAL FLOW IgM | 20 [24.4%] | 62 [75.6%] | 6 [7.3%] | 2 [2.4%] | 74 [90.2%] | 1 [1.2%] | 1 [1.2%] | 79 [97.5%] | |
Participant Responses to the Supplemental Questions
The majority of laboratories reporting results (935 of 1,110; 84.2%) provided answers to the supplemental questionnaire that accompanied the survey. Question 1 was designed to assess indications for testing at performing laboratories. The most common responses were “diagnosis of past infection” (N=615) and “immune status check” (N=458). Many laboratories (N=129) responded “Other” – research, clinical trials, employee screening, in-development/validation, and “not yet determined” were listed among these. “Diagnosis of acute infection” was chosen by 175 respondents, with some also choosing “Other” and specifying use in symptomatic patients with negative SARS-CoV-2 PCR results.
Regarding Question 2 – monthly approximate volume of SARS-CoV-2 serology orders – there was broad variability in responses (Figure 1). While the majority of respondents (N=328) reported between 0 and 100 tests per month, 254 survey respondents provided no answer to Supplemental Question 2. When plotted as a Pareto Chart (Figure 1), 80% of survey respondents either reported 1,000 or less serology orders per month, or provided no volume information. However, volumes greater than 100,000 per month were reported by nine participants.
Figure 1. Supplemental Question 2 Responses.
Pareto Chart of Responses to Supplemental Question 2 on the COVS-A 2020 Proficiency Testing survey. Left y-axis, count of binned responses; right y-axis, cumulative percent of responses; x-axis, reported volume in bins.
Regarding Question 3 – how many different manufacturer kits were in use in participating laboratories – most responding laboratories reported using one or two discrete manufacturer’s assay(s). [TABLE 6] However, as many as 25 discrete in-use kits for SARS-CoV-2 antibody serology were reported by a single laboratory. This may present a challenge for future proficiency testing events, as regulations allow reporting of only one result per analyte.
TABLE 6 –
Numbers of Kits In-Use Reported by Respondents to COVS-A 2020 Supplemental Questionnaire
| Kits in Use | Count (%) |
|---|---|
| 0 | 17 (1.8) |
| 1 | 752 (80.4) |
| 2 | 133 (14.2) |
| 3 | 20 (2.1) |
| 4 | 6 (<1) |
| 5 | 1 (<1) |
| 6 | 4 (<1) |
| 7 | 1 (<1) |
| 25 | 1 (<1) |
| Total | 935 (100) |
Regarding Question 4 – which manufacturers’ kits were represented by respondents – the majority of respondents used kits sold by Abbott (N=353), Roche Diagnostics (N=169), and Ortho Clinical Diagnostics (N=125); each other manufacturer was named by less than 50 respondents.
DISCUSSION
The SARS-CoV-2 pandemic has placed clinical laboratory testing in the public eye and raised awareness of the importance for reproducible and reliable test results. Currently, over 50 SARS-CoV-2 serology assays have received emergency use authorization (EUA) from the Food and Drug Administration (FDA). This list may continue to change over time as new assays are authorized, while other authorizations are revoked. The wide range in methodology and variability of target antigen employed by these assays has raised concerns that inconsistent test performance may limit the overall utility of serology testing.
Here, we describe results from the first CAP SARS-CoV-2 serology proficiency testing survey, which included results from 1,110 individual participating laboratories. Overall, we found excellent qualitative result agreement (>90% consensus for all 3 samples) across all participating laboratories for assays that detect IgG and total antibodies against SARS-CoV-2. Consensus for IgM and IgA was very high for one sample, which nearly all participants reported as negative. However, for the two samples with consensus positive results using total antibody and IgG assays, IgM and IgA were variably detected.
Although the proficiency testing survey also included reporting fields for quantitative and titer results, these were reported by a smaller number of participants and values were not standardized across platforms. These factors limited our ability to analyze the results. At the time this study was performed, the majority of EUA authorized tests were qualitative rather than quantitative. Thus, although many assays produce a numeric value, the clinical utility of these values remains largely uncertain. Further study may reveal antibody threshold levels that correlate with protection via passive immunity (therapeutic plasma transfusion) or active immunity, thus providing rationale for utility of quantitative antibody assessment. Development of reference materials could support efforts to standardize and harmonize quantitative antibody measures.
A primary strength of our study is the large number of results gathered; these results represent clinical assay outputs from over 1,000 clinical laboratories around the world. Further, this large sample size represents diverse testing and detection methodologies, manufacturers, and target antigen epitopes. However, a notable limitation is that this proficiency testing survey included only three samples, and we cannot necessarily gauge assay performance based on this small number. In addition, the samples chosen for proficiency testing were pretested using assays from multiple manufacturers and exhibited good qualitative result agreement. This pretesting effort likely contributed to the generally high result consensus. Future surveys will continue to monitor performance of clinically utilized assays with proficiency testing samples.
The results from our supplemental questionnaire shed light on the current indications and volumes for clinical SARS-CoV-2 serology testing. We also characterized the state of testing in participating laboratories in June/July 2020, finding that most laboratories were performing ≤1,000 serological tests per month, using 1 or 2 manufacturers’ assays. Some of our respondents indicated use of serology testing for diagnosis of acute infection. We refer to guidelines from the Infectious Diseases Society of America and other authorities for emerging evidence regarding use of serology as an adjunct to nucleic acid amplification tests (NAAT) for diagnosis of acute disease2. These guidance documents also contain information about the necessary testing interval post-symptom onset to maximize clinical utility of anti-SARS-CoV-2 antibody assays.
Most respondents stated that the primary indications for serology testing at their sites were detection of past SARS-CoV-2 infection and/or to check immune status. Importantly, it remains unclear whether antibody test results correlate with protective immunity and how long potential immune-mediated protection lasts. The observation that severe and fatal COVID-19 can occur in the setting of high antibody measurements11–13 suggests that an antibody response alone may not be sufficient to prevent severe disease and/or death, at least in the acute setting. Further studies to define whether antibody detection predicts protection from re-infection after exposure, or primary infection in the setting of vaccine administration, will help to answer these questions. We predict that clinical testing patterns may be dynamic as these data emerge. Furthermore, as vaccine studies enroll increasing numbers of participants, the role of serologic testing may further shift. Our understanding of the clinical utility and relevance of testing for SARS-CoV-2 antibodies is likely to continue to develop in the coming months and years.
Supplementary Material
Acknowledgments
Elitza S. Theel reports participation on an Advisory Board for Roche Diagnostics, consulting for Accelerate Diagnostics and Siemens, and sponsored research funding from Roche Diagnostics and Ortho-Clinical Diagnostics. Daniel D. Rhoads performs sponsored research from IVD companies such as BD, Bio-Rad, and Roche; but he has received no funding for SARS-CoV-2 immunology studies. Susan L. Fink was supported by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health under awards K08AI119142, R21AI130281, R21AI131269 and R21AI153487. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
REFERENCES
- 1.Theel ES, Couturier MR, Filkins L et al. Application, Verification and Implementation of SARS-CoV-2 Serologic Assays with Emergency Use Authorization. J Clin Microbiol. 2020; 59(1):e02148–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Hanson KE, Caliendo AM, Arias CA et al. Infectious Diseases Society of America Guidelines on the Diagnosis of COVID-19:Serologic Testing. [Published online ahead of print, September 12, 2020] Clin Infect Dis. doi: 10.1093/cid/ciaa1343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Theel ES, Harring J, Hilgart H, Granger D. Performance Characteristics of Four High-Throughput Immunoassays for Detection of IgG Antibodies against SARS-CoV-2. J Clin Microbiol. 2020;58(8):e01243–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Jaaskelainen AJ, Kuivanen S, Kekalainen E et al. Performance of six SARS-CoV-2 immunoassays in comparison with microneutralisation. J Clin Virol. 2020;129:104512. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Bryan A, Pepper G, Wener MH et al. Performance Characteristics of the Abbott Architect SARS-CoV-2 IgG Assay and Seroprevalence in Boise, Idaho. J Clin Microbiol. 2020;58(8): e00941–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Schnurra C, Reiners N, Biemann R, Kaiser T, Trawinski H, Jassoy C. Comparison of the diagnostic sensitivity of SARS-CoV-2 nucleoprotein and glycoprotein-based antibody tests. J Clin Virol. 2020;129:104544. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Charlton CL, Kanji JN, Johal K et al. Evaluation of Six Commercial Mid- to High-Volume Antibody and Six Point-of-Care Lateral Flow Assays for Detection of SARS-CoV-2 Antibodies. J Clin Microbiol. 2020;58(10):e01361–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Knight V, Long T, Meng QH, Linden MA, Rhoads DD. Variability in the Laboratory Measurement of Cytokines: A Longitudinal Summary of a College of American Pathologists Proficiency Testing Survey. Arch Pathol Lab Med. 2020;144(10):1230–1233. [DOI] [PubMed] [Google Scholar]
- 9.Lisboa Bastos M, Tavaziva G, Abidi SK et al. Diagnostic accuracy of serological tests for covid-19: systematic review and meta-analysis. BMJ. 2020;370:m2516. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Deeks JJ, Dinnes J, Takwoingi Y et al. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Database Syst Rev. 2020;6:CD013652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Zhao J, Yuan Q, Wang H et al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin Infect Dis. 2020;71(16):2027–2034. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Long QX, Liu BZ, Deng HJ et al. Antibody responses to SARS-CoV-2 in patients with COVID-19. Nat Med. 2020;26(6):845–848. [DOI] [PubMed] [Google Scholar]
- 13.Seow J, Graham C, Merrick B et al. Longitudinal observation and decline of neutralizing antibody responses in the three months following SARS-CoV-2 infection in humans. Nat Microbiol. 2020;5(12):1598–1607. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.