Authorized SARS-CoV-2 molecular methods show wide variability in the limit of detection.

Abstract

On February 29^th, 2020, the U.S. Food and Drug Administration issued the first Emergency Use Authorization (EUA) for a SARS-CoV-2 assay outside of the U.S. Centers for Disease Control and Prevention. As of May 3^rd, 2021, 289 total EUAs have been granted. Like influenza, there is no standard for defining limit of detection (LoD), but rather guidance that analytical sensitivity/LoD be established as the level that gives a 95% detection rate in at least 20 replicates. Here we compare the performance characteristics of SARS-CoV-2 tests receiving EUA by standardizing sensitivity to a common unit of measure and assess the variability in LoD between tests. Additionally, we looked at factors that may impact sensitivities due to lack of standardization of the test development process and compare results for a standardized reference panel for comparative analysis within a subset of EUA tests offered by the U.S. Food and Drug Administration.

1. Introduction

Coronavirus disease 2019 (COVID-19) is caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and has resulted in more than 500 million cases and 6 million deaths as of May 2022 [1]. From the beginning of the pandemic, significant efforts have been directed toward diagnostic testing for SARS-CoV-2 to help prevent, mitigate, and respond to this deadly virus. On February 4^th, 2020, the U.S. Food and Drug Administration (FDA) issued an Emergency Use Authorization (EUA) for the Centers for Disease Control and Prevention (CDC) COVID RT-PCR diagnostic panel. In the absence of a public health emergency, Clinical Laboratory Improvement Amendments (CLIA) typically govern Laboratory Developed Tests (LDTs). However, the declaration of a public health emergency meant no SARS-CoV-2 tests could be offered without FDA EUA [2]. While full FDA approval of LDTs has been seen as a hindrance to test development due to time and expense, the process for FDA EUA is in place to provide resources for expedited release of tests under FDA guidance [3,4]. It is important to note that EUA is not a full FDA approval and authorization of these tests is discontinued when there is no longer a public health emergency. Despite differences between full FDA approval, EUA and CLIA regulated LDTs, the latter of which is regulated by the Centers for Medicare and Medicaid Services rather than the FDA, both require documentation of sensitivity/Limit of Detection (LoD) [5], [6], [7]. Though LoD documentation for these tests is required, there is no benchmark set for a minimum detectable level.

Despite these challenges, SARS-CoV-2 test development ensued at a rapid pace with the CDC offering RT-PCR-based test kits on Feb. 5^th, 2020. On Feb. 29^th, 2020, the FDA opened the EUA process due to issues with control material in kits provided by the CDC, which at the time had the only EUA, and the first diagnostic test EUA was granted by the FDA that same day [8]. As of May 3^rd, 2021, the total number of tests having received FDA EUA for SARS-CoV-2 molecular diagnostic testing reached 290 [9,10]. As the number of available diagnostic tests for the SARS-CoV-2 virus grew, their performance relative to one another became increasingly important for those seeking to use an existing EUA test. With so many differences in how LoD is established, it is difficult to accurately compare between tests. The FDA therefore created a reference panel to provide a comparative analysis between tests using a common material and unit of measure [11]. As part of the study, participants are asked to perform an LoD study with the supplied sample material. Unfortunately, the reference panel was designed for nucleic acid tests and did not include antigen-based testing. While it is widely held that antigen testing has lower sensitivity than nucleic acid tests due to the lack of target amplification, the EUA documentation provides a robust dataset for comparison of antigen as well as nucleic acid tests. Furthermore, the FDA reference panel data provide an opportunity to compare LoD not only between tests but also serve as a comparator within tests. The objectives of this study were to create a comprehensive list of EUA reported LoD, convert their results to a common unit of measurement, investigate factors that may impact those results such as sample type, and finally to investigate discrepancies between the LoD reported in EUA documentation and that reported for the FDA reference panel.

2. Materials and Methods

2.1. Study Design

The FDA requires all manufacturers to report test development documentation to their publicly available website (https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/in-vitro-diagnostics-euas-molecular-diagnostic-tests-sars-cov-2) and this site was utilized on May 3, 2021, to obtain data for this investigation. At that time, 290 assays were reported as having received EUA, and Supplemental Table S1 documents this study's data provenance for these assays. Limit of detection data were retrieved from these documents accessed between 5/3/2021 and 8/1/2021. In our subsequent analyses, 247 of the 290 EUA tests were considered. Exclusion criteria were: non-nucleic acid target Antigen test (n=11), reference to or license from an existing EUA (n=2), validation of a direct-to-consumer or home collection device on an existing EUA with (n=25), sample pooling for an existing EUA (n=3), duplicate study data reported or reference to a LoD that could not be converted to Genome Copy Equivalents per milliliter (GCE/mL)(n=2). Other information collected for this study included date of issue, LoD, material used for determination of LoD (i.e., live virus, inactivated virus, extracted RNA, synthetic transcript(s) or synthetic whole viral genome), test method (e.g., antigen, RT-PCR hydrolysis probe, RT-PCR with other detection methods such as CYBR Green, or other amplification method such as Loop-mediated isothermal amplification), sample type (swab/saliva), target gene/protein, and control type (human, internal control non-human origin or both). Additionally, LoD data from the FDA reference panel were downloaded from the FDA website and compared against LoD data provided in EUA documents.

2.2. LoD Value Conversion

To enable comparison between values listed as plaque forming units (PFU), tissue culture infective dose 50 (TCID₅₀), nucleic acid amplification test-detectable units (NDU) and GCE, all units of measure were converted to GCE/mL using the following conversions: 1 NDU/mL = 1 GCE/mL -with one target copy (NDU) the equivalent of 1 full genome copy, 1 TCID₅₀/mL = 10^3.8 GCE/mL based upon the mean log10 ratio of RNA copies/infectious units in humans and 1 PFU/mL = 9.01 × 10³ GCE/mL. -with PFU converted to TCID₅₀/mL through multiplication by 0.57 × 10^4.2 with 10^4.2 representing copy number across all cell types tested by Sendet et al. [12,13] In instances where LoD was reported as TCID₅₀/mL and material lot information was given, the corresponding certificate of analysis was downloaded, and, if present, GCE/mL was taken from that document.

2.3. Analysis

Analysis of variance (ANOVA) was used to compare LoD values between test parameters (e.g. type, number of targets, etc.) Tukey's Honest Significant Difference (HSD) test was then used to evaluate the difference in means between EUA listed LoD and that reported in the FDA reference panel testing. Lastly, for reference to the average and the +/- 1 standard deviation of viral load as reported by Jones et al, GCE/mL was estimated by taking the 1^st test mean viral load per swab (10^6.39) and dividing by either 2mL (10^6.09) or 4.3mL (10^5.76) and then taking the average of the two values to arrive at 10^5.95 GCE/mL. The same steps were repeated with values for the +/- 1 standard deviation values [14].

3. Results

3.1. Emergency use Authorization

February 1^st, 2020, through September 30^th saw an average of 26 EUAs per month while October 1^st, 2020, through May 1^st, 2021, saw an average of 10.25 EUAs per month (Fig. 1 ). Of the 247 SARS-CoV-2 EUAs included in this study, 230 (93%) were swab-based while 17 (7%) were saliva-based (Table 1 A). There were 197 (80%) RT-PCR tests using hydrolysis probes (e.g., TaqMan®) with the remaining 50 (20%) EUAs for nucleic acid testing split between those using RT-PCR but other signal detection methodologies (e.g., sequencing, target capture, mass spectrometry) or alternate amplification methods at 30 (12%) and 20 (8%) EUAs, respectively. The reported LoD values for all nucleic acid tests had a mean of 8840 copies per milliliter with a minimum LoD of 6.31 copies per milliliter and a maximum LOD of 630957 copies per milliliter (Table 1B). For reference, the LoD of the major commercial SARS-CoV-2 molecular methods as reported by Campbell and Binnicker, are shown in Table 1C [15]. The LoD of these tests were then compared with summary statistics and while there is high variability amongst tests (standard deviation: 43397 GCE/mL) this variability is not statistically significantly dependent on test type. Namely comparing the means of Nucleic Acid - RT PCR TaqMan®, Nucleic Acid - RT PCR Other, and Nucleic Acid – Other, showed no significant difference between the mean LoD for each group (ANOVA p-value: 0.55).

Fig. 1.

Plot showing new cases/10,000, total cases/1,000,000, tests receiving EUA and total EUA's issued/10 from 2/4/2020 to 5/3/2021.

Table 1.

A-Characteristics of 247 SARS-CoV-2 tests receiving EUA (Feb 2020 through May 2021). B- limit of detection by test type. C- LoD values of major commercial molecular assays.

A	Parameter		Variables		N (%)
	Sample collection type		Swab		230 (93%)
			Saliva		17 (7%)
	Analysis method		RT-PCR – Hydrolysis probes		197 (80%)
			RT-PCR – Other		30 (12%)
			Nucleic Acid - Other		20 (8%)
	Control type		Human Control		137 (55%)
			Internal Control (non-human origin)		101 (41%)
			Both		9 (4%)
	LoD sample material		Inactivated virus		70 (28%)
			Extracted RNA		62 (25%)
			Live virus		35 (14%)
			Synthetic transcripts		74 (30%)
			Synthetic whole viral genome		6 (2%)
B	LoD (GCE/mL)	All Nucleic Acid Tests	Nucleic Acid – Other	Nucleic Acid - RT PCR Other	Nucleic Acid - RT PCR TaqMan®	Antigen*
		(n=247)	(n=20)	(n=30)	(n=197)	(n=11)
	Min	6	6	109	9	189287
	Max	630957	77143	6310	901368	47763471
	Average	8840	10180	1323	10834	9319512
	Standard Dev.	43397	22879	1623	66148	17234960
	Median	1000	1150	720	1000	887126
	Mode	1000	125	1000	1000	N/A
C	Assay		LoD GCE/mL	Assay Type	Reference
	ID NOW COVID-19		125	Nucleic Acid - Other	https://www.fda.gov/media/136525/download
	Abbott RealTime SARS-CoV-2 assay		100	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136258/download
	BD SARS-CoV-2 Reagents for BD MAX System		640	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136816/download
	BioGX SARS-CoV-2 Reagents for BD MAX System		40	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136653/download
	BioFire COVID-19 Test		330	Nucleic Acid - RT PCR Other	https://www.fda.gov/media/136353/download
	Xpert Omni SARS-CoV-2		400	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/144033/download
	Xpert Xpress SARS-CoV-2 test		126	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136314/download
	Simplexa COVID-19 Direct assay		500	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136286/download
	ePlex SARS-CoV-2 Test		1000	Nucleic Acid - RT PCR Other	https://www.fda.gov/media/136282/download
	Aptima SARS-CoV-2 assay		83	Nucleic Acid - Other	https://www.fda.gov/media/138096/download
	Panther Fusion SARS-CoV-2 Assay		74	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136156/download
	ARIES SARS-CoV-2 Assay		333	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136693/download
	Cobas SARS-CoV-2		66	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136049/download
	Amplitude Solution with the TaqPath COVID-19 High-Throughput Combo Kit		250	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/147548/download
	TaqPath COVID-19 Combo Kit		2000	Nucleic Acid - RT PCR TaqMan®	https://www.fda.gov/media/136112/download

3.2. Variables in Test Development

Next, the number of targets (i.e., number of viral genomic regions evaluated), control type, and LoD sample type were evaluated. Interestingly, there is no requirement to disclose the exact viral genomic location of the target, though some EUA submissions chose to do so. Due to the lack of exact target location, we were unable to assess if the genetic target location plays a role in test sensitivity though we evaluated whether the number of targets impacted sensitivity. None of the parameters evaluated including number of targets, material used for LoD (live virus, inactivated virus, extracted RNA, synthetic transcript(s) or synthetic whole viral genome), sample type (swab/saliva), control type (human, internal control non-human origin or both) demonstrated a significant difference in mean sensitivity with ANOVA p-values of 0.91, 0.18, 0.59 and0.46 respectively.

3.3. FDA reference Panel

We then compared the reported means and the differences between EUA LoD and the reference panel LoD to investigate sources of intra-test variability. Of the 247 tests evaluated in this study, 130 (63.1%) out of the 206 developers of SARS-CoV-2 nucleic acid tests contacted by the FDA took part in this challenge. There was a significant difference between the mean LoD values reported in the EUA documentation (mean = 9417 copies/mL) with those reported from the reference panel (mean = 43750 copies/mL) (P <0.0001) (Fig. 2 ). LoD was similar across methods, number of targets, type of material (i.e., live virus, inactivated virus, extracted RNA, synthetic transcript(s), or synthetic whole viral genome), and the control type (human, internal control non-human origin or both) when using the reference panel, in agreement with our initial findings. The parameters mentioned above did not statistically significantly contribute to the observed differences (Tukey's HSD p-value: 0.54, 0.49, 0.65 and 0.63 for method, number of targets, type of material and control type respectively). Thus, the reason for the difference between EUA and reference panel remains unexplained.

Fig. 2.

Violin plots of EUA and FDA reference panel LoD p-value<0.0001

3.4. Intra-test Analysis

Finally, a comparison of the LoD values reported in EUA to those reported in the reference panel within the same test was performed (Fig. 3 A). In almost all cases, the LoD for the reference panel was higher than the LoD for the EUA, and a paired t-test showed this trend was statistically significant (p-value: <0.0001). Additionally, we looked at how these values compare against previously reported data [14]. In their study of presymptomatic, asymptomatic, and mildly symptomatic SARS-CoV-2 infections, Jones et al estimated viral loads, via a Bayesian regression, of individuals’ first positive test result. When looking at EUA LoD alone, no test had a higher LoD than the mean first test value of 8.91 × 10⁵ GCE/mL, while 20 had a higher LoD than the mean 1^st test value minus 1 standard deviation, estimated for this study to be 1.33 × 10³ GCE/mL, suggesting that ∼8.1% of positive tests observed in that study would be missed by these tests (Fig. 3B). For tests involved in the FDA reference panel, each reported LoD (EUA or FDA reference panel) was below the mean 1^st test viral load reported by Jones et al; however, 52 (40%) of these devices (Fig. 3C) had a reported FDA reference panel LoD above the mean minus one standard deviation value again suggesting that ∼16% of positive tests observed in the Jones et al study would be missed by these tests.

Fig. 3.

Reported LoD in EUA and FDA reference panel. A- Tests were numbered 1-130 from lowest to highest EUA LoD (x-axis) with GCE/mL as the y-axis. LoD values reported in EUA are plotted in blue, LoD values reported in FDA Reference Panel are plotted in orange. B&C- EUA LoD relative to 1^st test viral load. Gray and gold represent first test mean viral load and mean-1 standard deviation viral load as calculated from Jones et al. B- LoD values as reported in EUA. Tests were numbered 1-247 from lowest to highest EUA LoD (x-axis) with GCE/mL as the y-axis. C- LoD values as reported in FDA reference panel results. Tests were numbered 1-130 from lowest to highest by FDA reference panel LoD (x-axis) with NDU/mL as the y-axis.

4. Discussion

Our data indicate that new test EUAs resulted in a plethora of new tests (n=290) for the detection of SARS-Co-V2. The data show extensive variability in their reported sensitivities (LoD) of nucleic acid tests. While antigen tests were excluded from this study due to the limited number of EUA-approved tests, the reported values (Table 1B) are consistent with previous studies with the average LoD s being approximately 10³-times less sensitive for these devices though not to the extent (10⁵-times less sensitive) found in the Mak study [16,17].

The mean LoD of tests did not show a significant difference when assessed by the material used for LoD, sample type, number of targets, or control type. For those tests that engaged in FDA reference panel testing, the mean LoD reported for that study was significantly different than the mean of LoD reported in EUAs. Lastly, while most of these tests are sensitive enough to detect SARS-CoV-2 at levels seen by Jones et al based upon their EUA reported LoD, 40% of the tests performing reference panel testing reported an LoD higher than 1.33 × 10³ GCE/mL, which is one standard deviation below the mean first test viral load. It should be noted that, in their study, Jones et al calculated viral load based upon cycle threshold (Ct) value which provides a vague estimate of viral load and these formulae are not applicable between laboratories. These findings are of note considering that viral load and disease severity have been shown to be unrelated and that patients with low viral loads are capable of shedding active viruses [18], [19], [20].

We observed a new test EUA rate of 26 EUAs per month through October 2020 followed by a halving of that rate through the remainder of this study. While this slowing of new EUAs overlapped with the largest spike in new cases in the US, it is important to note that this is not a count of total SARS-CoV-2 tests performed nor a representation of total sites performing such testing. Additionally, this decrease in the pace of EUAs may be attributed to efforts to increase testing capacity, through the licensing of tests already granted EUA, sample pooling, or increasing access to testing through at-home collection. Several of the EUAs omitted from this study were from new testing facilities licensing existing EUAs from manufacturers or validation of the use of at-home collection devices. Additionally, other EUAs omitted from this study were for sample pooling to increase the throughput of an existing EUA – these were omitted as samples from a pool testing positive were then run individually.

When looking at the current gold standard method RT-PCR, these assays have demonstrated the ability to detect the virus at levels lower than 100 copies per milliliter of transport media [21]. Our investigation was challenged by a lack of a common unit of measurement. Given the difference in analytes between antigen and nucleic acid tests this could be anticipated, since published materials for antigen tests often list the sensitivity of the device in nanograms per microliter of a control peptide rather than reporting viral copy number [16,22,23]. Additionally, values such as Tissue Culture Infectious Dose 50, Plaque Forming Units, Nucleic Acid Detectable Units or Genome Copy Equivalents are not necessarily interchangeable and necessitate calculations to arrive at a common unit of measure. This is similar to FDA approval for influenza, where no set unit of measurement is required and thus can be reported as Tissue Culture Infectious Dose₅₀, Plaque Forming Units, Nucleic Acid Detectable Units, or Genome Copy Equivalents [24]. There is also no set minimum for LoD of the influenza virus but rather guidance that analytical sensitivity/limit of detection be reported as the level at which gives a 95% detection rate confirmed in at least 20 replicates. While influenza material is readily available and a list of at least 28 strains is provided to developers, SARS-CoV-2 test developers can choose the preferred material from which to perform their LoD study thus adding another variable to consider when comparing tests.

As suggested in the background for the FDA reference panel, a lack of a common testing material can further convolute comparison, though we were unable to demonstrate this as a reason for the significant difference in LoD values between EUA and the FDA reference panel. Likewise, none of the parameters investigated for this study were able to elucidate the variability in the tests’ sensitivities. Regarding the intra-test variability between EUA listed LoD and that reported for the FDA reference panel, one manufacturer did state in their EUA documentation that the matrix used for the FDA reference panel, Minimal Essential Media -the only transport media used for the reference panel study, was not evaluated as part of their interfering substances study and may lead to a decrease in sensitivity. While this may be the case and matrix effects cannot be overlooked, it is certainly interesting to note that, based upon LoD values returned from the FDA reference panel, 52 of the 130 (40%) devices would not be able to detect positive samples from samples outside the -1SD of the mean 1^st test viral load, which was estimated here to be 1.33 × 10³ GCE/mL. Further research using a more standardized testing material with a common unit of measurement would help compare test sensitivity amongst SARS-CoV-2 tests.

Even in the absence of set LoD minimums for test approval, agreement on a standard sample type and matrix as well as a common unit of measurement would allow for greater confidence in comparison of tests for SARS-CoV-2. While this alone is not likely to alleviate the wide variability in LoD reported here, having a direct comparison between tests can help guide decisions surrounding which test to choose. While it may be difficult to imagine a more rapid and coordinated response towards test development given the demand for such testing was higher than ever before, a push towards the standardization of how LoD is reported and improvements in the process for providing a consistent testing material (even if it is suboptimal e.g. synthetic transcripts in place of live virus, or a less common sample media such as minimal essential media) will help to better gauge test performance between tests. Further study is needed to make direct comparisons between these devices or to determine the impacts of reduced test sensitivity on clinical practice or public health.

5. Limitations

This study was limited to information available through the FDA website https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/in-vitro-diagnostics-euas-molecular-diagnostic-tests-sars-cov-2 for the devices listed in supplemental table s1. These values represent a snapshot in time over the course of the pandemic that continues to evolve.

Authors' Contribution

B.J.H., Conceptualization, Methodology, Formal analysis, Investigation, Data curation, Writing - original draft; J.G., Formal analysis, Writing - review & editing; B.M., Writing - review & editing; K.B.S., Writing - review & editing; B.L., Writing - review & editing; I.D.S., Writing - review & editing; S.S.M., Writing - review & editing; K.B.R., Writing - review & editing

Declaration of Competing Interest

M.J. Binnicker is a scientific advisory board member for DiaSorin Molecular and Mammoth Biosciences.