Covid-19 antigen testing: better than we know? A test accuracy study

Abstract

Background

Antigen testing for SARS-CoV-2 is considered to be less sensitive than the standard reference method – real-time PCR (RT-PCR). It has been suggested that many patients with positive RT-PCR ‘missed’ by antigen testing might be non-infectious.

Methods

In a real-world high-throughput setting for asymptomatic or mildly symptomatic patients, 494 patients were tested using RT-PCR as well as a single lateral flow antigen test (Ecotest, AssureTech, China). Where the results differed, virus viability was evaluated by cell culture. The test parameters were calculated with RT-PCR and RT-PCR adjusted on viability as reference standards.

Results

The overall sensitivity of the used antigen test related to the RT-PCR only was 76.2%, specificity was 97.3%. However, 36 out of 39 patients ‘missed’ by the antigen test contained no viable virus. After adjusting on that, the sensitivity grew to 97.7% and, more importantly for disease control purposes, the negative predictive value reached 99.2%.

Conclusions

We propose that viability testing should be always performed when evaluating a new antigen test. A well-chosen and validated antigen test provides excellent results in identifying patients who are shedding viable virus (although some caveats still remain) in the real-world high-throughput setting of asymptomatic or mildly symptomatic individuals.

Introduction

The covid-19 pandemic has put the healthcare systems as well as the population worldwide under great stress. Accurate and timely identification of covid-19 cases is one of the cornerstones of successful management of the disease both at the level of individual patients and, more importantly, at the level of disease control and prevention of its spreading. At present, the bulk of the testing worldwide relies on real-time PCR (RT-PCR), although several countries, including, for example, Germany, France, or United Kingdom, have already adopted antigen testing in their strategies [1,2].

Antigen tests (AGTs) come with many advantages compared to RT-PCR. Most notably, the test can be performed directly at the point of care without the need for transporting samples to specialized laboratories, which is associated with the rapid availability of results (usually 10–30 min). They have also lower demands for instrumentation and personnel qualification and, usually, a lower price. On the other hand, these tests typically suffer from lower sensitivity compared to the standard method for SARS-CoV-2 detection, that is, RT-PCR, while the specificity is generally high. This field is rapidly evolving, with 56 antigen tests with a CE-marking listed in the FIND database by 11th November 2020 [1] but over 120 tests available by 23rd December 2020. Numerous studies on antigen testing compared to RT-PCR results using various tests on various populations were published with sensitivities ranging anywhere from 0 to 100%, reviewed, for example, by Dinnes et al. [3] or ECDC [1]. The majority of the tests, however, fell within the pooled sensitivity range between 70 and 90%.

The WHO guidance on the use of SARS-CoV-2 AGTs set the minimum performance requirements of 80% sensitivity and 97% specificity (ideally ≥99% specificity) compared to the RT-PCR reference assay [4]; the European Centre for Disease Prevention and Control (ECDC) agrees with these limits but emphasizes that the aim should be at least 90% sensitivity [1]. However, the character, in particular the infectivity, of samples, incorrectly classified by AGTs (mostly false negatives) is at present not known and can be only assumed and inferred from other research results.

It has been shown that the sensitivity of antigen tests is much lower for samples in which the RT-PCR cycle threshold (Ct) was ≥25 cycles and/or samples taken from patients (typically) after more than seven days since the symptom onset [5–7].

This corresponds very well with several studies on the infectivity and viability of the SARS-CoV-2 reporting that swabs that were taken later than on Day 7 [8] or Day 8 [9] after the onset of symptoms contained no viable virus. He et al. [10] concluded in their study that infectiousness is close to zero after 8 days from the symptom onset. However, patients with a severe course of the disease were infectious for a much longer period [11]. Low or nil virus viability was also associated with weak RT-PCR positivity (Ct ≥ 25) [9]. It should be noted that RT-PCR on itself cannot differentiate between dead viral particles and viable viruses. The above-mentioned facts, therefore, naturally lead to a question of whether samples that are ‘missed’ by AGTs (i.e. patients who are, if related to RT-PCR results, classified as ‘false negatives’) do contain any viable virus or not, as suggested, for example, by Corman et al. [12].

In this study, we evaluated one of the candidate AGTs considered by the Ministry of Health of the Czech Republic for inclusion into the high-throughput testing scheme. However, in addition to the standard testing requirements outlined by the Foundation for Innovative New Diagnostics (FIND) [13], we performed also an analysis of the virus viability (and, hence, infectiousness from the upper respiratory tract) in all samples where the RT-PCR and AGT results differed. We hypothesized that the tests missed by AGTs are to a great degree those that contain no viable virus and, in effect, the AGT false negatives originate from patients who are no longer infectious and do not pose a threat from the perspective of the virus spreading and disease control.

Materials and methods

Study group and sample collection

In this prospective study, the methods were compared in a real-world high-throughput testing setting. Consecutive individuals referred for testing by their physician, their local public health agency (due to manifesting covid-like symptoms or due to having been in contact with a Covid-19 patient) were admitted for testing. In effect, no individuals with a severe course of the disease (i.e. requiring hospitalization) at the time of testing were included in the study as such patients are not subject to high-throughput testing but rather to targeted in-hospital testing. The study was approved by the local Ethics Committee, No. NsPKar/19956/2020/SEK. All patients were informed about the study and signed an informed consent with participation.

Two nasopharyngeal swabs (one from each nostril) were taken from each patient by trained healthcare personnel. One swab was placed into the viral transport medium (D-MEM with 0.5% bovine serum albumin), the other into the medium supplied with the antigen test. The AGT was performed immediately, RT-PCR and (where needed) viability tests were performed after the transport of the samples into the laboratory. The tests were performed within 24 h of sampling; where this was not possible (viability testing) samples were frozen at −80 °C and thawed immediately before the analyses.

Antigen test

The antigen test used in this study was the ECOTEST Covid-19 Antigen Rapid Test (Assure Tech, Hangzhou, China) provided by the Ministry of Health, Czech Republic. The test was performed according to the manufacturer’s instructions.

RT-PCR

RT-PCR analysis was performed using the Automated RNA Isolation Kit for Agilent Bravo (Diana Biotechnologies, Czech Republic) and the RT-PCR detection kit COVID-19 Multiplex RT-PCR Kit (Diana Biotechnologies, Czech Republic) according to the manufacturer’s instructions. The isolation procedure results in ∼3× increase in RNA concentration, the detection is based on the proof of two SARS-CoV-2 genes, namely genes coding the Spike protein and EndoRNAse. The entire process was controlled through isolation and amplification of the synthetic internal control. Cycle thresholds (Ct) for both genes were recorded and the mean Ct from each sample was included in the analysis.

Virus viability testing

The viability of the virus was assessed only in samples where the result of the antigen test differed from that of RT-PCR. The test was performed in monolayer CV-1 cells (African green monkey kidney fibroblasts) cultured in the Leighton tubes. Cells cultured in the E-MEM medium (Sigma-Aldrich, USA) at 37 °C were inoculated with 300 µl of the sample used for the RT-PCR testing or blanks. The cultures were daily examined under the microscope (100× to 200× magnification) for morphological changes that would indicate a cytopathic effect of the virus present in the sample. After 7 days (or sooner if the cytopathic effect was observed in ∼75% of cells) the cells were passaged (1:6) and cultured for additional 7 days. If no virus action (cytopathic effect) was observed over the next 7 days, the sample was declared free of viable virus. Where cytopathic effect was observed, SARS-CoV-2 presence was verified through RT-PCR.

The method sensitivity was verified through a serial dilution of the virus stock suspension (3 × 10¹¹ RNA copies/ml), both without and after freezing at −80 °C. The cytopathic effect was observed from ∼10⁴ RNA copies/ml, which corresponds to 1–10 infectious particles/ml [14,15]. To verify that samples after freezing were not compromised, we performed (besides the above-described verification of the method sensitivity) also an analysis before and after freezing on 10 real-world samples with cycle thresholds 25–30 (5 samples) and 30–40 (5 samples). We recorded a 100% agreement between results before and after freezing.

Statistical analysis

Antigen test parameters (sensitivity, specificity, positive and negative predictive values, test accuracy, and positive and negative likelihood ratios) were calculated in Stata v.14 (https://www.stata.com/stata14/) and Clinical Calculator 1 (http://vassarstats.net/clin1.html). comparing the AGT results to the result of (a) RT-PCR only and (b) RT-PCR adjusted on the viability of discrepant samples (i.e. where RT-PCR test was positive but the virus viability test was negative, samples were considered negative). For detailed analysis, stratification according to Ct thresholds (<20, 20–25, 25–30, 30–40, RT-PCR-negative) and according to the presence or absence of symptoms was performed. 95% confidence intervals were calculated for all parameters.

Results

In all, 494 patients were examined in this study. The mean age was 42.2 ± 15.1 years (min. 7; max. 81). 297 (60.1%) patients were men, 197 (39.9%) women. 219 (44.8%) of patients had symptoms consistent with covid-19 (self-reported). In all, SARS-CoV-2 was detected in 164 patients (33.2%) by RT-PCR and in 134 patients (27.1%) by the AGT. The disagreement between RT-PCR and AGT results was recorded in 48 cases (AGT yielded 39 false negatives and 9 false positives when related to RT-PCR results). However, virus viability testing revealed that 36 out of those 39 false negatives contain no active virus.

Hence, we performed the adjustment of the reference value from ‘RT-PCR positive’ to ‘RT-PCR positive containing viable virus’ and calculated ‘adjusted’ test parameters. Table 1 shows the profound effect this change had on the test parameters. Underlying data for the calculations of test parameters detailed in Tables 1 and 2 can be found in Table S1.

Table 1.

Test parameters for the antigen test related to RT-PCR positivity and to the positivity in the combined endpoint of RT-PCR + virus viability as reference values; 95% confidence intervals are shown in brackets.

	RT-PCR	RT-PCR + viability
Sensitivity	76.2 (68.8–82.4)	97.7 (92.8–99.4)
Specificity	97.3 (94.7–98.7)	97.5 (95.2–98.8)
PPV	93.3 (87.3–96.7)	93.3 (98.3–96.7)
NPV	89.2 (85.4–92.1)	99.2 (97.4–99.8)
LR+	27.9 (14.6–53.5)	39.7 (20.8–75.8)
LR−	0.24 (0.19–0.32)	0.02 (0.01–0.07)
ACC	90.3 (87.3–92.7)	97.6 (96.2–98.9)

PPV: positive predictive value; NPV: negative predictive value; LR+: positive likelihood ratio; LR−: negative likelihood ratio; ACC: total accuracy.

Table 2.

Test parameters for antigen test related to RT-PCR positivity and to the combined endpoint of RT-PCR + virus viability for symptomatic and asymptomatic patients; 95% confidence intervals are shown in brackets.

	Symptomatic (n = 219)		Asymptomatic (n = 270)
	RT-PCR	RT-PCR + viability	RT-PCR	RT-PCR + viability
Sensitivity	82.2 (74.5–88.1)	99.1 (94.4–100.0)	46.4 (28.0–65.8)	86.7 (58.4–97.7)
Specificity	95.2 (87.6–98.5)	96.3 (90.1–98.8)	97.9 (95.0–99.2)	98.0 (95.2–99.3)
PPV	96.5 (90.8–98.9)	96.5 (90.8–98.9)	72.2 (46.3–89.3)	72.2 (46.4–89.3)
NPV	76.9 (67.4–84.4)	99.0 (94.0–99.9)	94.0 (90.2–96.5)	99.2 (96.9–99.9)
LR+	17.3 (6.6–45.1)	25.5 (10.1–69.4)	22.5 (8.6–58.4)	44.2 (18.1–107.7)
LR−	0.19 (0.13–0.27)	0.01 (0.00–0.07)	0.55 (0.39–0.77)	0.14 (0.04–0.49)
ACC	87.2 (82.1–91.3)	97.7 (95.7–99.7)	92.6 (88.8–95.4)	97.4 (95.5–99.3)

PPV: positive predictive value; NPV: negative predictive value; LR+: positive likelihood ratio; LR−: negative likelihood ratio; ACC: total accuracy.

Five patients did not state whether they were symptomatic or not in their questionnaires.

In addition, we have performed an analysis of the results separately for symptomatic and asymptomatic patients. These results are detailed in Table 2. A notable increase in the sensitivity and negative predictive value after adjustment on viability was observed in both symptomatic and asymptomatic patients. It is necessary to point out that only 28 asymptomatic patients were RT-PCR-positive; out of these 28 samples, 13 contained no viable virus, leaving only 15 patients with RT-PCR positivity and viable virus.

Table 3 shows the numbers of discrepant results stratified according to the Ct number. Again, we can see that following the adjustment on viability, most of the false-negative results disappeared. In the samples identified as AGT false positives when compared to RT-PCR, no viable virus was detected.

Table 3.

The total numbers of RT-PCR-positive patients according to the cycle thresholds (n), of RT-PCR-negative patients and of discrepant results in individual categories using both reference standards, that is, RT-PCR only and RT-PCR + virus viability.

Ct	n	Discrepant results, n (%)
		RT-PCR vs AGT	RT-PCR + viability vs AGT
0–19.99	8	0 (0%)	0 (0%)	False negatives of the antigen test
20–24.99	40	2 (5.0%)	2 (5.0%)
24.99–30	42	1 (2.4%)	0 (0%)
30–40	74	36 (48.7%)	1 (1.4%)
PCR-negative	327	9 (2.7%)	9 (2.7%)	False positives of the antigen test

Ct: cycle threshold; n: number of patients within the threshold.

Discussion

Overall results

Antigen testing is currently not considered a full substitute for RT-PCR. This is, in particular, due to the lower sensitivity reported in many studies [16–18], most recently reviewed in [1]. At the first glance, the sensitivity of 76.2% (95% CI 68.8–82.4%) when comparing AGT to RT-PCR according to the standard FIND methodology [13] is not particularly good for the purposes of disease control.

However, further analysis of the discrepant results revealed that 36 out of 39 (92.3%) of RT-PCR positive samples that passed undetected by the AGT did not contain any viable virus. Adjusting on this and considering the RT-PCR positive samples without any viable virus to be negative for disease control purposes (thus setting a new standard for calculation), the amended test parameter calculation yielded a very high sensitivity of 97.7% for detecting highly contagious patients with viable virus in the nasopharynx. This is already very close to the sensitivity of RT-PCR and suggests that for disease control purposes, the use of pre-validated antigen tests with high performance could be sufficient without the necessity of confirming the results by RT-PCR as proposed by ECDC [1].

The specificity of 97.5% is also excellent, although this is not a surprise as it is in line with results reported in many other studies [5,16,19]. However, the negative predictive value of 99.2% is, in the context of disease control, equally (if not even more) important. In other words, only less than 1% of patients who are evaluated as SARS-CoV-2 negative by the used AGT actually carry a viable virus in the nasopharynx and are, therefore, capable of transmitting the disease from the upper respiratory tract. It must be, however, noted that the absence of live virus in the upper respiratory tract has not yet been proven to rule out transmission [20]. This issue will be discussed in greater detail below.

Stratification by the presence of symptoms

We have also performed more detailed analyses, stratifying the patients according to the presence or absence of symptoms and the RT-PCR cycle threshold. Here, we can also observe the huge improvements in AGT sensitivities when the reference standard was changed into RT-PCR combined with virus viability. It is, however, necessary to take into account the low number of asymptomatic patients who met these criteria, which is also reflected in the wide 95% confidence intervals for sensitivity, positive predictive value, and other test parameters.

Stratification according to the cycle threshold

As can be seen from Table 3, most discrepancies between RT-PCR and AGT were detected in the samples with weak RT-PCR positivity (Ct 30–40 cycles). This is in agreement with many other studies, reviewed, for example, in ECDC [1]. High Ct values have been associated with limited viability, for example, by Bullard et al. [9] or Corman et al. [12]. Still, AGT was able to detect ∼50% of the RT-PCR positive patients in this group. Of those who were missed by the AGT, however, only one sample out of 36 contained the viable virus, which further supports the results of the aforementioned studies.

Practical implications

Now, let’s consider the implications for the evaluation of antigen tests, for the prevention of disease spreading, and for the number of patients who are isolated based on the RT-PCR test outcomes.

According to our results, we can state with a high degree of certainty that even in a high prevalence population, only a negligible number (less than 1%) of patients who were tested as negative by the used AGT have a viable (communicable) virus in the nasopharynx. The crucial question at this stage is: Are these patients infectious or not? This is something we can, at present, only assume from indirect evidence. Wölfel et al. [8] demonstrated that there is no difference in the RT-PCR loads between the throat swabs and nasopharyngeal swabs. It is, therefore, likely that there is no significant difference between the virus viability in both regions and, therefore, that such patients do not spread the virus during normal activities, sneezing, or mild cough originating in the upper respiratory tract. Of course, if the lower respiratory tract is affected, the situation can change significantly – in the same study, the authors also demonstrated that infectious virus persists in the sputum for much longer than in the upper respiratory tract. For this reason, we emphasize that we are still discussing the use of AGTs in a real-world high-throughput that is likely attended by individuals whose symptoms are (at the worst) mild. We cannot recommend the use of AGTs as a diagnostic tool for highly symptomatic patients showing signs of lower respiratory tract infection – for these patients, RT-PCR is definitely the method of choice (although AGT may still represent a valuable rapid tool for getting preliminary information). Anyway, in the group of asymptomatic individuals and mildly symptomatic patients, the antigen test evaluated in our study demonstrated excellent agreement with RT-PCR adjusted on viability, which is (although there are still many caveats and questions to be satisfactorily answered) likely to mean that the test was capable of very good recognition of infectious patients in that group.

We can also see that at least 36 out of 164 patients (22.0%) who were diagnosed as SARS-CoV-2 positive by RT-PCR but did not contain any viable virus and, in all likelihood, do not pose any threat of SARS-CoV-2 transmission (in other words, they could be considered for ‘RT-PCR false positives’ for disease control purposes). This would also mean that these patients (and their contacts) do not need to be isolated, which would have a highly beneficial effect on the society and economy. From this perspective, the AGT was much more successful in identifying patients with viable SARS-CoV-2, with only 9 out of 134 patients (6.7%) being falsely positive and only 3 out of 360 patients falsely negative. Nevertheless, to be able to state this with full certainty, this conclusion needs to be confirmed by additional studies comparing the relationship between the isolation of the viable virus in the nasopharynx and throat.

We would also like to propose a change in the AGTs validation and testing protocol set by FIND [13]. Namely, our results strongly indicate that at least the discrepant tests should be subjected to virus viability analysis to get a much better idea of the true risk posed to the population by the patients ‘missed’ by the particular AGT from the perspective of disease spreading. Of course, it would be optimal to analyze the virus viability in all samples used for validation; however, considering the laboriousness of that test and the fact that it can only be performed in a laboratory with a biosafety level 3 as a minimum, such testing would likely hamper the rate at which AGTs can be validated.

It is necessary to point out that our adjusted test parameters do not represent the ‘true’ test parameters as it is likely that if we tested all samples (including those where RT-PCR and AGT results are in agreement), some of those would be also reclassified. However, our intention is not to say that our approach provides 100% true results; we believe that our approach is superior to that of validating the tests against RT-PCR only.

The simple reduction of the cycle threshold for considering patients positive could be an alternative (and a much simpler one) to our approach; however, our results show that some positive samples can be present even at Ct > 30; moreover, we have only analyzed discrepant samples and it is possible (and, actually, given the good agreement between AGT testing and viability in discrepant samples, even likely) that cell culture would reveal additional samples with the viable virus even among samples with higher Ct.

In general, we fully agree with the concept proposed by Corman et al. [12] that antigen testing could play a crucial role in identifying patients who need to be isolated rather on the basis of infectivity than of the proof of the presence of the viral RNA.

Of course, due to the major differences reported for individual AGTs, it is necessary to select tests with sufficient performance only. It is, nevertheless, very likely that besides the test used in our study, other tests will also be able to demonstrate a test performance surpassing 97% sensitivity for detecting patients capable of transmitting the disease. In our opinion, such test parameters would be sufficient for the use of such tests in asymptomatic or mildly symptomatic patients without the necessity of confirmatory RT-PCR testing proposed by the ECDC [1].

Study limitations

This study is burdened with several limitations, the most notable of which is the assessment of viability only in nasopharyngeal swabs. As mentioned above, we suggest that a similar study should be performed with oropharyngeal swabs as well. Also, a better understanding of the infectivity of patients with lower respiratory tract symptoms is needed. On the other hand, we believe that our conclusions on infectivity are valid in the target group suitable for real-world high-throughput antigen testing or screening (i.e. patients with mild symptoms or asymptomatic patients who were in contact with a SARS-CoV-2 positive person or, in other words, in a population with a high prevalence).

We have neither evaluated the duration from the disease onset nor the type and severity of the symptoms in this study. Similarly, assessing the viability for all positive samples would help in the deeper understanding of the stratification and success of antigen (and RT-PCR) testing in individual groups; however, with almost 500 samples, this would prove extremely laborious and, would not be in direct relation to the principal aim of this study.

Conclusions

In this study, the results of an antigen test for detection of SARS-CoV-2 from nasopharyngeal swabs of asymptomatic or mildly symptomatic patients were compared to those of the reference standard RT-PCR. Besides the standard comparison, we also analyzed the viability of the virus. After adjusting on viability, the sensitivity and negative predictive value of the antigen test increased from 76% to 98% and from 89% to 99%, respectively. Although no definitive information about the infectivity of the patients from the throat oropharynx or lower respiratory tract is available at this time, it is likely that the used AGT has a great capacity for singling out infectious individuals during high throughput testing or screening in the group of asymptomatic patients who were in contact with a SARS-CoV-2 positive person or mildly symptomatic patients. Hence, we propose the following:

• Validation of antigen tests should take into account virus viability in addition to calculation solely based on RT-PCR results, at least for samples where RT-PCR and AGT results differ

• AGT using sufficiently validated tests is highly suitable for the detection of patients with a viable virus (infectious patients) in the high prevalence group of asymptomatic patients or patients with mild symptoms only, with excellent test parameters

• In our patient group, it appears that when deciding on the isolation of patients, the used (well-performing) antigen test could represent an excellent solution. In effect, it would in a way even outperform RT-PCR as RT-PCR detects also patients without viable virus who (if not having symptoms of the lower respiratory tract) are likely not infectious.

• Should our results be confirmed by other studies, ideally with the culture of all samples, the approach of isolating patients based on infectivity rather than on RT-PCR results could help in opening the society.

• On the other hand, we still consider RT-PCR to be the method of choice for patients with lower respiratory tract symptoms.

Funding Statement

This research was internally funded by the Hospital Karvina-Raj and the Public Health Institute Ostrava.

Author contributions

Miroslav Homza – conceptualization, supervision, funding acquisition, writing-review & editing; Hana Zelena – literature search, conceptualization, investigation, methodology, validation, data interpretation, writing-review & editing; Jaroslav Janosek – literature search, methodology, validation, data interpretation, writing-original draft, writing-review & editing; Hana Tomaskova – literature search, data analysis, data interpretation, validation, writing-review & editing; Eduard Jezo – conceptualization, project administration, funding acquisition, resources, writing-review & editing; Alena Kloudova – methodology, investigation, data analysis, writing-review & editing; Jakub Mrazek – methodology, investigation, data analysis, writing-review & editing; Zdenek Svagera – methodology, investigation, writing-review & editing; Roman Prymula – resources, literature search, supervision, writing-review & editing.

Disclosure statement

All Authors declare that they have no conflict of interest regarding the research presented in this paper

Data availability statement

A dataset containing relevant anonymized patient data (patient ID, RT-PCR results including cycle threshold where available, antigen testing results, presence/absence of symptoms and result of viability testing where available) is freely available from Figshare at: https://doi.org/10.6084/m9.figshare.13490292.v3