Field evaluation of the performance of a SARS-CoV-2 antigen rapid diagnostic test in Uganda using nasopharyngeal samples

Abstract

Objectives

There is a high demand for SARS-CoV-2 testing to identify COVID-19 cases. Real-time quantitative PCR (qRT-PCR) is the recommended diagnostic test but a number of constraints prevent its widespread implementation, including cost. The aim of this study was to evaluate a low cost and easy to use rapid antigen test for diagnosing COVID-19 at the point of care.

Methods

Nasopharyngeal swabs from suspected COVID-19 cases and low-risk volunteers were tested with the STANDARD Q COVID-19 Ag Test and the results were compared with the qRT-PCR results.

Results

In total, 262 samples were collected, including 90 qRT-PCR positives. The majority of samples were from males (89%) with a mean age of 34 years and only 13 (14%) of the positives were mildly symptomatic. The sensitivity and specificity of the antigen test were 70.0% (95% confidence interval (CI): 60–79) and 92% (95% CI: 87–96), respectively, and the diagnostic accuracy was 84% (95% CI: 79–88). The antigen test was more likely to be positive for samples with qRT-PCR Ct values ≤29, with a sensitivity of 92%.

Conclusions

The STANDARD Q COVID-19 Ag Test performed less than optimally in this evaluation. However, the test may still have an important role to play early in infection when timely access to molecular testing is not available but the results should be confirmed by qRT-PCR.

Background

In the absence of treatment for coronavirus disease 2019 (COVID-19) or a vaccine for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), public health experts recommend speedy and accurate testing, followed by case identification, isolation, and contact tracing as the best approaches to contain this new disease. COVID-19 was declared a pandemic by the World Health Organization (WHO) on March 11, 2020. Identifying people infected with the virus is complicated by the high overlap between the clinical symptoms of COVID-19 and those of other respiratory infections, and by the fact that many infected individuals are asymptomatic (Yang et al., 2020, Lavezzo et al., 2020). Therefore, accurate diagnostic testing for case identification, quarantine, and contact tracing is essential for managing this pandemic.

The global demand for testing has put a substantial strain on governments and institutions. The gold standard diagnostic test recommended by the WHO (WHO, 2020a) is real-time quantitative reverse transcription PCR (qRT-PCR), where the viral RNA is detected by nucleic acid amplification testing.

In Uganda, SARS-CoV-2 qRT-PCR testing started at the Uganda Virus Research Institute (UVRI) in early February 2020 using the Berlin protocol (Corman et al., 2020). All testing in Uganda was performed at the UVRI for the initial two months before decentralization. The first confirmed case was detected on March 21, 2020. By mid-July, Uganda had conducted 230,680 tests, which was the highest number in East Africa, and 1040 COVID-19 infections were reported. The majority of these infections were imported by truck drivers at the border crossings with neighboring countries. These point-of-entry cases require rapid result turnaround times to ensure early interventions and avoid the transmission of infection to local communities as well as limiting the disruption of commercial transportation.

qRT-PCR testing is usually performed in designated, specialized laboratories that require well-trained staff and the test typically takes 4–6 h to complete. The time required to ship clinical samples to the laboratory and then return the results to health facilities leads to an overall turnaround time of 24–48 h. Thus, prolonged periods of isolation are required for suspected cases and delayed contact tracing could further increase the spread of the infection within the country. The cost of qRT-PCR is also prohibitive and further complicated by global procurement challenges, where most reagents currently take 1–2 months for delivery. To overcome these constraints, simple, low cost, and easy-to-use rapid antigen diagnostic tests are urgently required at the point of care.

Several antibody and antigen rapid diagnostic tests (RDTs) that can be used by staff with minimal training are now available on the market and they are attractive options for the decentralization of testing. Antibody tests are used for surveillance and epidemiological research, and to identify recent or past infections, whereas antigen RDTs could be an alternative to qRT-PCR for detecting acute infection. Moreover, in recent guidance, the WHO proposed settings where antigen RDTs can be used in patient management (WHO, 2020b). However, there have been few field evaluations of these antigen RDTs (Scohy et al., 2020, Porte et al., 2020, Lambert-Niclot et al., 2020, Mertens et al., 2020, Diao et al., 2020, Mak et al., 2020, ICMR, 2020) and those evaluated yielded disappointing results in terms of their performance.

In Uganda, all new molecular testing kits and immunoassays that are introduced into the market must undergo in-country laboratory verification at UVRI, which is a designated WHO and Africa CDC SARS-CoV-2 reference laboratory, before being recommended to the Ministry of Health for use in the country. The WHO also advises that tests should be verified in appropriate populations and settings before they are recommended (WHO Bulletin, 2017, ECDC, 2020).

No immuno-assays have been recommended for use in Uganda. In the present study, we evaluated the performance characteristics of the STANDARD Q COVID-19 Ag Test (SD Biosensor, Gyeonggi-do, 16690, Korea) compared with qRT-PCR (Berlin protocol) using nasopharyngeal swabs. Our results may be useful for other reference laboratories in order to meet the demand for SARS-CoV-2 diagnosis.

Materials and methods

Study design and participants

In this study, we conducted a cross-sectional, prospective, un-blinded verification of the performance of the STANDARD Q COVID-19 Ag Test. Participants were recruited at the regional referral hospitals (RRHs) in Arua, Entebbe, Fort Portal, Gulu, Jinja, Lira, Masaka, Mbale, and Mulago National Referral Hospital. The non-case controls were volunteers at Kasenyi Military Barracks and the UVRI clinic (Table A1 ).

Table A1.

Participating sites and samples collected.

Facility	Number of participants	Proportion (%)
Arua RRH	8	3.1
Entebbe RRH	54	20.6
Fort Portal RRH	1	0.4
Gulu RRH	11	4.2
Jinja RRH	2	0.8
Kasenyi Military Barracks	91	34.7
Lira RRH	8	3.1
Masaka RRH	20	7.6
Mbale RRH	4	1.5
Mulago NRH	30	11.5
UVRI Clinic	33	12.6
Total	262	100

Sample collection

Sample collection was coordinated by a team of laboratory staff from UVRI. Two nasal swabs were collected by laboratory personnel at the COVID-19 treatment facilities while taking all necessary biosafety precautions, where a swab was collected from each nostril. One swab was tested immediately at the facility using the STANDARD Q COVID-19 Ag Test, and the result was interpreted according to the manufacturer’s guidelines and recorded in an Excel worksheet. The second swab was preserved in specimen transport medium and transported at 4 °C to the UVRI laboratory for extraction and qRT-PCR testing (Corman et al., 2020).

All testing followed the procedures described in the Uganda National Council for Science and Technology (UNCST) approved protocol “Uganda Virus Research Institute; Performance Evaluation for CVID-19 Diagnostic Tests” (http://www.uvri.go.ug/projects/covid-19).

Antigen test

The STANDARD Q COVID-19 Ag Test is a rapid chromatographic immunoassay for the qualitative detection of SARS-CoV-2 specific antigens present in the human nasopharynx. According to the manufacturer's “Information for Use” (IFU), the results are available within 30 min. All necessary reagents to perform the assay are provided by the manufacturer and no assay-specific, specialized equipment is needed. According to the IFU, the assay kits are stable when stored at 2–30 °C.

qRT-PCR

RNA extraction

RNA was extracted from clinical samples using a viral RNA mini kit (QIAGEN, Hilden, Germany).

qRT-PCR (Berlin Protocol)

Oligonucleotides were synthesized and provided by Metabion (http://www.metabion.com). Thermal cycling was performed at 55 °C for 10 min for reverse transcription, followed by 95 °C for 3 min and then 45 cycles at 95 °C for 15 s and at 58 °C for 30 s. qRT-PCR was performed using an Applied Biosystems PCR platform.

Statistical methods

Sensitivity

The sensitivity was calculated as the number of specimens identified as positive by the STANDARD Q COVID-19 Ag test divided by the number of specimens identified as positive by the qRT-PCR reference assay, and expressed as a percentage.

Specificity

The specificity was calculated as the number of specimens identified as negative by the STANDARD Q COVID-19 Ag test divided by the number of specimens identified as negative by the qRT-PCR reference assay, and expressed as a percentage.

Accuracy

The accuracy was calculated as the proportion of STANDARD Q COVID-19 Ag Test results that agreed with the qRT-PCR results (positive and negative), and expressed as a percentage.

The sensitivity, specificity, and accuracy calculations were performed using the proportion command in STATA® 15, which also generated the 95% confidence intervals (CIs).

Comparison of cycle threshold (Ct) values with antigen assay results

We determined the relationship between the viral load measured as the qRT-PCR Ct value and antigen detection. Ct values were categorized as strongly positive (Ct ≤ 29) indicating abundant target nucleic acid in the sample, moderately positive (Ct = 30–37), and weakly positive (Ct = 38–39), and compared with the STANDARD Q COVID-19 Ag Test results.

Ethical considerations

The evaluation protocol was reviewed and approved by UVRI’s Research Ethics Committee and the UNCST. Specimens were unlinked to personal identifiers and results could not be traced to individual patients. Consent to participate and to store samples for future use was also sought.

Results

Demographic characteristics

In total, 90 COVID-19 cases and 172 controls (total 262) were included in this evaluation. The majority were males (89%) and the overall mean age was 34 years (95% CI: 32–35 years; Table 1 ).

Table 1.

Demographic characteristics of study participants.

Participants	Mean Age (95% CI)	Gender		Total
		Males (%)	Females (%)
Cases	37 (35–39)	85 (94)	5 (6)	90 (34)
Not Cases	32 (31–34)	149 (87)	23 (13)	172 (66)
Total	34 (32–35)	234 (89)	28 (11)	262

Performance characteristics

Test results distribution

The distribution of the STANDARD Q COVID-19 Ag Test results versus the qRT-PCR results is presented in Table 2 . Overall, 76 (29.0%) specimens were antigen-positive and 186 (71%) specimens were antigen-negative.

Table 2.

STANDARD Q COVID-19 Ag test results compared with the qRT-PCR reference assays.

STANDARD Q COVID-19 Ag test result	qRT-PCR test result		Total
	Positive	Negative
Positive	63 (83)	13 (17)	76 (29)
Negative	27 (15)	159 (86)	186 (71)
Total	90 (34)	172 (66)	262 (100)

Sensitivity and specificity

The sensitivity of the STANDARD Q COVID-19 Ag Test was 70% (95% CI: 60–79%) and the specificity was 92% (95% CI: 87–96%).

Accuracy, false positive rate, and false negative rate

The accuracy of the STANDARD Q COVID-19 Ag Test was 84% (95% CI: 79–88%). The false positive rate was 8% (95% CI: 4–13%) and the false negative rate was 30% (95% CI: 21–40%). No factors were associated with false positives, and they found in all age categories and RRHs.

Associations between Ct values and STANDARD Q COVID-19 Ag Test results

The associations between the Ct values and STANDARD Q COVID-19 Ag test results are shown in Table 3 . As expected, the STANDARD Q COVID-19 Ag Test result was more likely to be positive when a specimen contained abundant target nucleic acid, where 92% of the specimens with a strong positive qRT-PCR result were positive according to the STANDARD Q COVID-19 Ag test. Only 50% of the specimens categorized as moderately and weakly positive samples by qRT–PCR were positive with the antigen test (p < 0.01).

Table 3.

STANDARD Q COVID-19 Ag test results compared with Ct value categories.

STANDARD Q COVID-19 Ag test result	qRT-PCR Ct Value Category (col %)			Total
	Strongly Positive (≤29)	Moderately Positive (30–37)	Weakly Positive (38–39)
Positive	34 (92)	24 (55)	5 (56)	63
Negative	3 (8)	20 (45)	4 (44)	27
Total	37	44	9	90

Discussion

The WHO recently issued interim guidance regarding the use of antigen RDTs for patient management. As a minimum requirement, Ag-RDTs should correctly identify significantly more cases than they miss (sensitivity ≥ 80%) and exhibit very high specificity (97–100%) (WHO, 2020b).

If any of the antigen detection tests that are under development exhibit adequate performance, they could potentially be used as triage tests to rapidly identify patients who are very likely to have COVID-19, thereby reducing or eliminating the need for expensive nucleic acid amplification testing (ICMR, 2020).

In this evaluation of the STANDARD Q COVID-19 Ag Test, a commercial antigen RDT, we found that the sensitivity and specificity of the test were 70% and 92% respectively. Overall, after combining both the positive and negative samples, the accuracy of the test was determined as 84%.

In the present evaluation, the sensitivity was lower than that reported by the manufacturer (70% compared with 84.38%) and the difference was statistically significant (p < 0.001). Similarly, the specificity in this evaluation was significantly lower than that reported by the manufacturer (92% compared with 100%). These discrepancies may have been due partly to the limitations of this study, as discussed in the following.

With a sensitivity of 70%, the STANDARD Q COVID-19 Ag Test would not detect 30 out of 100 qRT-PCR positive samples. Similarly, with a specificity of 92%, the STANDARD Q COVID-19 Ag Test would find that eight out of 100 qRT-PCR negative samples were positive.

Timely testing is crucial in order to contain the pandemic. Currently, most qRT-PCR-based testing is conducted in designated, specialized laboratories, which are far from many sample collection sites or patients and their contacts, thereby leading to long turnaround times for results reporting, as well as delaying isolation and contact tracing to possibly risk increasing the further spread of the virus within the country.

Decentralized testing using mobile PCR laboratories and point-of-care GeneXpert platforms has been introduced in Uganda to shorten the turnaround time for results. However, challenges still remain in terms of the supply of reagents, cost, and low testing capacity at times.

Introducing an antigen RDT with good performance at the point of entry would be a significant improvement compared with current practice. The results obtained in this study showed that the SB Biosensor antigen RDT exhibited less than optimal performance because the ideal test would have a sensitivity >95% and specificity of 100%. The STANDARD Q COVID-19 Ag Test (Gyeonggi-do, 16690, Korea) performed better (sensitivity = 84.38%, specificity = 100%) during its validation by the manufacturer in Malaysia (STANDARD Q COVID-19 Ag Test IFU) and in a two-site evaluation in India (ICMR 2020) (sensitivity = 50.6% and 84.0%, specificity = 99.3% and 100%).

The few previously published field evaluations of SARS-CoV-2 antigen RDTs reported mixed results. In three separate studies (Scohy et al., 2020, Lambert-Niclot et al., 2020, Mertens et al., 2020), the COVID-19 Ag Respi-Strip (Coris Bioconcept, Gembloux, Belgium) was found to exhibit good specificity (99.5–100%) but low sensitivity (30.2–57.6%) compared with qRT-PCR. Another RDT called the BIOCREDIT COVID-19 Ag test (RapiGEN Inc., Gyeonggi-do, 14119, Korea) (Mak et al., 2020) also exhibited low sensitivity (11.1–45.7% with specimens from the nasopharynx, throat, saliva, and sputum) and the authors concluded that this test can only be used as an adjunct to the qRT-PCR test because of the potential for false-negative results.

However, two other studies reported the good performance of another antigen test called the Fluorescence Immunochromatographic SARS-CoV-2 Antigen Test (Bioeasy Biotechnology Co., Shenzhen, China). In a study of 127 participants in Chile (Porte et al., 2020), the overall sensitivity and specificity were 93.9% and 100%, respectively, and the accuracy was 96.1%. In a second study of 239 participants in China (Diao et al., 2020), the overall sensitivity and specificity were 67.8% and 100%, respectively. Similar to our study, the sensitivity in both studies was significantly higher in samples with high viral loads.

Despite its less than optimal performance, the STANDARD Q COVID-19 Ag test could still play a role in triage (but with subsequent selective testing by qRT-PCR) in situations where the rapid isolation of cases is required, e.g., for symptomatic cases or “high-risk” truck drivers at border crossings. In these situations, all individuals identified as COVID-19 positive would require individualized isolation until the qRT-PCR results are available and to avoid contact with true positives because a few antigen false-positives are actually not infected. This approach can reduce the time in isolation for those who are infected and simplify the logistics of contact tracing, although the antigen test incurs an additional cost.

India has taken a different approach (ICMR, 2020). Following their evaluation of the STANDARD Q COVID-19 Ag Test, the Indian Council of Medical Research recommended that for high-risk groups in containment zones or hotspots and healthcare settings, “suspected individuals who test negative for COVID-19 by rapid antigen test should definitely be tested sequentially by RT-PCR to rule out infection, whereas a positive test should be considered as a true positive and does not need reconfirmation by RT-PCR.”

Our prospective and un-blinded study had the following limitations. The staff who evaluated the antigen RDT knew which participants were likely to be infected or uninfected in most cases, which may have biased their interpretation of the antigen test result, especially where the RDT bands were weak.

The staff who collected swab samples were different at each health facility and they had minimal training, which may have introduced variability into the content or yield of swab material. Moreover, two swabs were taken from each subject, with one from each nostril. One was tested immediately with the antigen RDT and the other was placed in transport medium for subsequent qRT-PCR testing at UVRI. Thus, the two swabs could be effectively regarded as two different samples.

The date of exposure was not known for any of the participants and only 13 of the cases had a date of symptom onset because the majority of the participants were asymptomatic (86%). Day zero was taken as the date when the first swab was taken. The analysis would have benefitted from details of the date of exposure because the viral load increases over the first 1–2 weeks (Wolfel et al., 2020, Zou et al., 2020) and it appears to be correlated with the performance of the antigen test. We observed better sensitivity for the samples with low Ct values, which were assumed to have higher viral loads.

Due to logistical constraints, some swab specimens were transported to UVRI in normal saline rather than viral transport medium. The WHO recommends the use of normal saline in the absence of the viral transport medium, but its effect on the sensitivity of qRT-PCR is not well documented.

The qRT-PCR result was taken as the gold standard. However, both the qRT-PCR and antigen RDT will miss some infections, particularly those mentioned above due to limitations in the collection and transport of swabs. A more significant source of error in this evaluation was probably the determination of the qRT-PCR Ct cut-off value for a positive sample. Thus, the results for specimens with a Ct value near the cut-off should probably be regarded as provisional and ideally repeated with a fresh sample until the Ct cut-off value for a positive result can be defined more precisely. In the future, it will be important to develop standard operating procedures that address the limitations mentioned above in the pre-analytical, analytical, and post-analytical phases of testing.

Conclusion

The STANDARD Q COVID-19 Ag Test obtained less than optimal performance in this evaluation. However, the test may still have an important role to play early in infection when timely access to molecular testing is not available, but the results should be confirmed by qRT-PCR.

Contributors

TL, CW, BK, DS, DA, HM, and PK conceived and designed the study, and/or wrote and proof-read the manuscript. AM, DO, EO, JS, SN, IS, HK, MC, and CN provided specimens and/or demographic data, and/or conducted antigen tests and/or interpreted the data. JKa, SB, JKi, JN, CN, and JL conducted the molecular testing. TL handled the data and analysis. TL, RD, and PK revised/edited the manuscript for intellectual content.

All authors reviewed the manuscript and gave final approval for submission

Declaration of interests

All authors declare no competing interests.

Funding

No specific funding was received from funding bodies in the commercial, public, or not-for-profit sectors.

Appendix A. Supplementary data

The following is Supplementary data to this article: