Evaluation of the Visby medical COVID-19 point of care nucleic acid amplification test

Abstract

Rapid and widespread diagnostic testing is critical to providing timely patient care and reducing transmission of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Recently, the Visby Medical COVID-19 point of care (POC) test was granted emergency use authorization (EUA) for qualitative detection of SARS-CoV-2 nucleic acid at the point of care. We evaluated its performance characteristics using residual specimens (n = 100) collected from Mayo Clinic patients using nasopharyngeal (NP) swabs and placed in viral transport media (VTM). The same specimen was tested using both the laboratory reference method (RT-qPCR) and Visby test. The reference methods utilized included a laboratory developed test with EUA (Mayo Clinic Laboratories, Rochester, MN) using the TaqMan assay on a Roche Light Cycler 480 or a commercially available EUA platform (cobas® SARS-CoV-2; Roche Diagnostics, Indianapolis, IN). Positive, negative, and overall percent agreement between the Visby COVID-19 test and the reference method were calculated. Additionally, the limit of detection (LoD) claimed by the manufacturer (1112 copies/mL) was verified with serial dilutions of heat inactivated virus. The Visby COVID-19 test correctly identified 29/30 positive samples and 69/70 negative samples, resulting in an overall concordance of 98.0%, positive percent agreement of 96.7%, and negative percent agreement of 98.6%. The abbreviated LoD experiment showed that the analytical sensitivity of the method is as low as or lower than 500 copies/mL. Our study demonstrated that Visby COVID-19 is well-suited to address rapid SARS-CoV-2 testing needs. It has high concordance with central laboratory-based RT-qPCR methods, a low rate of invalid results, and superior analytical sensitivity to some other EUA POC devices.

1. Introduction

Rapid and widespread testing is crucial to contain infections due to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Nucleic acid amplification tests (NAAT) performed in a central laboratory, most often reverse transcription quantitative polymerase chain reaction (RT-qPCR), are considered the gold standard for the diagnosis of COVID-19. However, there are formidable challenges with its use. Significant delays in the return of test results are attributed to the lengthy processing time required for many laboratory RT-qPCR methods; and the need for certified medical laboratory scientists and expensive equipment has been hampered by staffing and supply chain shortages.

In early 2020, the World Health Organization (WHO) prioritized research toward the development of rapid COVID-19 tests as one of the most important actions in successfully addressing the public health crisis [1]. At the time of writing, the Visby COVID-19 PCR test is one of eight rapid point-of-care (POC) NAATs that have been authorized by the Food and Drug Administration (FDA) for emergency use. These tests include: Visby Medical COVID-19 POC test, Lucira COVID-19 All-In-One Test Kit, BioFire Respiratory Panel 2.1-EZ, Roche cobas SARS-CoV-2 & Influenza A/B Nucleic Acid Test on the cobas Liat System, Cue COVID-19 test, Abbott ID NOW COVID-19, Mesa Biotech Accula SARS-Cov-2 test, and Cepheid Xpert Xpress SARS-CoV-2 test. The Visby Medical COVID-19 POC test was granted emergency use authorization (EUA) for point of care use on February 8, 2021 for qualitative detection of SARS-CoV-2 nucleic acid via its single-use, fully disposable POC device. We evaluated the performance characteristics of the test (Visby Medial COVID-19 POC) using residual specimens submitted to a central laboratory for RT-qPCR testing.

2. Materials and methods

Residual clinical samples (n = 100) collected from Mayo Clinic patients using nasopharyngeal (NP) swabs and placed in viral transport media (VTM) were used to assess accuracy. Specimens were refrigerated at 2–8 °C for up to 3 days prior to testing. The same specimen was tested using both the laboratory reference method and Visby test for comparison, with both reference and Visby testing completed within 72 h of specimen collection. The Mayo Clinic Institutional Review Board (IRB) determined that this study met institutional criteria for a quality assurance/improvement initiative and did not require IRB review.

2.1. Visby COVID-19 test

The Visby COVID-19 test (Visby Medical, San Jose, CA) is a single-use (disposable) RT-PCR-based diagnostic assay intended for the qualitative detection of SARS-CoV-2 RNA in NP, nasal, or mid-turbinate swabs. The sample mixes with lyophilized PCR reagents containing biotinylated primers specific to the nucleocapsid region of SARS-CoV-2 and to 18 s ribosomal RNA (internal process control). The mixture is thermocycled such that the DNA molecules present are amplified and can be detected via a colorimetric system. Amplified target (if present) is hybridized to specific locations along a flow channel. This flow channel is configured to facilitate an enzymatic reaction that utilizes horseradish peroxidase (HRP) and a color producing substrate. This will result in an observable color change for a positive reaction. At the time of this evaluation, the Visby test did not have EUA approval for use at the point of care; but that approval was subsequently received in February 2021.

2.2. Visby testing protocol

Per the manufacturer’s instructions for the Visby test, the provided disposable pipet was used to transfer viral transfer media (VTM) from the residual specimen into the provided Covid-19 dilution tube. The dilution tube was mixed by inverting five times, then a second disposable pipet was used to transfer sample into the sample port on the Visby device. After loading sample, the sample port was closed, buttons 1–3 were pushed in order, and the device was plugged in. Results were available in approximately 30 min. If invalid results were obtained upon initial evaluation, testing was repeated once, and the second valid result was treated as final and used in the calculations of assay performance.

2.3. Reference RT-qPCR methods

The reference methods utilized included a laboratory developed test with EUA (Mayo Clinic Laboratories, Rochester, MN) using the TaqMan assay on a Roche Light Cycler 480 [2] or a commercially available EUA FDA platform (cobas® SARS-CoV-2; Roche Diagnostics, Indianapolis, IN). The Roche test was performed on the cobas® 6800 instrument according to the manufacturer’s instructions for use. Positive samples were defined as those with a crossing point (Cp) value of 40 or below for both reference methods utilized.

2.4. Limit of detection experiment

The manufacturer claims a limit of detection (LoD) at 1112 genomic copies/mL. To verify the LoD, we created 3 sample pools with known concentrations 500, 1000, and 10,000 copies/mL) by diluting SARS-Related Coronavirus 2, Isolate USA-WA1/2020, Heat Inactivated material from BEI Resources (Manassas, VA) into VTM. The 500 and 1000 copies/mL pools were analyzed in triplicate and the 10,000 copies/mL pool was analyzed in duplicate.

2.5. Statistical analysis

Positive, negative, and overall percent agreement between the Visby COVID-19 test and the reference method were calculated. Cohen’s kappa coefficient to quantify agreement for qualitative results (detected/not detected) between the Visby COVID-19 test and the reference method was also calculated with 95% confidence intervals (CI). The kappa coefficient was evaluated using the guideline outlined by Landis and Koch [3], where the strength of the kappa coefficient = 0.01–0.20 “slight agreement”; 0.21–0.40 “fair agreement”; 0.41–0.60 “moderate agreement”; 0.61–0.80 “substantial agreement”; 0.81–1.00 “almost perfect agreement”. Agreement analysis was performed using GraphPad QuickCalcs (web browser calculator, https://www.graphpad.com/quickcalcs/kappa2/).

3. Results

A total of 100 samples were tested on both the Visby Medical COVID-19 POC test and reference NAAT, 70 previously tested negative samples and 30 previously tested positive samples. The reference test was the lab-developed RT-qPCR test for 30 samples and the Roche 6800 for 70 samples. Two samples generated an “invalid” result with Visby. Upon repeat testing, the results from these two samples matched the result obtained from the reference method (one positive and one negative). The Visby COVID-19 test correctly identified 29/30 positive samples and 69/70 negative samples, resulting in an overall concordance of 98.0% (Table 1 ). The positive percent agreement (PPA) was 96.7%, and the negative percent agreement (NPA) was 98.6%. The Cohen’s kappa coefficient was 0.95 (95% CI: 0.89–1.00) which falls within the range of “almost perfect agreement” as described by Landis and Koch [3]. The presumed false negative sample obtained with the Visby COVID-19 test had a Cp value of 33 using the Roche 6800 reference testing method, indicating low viral load. However, two samples with Cp values of 35 on the Roche 6800 reference method were detected with the Visby POC test. The presumed false positive sample obtained with the Visby test was negative according to the LDT reference method.

Table 1.

Comparison of the Visby Medical COVID-19 POC test and RT-qPCR reference test for SARS-CoV-2.

	Visby Health COVID-19 POC Test
Reference Test*	Detected	Not Detected
Detected	29	1
Not Detected	1	69
Total	30	70
Positive Percent Agreement (PPA)	96.7%
Negative Percent Agreement (NPA)	98.6%
Overall Agreement	98.0%
Cohen's Kappa	0.95 (95% CI: 0.89–1.00)

^*Either Mayo Clinic RT-qPCR LDT or Roche cobas SARS-CoV-2 test.

The LoD of the Visby test was probed using a heat inactivated clinical SARS-CoV-2 isolate that was serially diluted (500, 1000, and 10,000 copies/mL) in VTM. Duplicate (10,000 copies/mL) and triplicate (500 and 1000 copies/mL) measurements of each dilution were all positive, demonstrating that the analytical sensitivity of the method is as low as 500 copies/mL. These data agree with the manufacturer’s claim of a 95% detection rate for 1112 genomic copies/mL and a 75% detection rate for 500 genomic copies/mL.

4. Discussion

To facilitate increased access and more rapid testing outside of laboratory settings such as in walk-in clinics, emergency departments, and pre-procedural locations, several manufacturers have developed tests for COVID-19 designed for use at the point of care. Compared to their laboratory-based counterparts, the advantages of these tests are more rapid result generation, they are typically portable and easy to use, and do not require highly skilled personnel to perform testing. The aim of our study was to evaluate the performance of one of these POC solutions, the Visby COVID-19 test, for the detection of SARS-CoV-2.

Results obtained on the Visby POC device, were compared to RT-qPCR reference methods (LDT or Roche cobas 6800). The rapid test had positive and negative percent agreement of 96.7% and 98.6%, respectively, with the overall concordance calculated as 98.0%. These results agree with the findings (95% sensitivity, 100% specificity, 96.1% overall concordance) from a recent 78 specimen evaluation of the Visby device described by Renzoni, et al. [4]. Of the two specimens that were discordant on Visby and the reference method, the presumed false negative by Visby was obtained from a specimen near the upper limit of positive by the reference method at Cp = 33. This suggests that the viral load in this sample may have been too low for detection on the Visby test. However, two other samples with Cp values of 35 on the reference method were positive by Visby. One result was presumably false positive by Visby, with no clear explanation for that discrepant result. Discrepant results may be a result of the differences in the assay design—differing in both sample processing procedures and buffers utilized or differences in limit of detection of the assays. We have found that some rapid NAAT for SARS-CoV-2 are more prone to false positive results compared to central laboratory NAAT [2], so a 1–2% rate of false positives was not unexpected. The potential for false positive results may need to be factored into testing protocols when using rapid NAAT at the point of care.

Only 2.0% of specimens initially tested on the Visby device were invalid and required repeat testing in accordance with the manufacturer’s instructions for use. Upon repeat, both samples agreed with the results obtained from the reference method. The rate of invalid results obtained on the Visby POC device is substantially lower than that of the invalid rates reported for similar molecular POC devices, namely, the Cue COVID-19 rapid test (8.6%) and the Abbott ID NOW test (8.6%) [2], [5]. Renzoni, et al. reported a similar invalid rate for the Visby device of 1.1% [4].

Finally, in an abridged LoD experiment performed using serial dilutions of a heat inactivated isolate of SARS-CoV-2, we found that the analytical sensitivity for the Visby test was as low as 500 copies/mL; with all three replicates testing positive at this concentration. While we did not dilute below 500 copies/mL, the Visby evaluation published by Renzoni described an analytical sensitivity down to 100 copies/mL, comparable to LoDs obtained using several central laboratory-based RT-qPCR methods [6]. In contrast, one other rapid NAAT using lateral flow technology, the Accula SARS-CoV-2 POC test, was found to detect only 66.7% of samples with cycle threshold between 30 and 35 [7]. The LoD of the Abbott ID NOW rapid NAAT was reported in one study to be much higher at 20,000 copies/mL [5]. Thus, the LoD of the Visby RT-qPCR assay is significantly lower than that described for some other rapid NAAT for SARS-CoV-2.

It is important to highlight a few of the limitations of our study. This small, single-center study included a random sampling of residual non-frozen clinical specimens previously tested on the reference methods. Due to limited access to other sample types at the time of the study (eg. nasal and mid-turbinate), only NP swabs were included in this evaluation. Furthermore, comparisons between the test methods were not performed through prospective recruitment of patients. Patient data regarding the presence/absence of symptoms at the time of presentation was not known as this information could not be obtained due to the nature of our IRB approval. Likewise, information about the time of presentation as it relates to the onset of symptoms in symptomatic individuals could not be captured. The LoD experiment performed was significantly abbreviated. Therefore, it’s possible that we would have verified a lower value had a full, formal LoD experiment been performed. Further evaluations are warranted to probe positive and negative percent agreement when symptomatic and asymptomatic patients are prospectively tested by both Visby and reference NAAT methods.

5. Conclusion

In summary, our study demonstrated that the Visby COVID-19 test has many desirable POC test characteristics—it is highly concordant with the central laboratory-based RT-qPCR reference method, has a low rate of invalid results that require repeat testing, can detect lower viral loads than some other rapid tests, and provides a qualitative assessment of COVID-19 status within minutes. Combined with its portability and ease of use, this device may be an attractive alternative to central laboratory RT-qPCR testing, specifically in settings staffed by non-lab personnel where rapid results at the point of care are necessary for timely patient care.