VA-Wide, Multicenter Verification Study of the Cepheid Xpert SARS-CoV-2 Assay

Abstract

Early in the Severe Acute Respiratory Syndrome Coronavirus 2 pandemic, there was a progressive increase in diagnostic demands that developed within a relatively short period of time. On February 4, 2020, the Secretary of Health and Human Services issued the Emergency Use Authorization for in vitro diagnostics assays for the Severe Acute Respiratory Syndrome Coronavirus 2 virus. Subsequently, multiple assays were approved under the Emergency Use Authorization, including the Cepheid Xpert SARS-CoV-2 assay. Presented here is a description of the nationally coordinated verification study of the Cepheid assay that was performed within the Veteran’s Affairs Health System. This coordinated study helped to expedite the verification process for a majority of the Veteran’s Affairs system labs, preserved precious system resources, and highlighted the power of a national medical system in response to an emergency.

Introduction

In December 2019, an outbreak of influenza-like illness caused by a novel coronavirus began spreading in China and other eastern countries. Subsequently named Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), the RNA-based virus eventually spread to other countries, including those in the western world. In March of 2020, the World Health Organization deemed the disease as pandemic, and the disease caused by SARS-CoV-2 was given the name of COVID-19. At the time of writing this article, over 15 million cases of COVID-19 disease and over 250 000 deaths have been documented in United States alone.¹

COVID-19 primarily affects the respiratory system and clinical manifestations range from mild flu-like symptoms to severe respiratory failure requiring in-house management and, in many occasions, leading to death in high-risk groups.² Diagnosis was initially dependent on clinical symptoms +/− radiological studies and confirmed by laboratory testing that was only available through the Centers for Disease Control and Prevention (CDC) or their associated network of public health laboratories. That testing relied on detecting SARS-CoV-2 RNA by real-time reverse transcriptase polymerase chain reaction (RT-PCR) which continues to be the most reliable method of testing for detecting the virus. There are a number of RT-PCR methods that target different genes; however, the envelope (E-gene) or the nucleoprotein (N-gene) genes of the virus are most commonly targeted.³

With the rapid increase in cases early in the pandemic and the strict limitation on testing resources centralized in the public health sector, there was a glaring need for testing in hospital labs out in the field. On February 4, 2020, the Secretary of the Department of Health and Human Services issued permission for Emergency Use Authorization (EUA) of in vitro diagnostics for the detection and/or diagnosis of COVID-19. Among the early PCR assays available to the broad community was the Cepheid Xpert SARS-CoV-2 test. This is a real-time RT-PCR assay that obtained its EUA status on March 20, 2020. The test qualitatively detects nucleic acid (the nucleocapsid (N2) gene and the envelope (E) gene) from SARS-CoV-2 and generates results within approximately 45 minutes.⁴

Verification of an assay in the clinical lab is typically done according to a relatively standard template, at least within an individual laboratory. However, not many clinical labs have experience in doing verification studies for an EUA assay in an emergency scenario. In such a scenario, there is less of an accepted script on how a verification needs to take place, and it is easy for a lab to over verify an emergency use test. Typical protocols would require at least running 20 to 30 tests over a multiday period. Indeed, that is a minimum and could involve many more. If performed individually, this would have consumed up to 3750 tests among 125 Veteran’s Affairs (VA) labs involved in this combined verification. Alternatively, when the same test is being verified by many labs in the same system at the same time, it is possible to pool data to collectively power a larger verification study using a minimum amount of total and local resources. At the time of the first availability of the Cepheid assay, VA labs were each only getting 30 tests per week that were sorely needed for clinical care. Herein, we describe a novel multicenter verification plan of the Xpert SARS-CoV-2 test that standardized the verification process for labs across the entire VA system, allowing labs to begin clinical testing very quickly after delivery of testing supplies while using the minimum amount of those critical testing reagents.

Methods

Through national VA distribution, 125 VA medical centers were initially provided with 30 Xpert SARS-CoV-2 cartridges in early March of 2020. The identical positive and negative controls were obtained by each participating local site directly from SeraCare (SeraCare AccuPlex™ SARS-CoV-2 Reference Material Kit 0505-0126). Each site was provided with a standardized verification study protocol and data collection sheet (see Supplemental Materials). Briefly, both positive and negative controls were run in duplication on day 1 of the local study using procedures outlined in the Cepheid SARS-CoV-2 assay package insert. On day 2, the positive and negative controls were run again. It was recommended the 3 runs be performed by different operators. All results were recorded on the provided data collection sheet and returned to the coordinating site (Iowa City VA Medical Center). If the instrument reported an “Error” or “Invalid” result on any control sample or if a positive control had a “Presumptive Positive” result that control was to be repeated the same day after any corrective actions. In the setting of a fully discordant result (negative result on a positive control or positive/presumptive positive on a negative control), then that day’s testing was to be repeated on the following day. If all results were concordant on both days 1 and 2, then the lab can go live that second day after sending the data collection sheet to the central site plus completion of internal procedures, director sign off, and appropriate information technology/lab information system setup.

Initial versions of the protocol involved sending a set number of clinical samples to state health labs or other reference laboratories as a backup verification of performance. However, due to the lack of resources in these labs, National VA Path and Lab thought this was unnecessary and would utilize rather precious community resources at that point in time. As such, the requirement to send clinical samples to outside labs after going “live” with the Cepheid assay was dropped.

Data were collected by the Iowa City VA from each of the 125 sites and collated. Data were monitored in real time by the authors to detect any emerging issues. The collated data set was returned to the sites in early April for their verification files.

Results

No set number of total completed study sites were required for any particular site to go live. However, 19 sites submitted their data on the first possible day (all data as expected; data not shown). Forty-two more sites submitted data the following day. Overall, as shown in Table 1, of the 750 initial total control runs among the 125 facilities, only 3 unexpected results from 3 facilities were observed. Two of those were instrument “errors” and were likely related to operation of the Cepheid system. One was an actual discordant result that was thought to be secondary to a potential mislabel of a control material tube. On repeating with appropriate reagents, all results were as expected; 122 of the labs that participated were able to begin using the assay for clinical use within 2 days of beginning the validation study. The remaining 3 labs were able to be live in 3 or 4 days. A total of 754 Cepheid tests were utilized to complete this combined verification study of 125 instruments.

Table 1.

Summary of National Verification Study Data.

Verification Phase	Day 1	Day 2
Positive control Run 1 Concordance	125/125 (100%)	125/125 (100%)
Positive control Run 2 Concordance	125/125 (100%)	Not Applicable
Negative control Run 1 Concordance*,†	123/125 (98.4%)	125/125 (100%)
Negative control Run 2 Concordance*	124/125 (99.2%)	Not Applicable
Overall concordance after first run	747/750 (99.6%)
Total number initial runs	750
Final total number of runs (including repeats)	754

* There were 2 “Error” results obtained due to incorrect assay set up. Repeat runs were as expected.

^† One presumptive positive result was obtained with a negative control vial. It was determined to be a faulty vial of control material. SeraCare sent a new negative control overnight, and the problem was resolved the next day after protocol-prescribed repeat testing (both positive and negative controls were used using the new negative control vial).

Bold text indicates cumulated data/results

Discussion

In the beginning of the COVID-19 pandemic, there were very few testing options available in individual hospitals. Early PCR testing involved mostly lab developed testing with the CDC-designed test followed by the introduction of specific assays on larger systems such as the Abbott m2000. Cepheid was among the first to introduce a test to the field that could be performed more easily in most hospital labs, even if it wasn’t as high throughput as some of the existing platforms. The accuracy of that assay at the time of introduction had been demonstrated by Cepheid in comparison to existing platforms, and those data were submitted to the Food and Drug Administration.⁵ There were enough data to suggest that the assay was functioning sufficiently in comparison to existing assays. Subsequent studies that investigated the Cepheid Xpert assay performance as compared to other commercially available tests such as the Cobas SARS-CoV-2 (Roche Molecular Systems)⁶ and Panther Fusion SARS-CoV-2 (Hologic)⁷ confirmed that these tests demonstrated agreement.

After the assay was available and before starting testing and reporting patient’s results in each of the VA laboratories, a verification was required by CLIA88 to confirm the performance of the assay in each lab with the specifications established by the manufacturer.⁸ Although the number of specimens needed for verification may vary depending on the test, a typical study involving the testing of 20 to 30 specimens is usually done to “check-in” a new test of this type. As stated above, VA was only receiving 30 Cepheid tests per week at the early stages of the pandemic. The use of 20 to 30 tests in each facility to do a verification study would have diverted much needed resources away from clinical testing. The lack of widespread availability of positive patient samples was an additional challenge for any verification study at that time. This provided the impetus for designing and executing a national verification study where combined data were used to create a larger powered study using a smaller number of specimens in each lab. This would also expedite the verification and provide much needed guidance to smaller VA labs during a very hurried and confusing time. The involved hospitals following the study plan used only 6 cartridges for verification. With 125 hospitals participating, the total number of cartridges used in the validation study was 750, with 4 additional cartridges due to having to repeat a few tests. In comparison with standard verification for which at ∼30 cartridges would have been used per hospital, there was a savings of about 3000 CoV-2 tests in the system.

This initial impact on the system is difficult to quantify, but we know that early detection of COVID-19 patients during the earliest waves of disease was critical for limiting the expansion of local outbreaks. This test proved a critical tool for that purpose and continues to be a major diagnostic tool for VA. From April 2020, when VA first went live with the Cepheid single target CoV-2 assay until the assay was no longer commercially available in November 2020, the VA system ran approximately 750 000 of these assays, with the vast majority within the 125 centers that were involved in the initial verification study (internal, unpublished VA data).

The coordinated verification study highlights the power of having a coordinated system of hospitals. This type of shared data collection and utilization would not have been feasible outside of such a system where there is a common communication medium and a single stream for resources. However, this does highlight the limitations of such a coordinated approach. For this type of approach to be successful, there must be an assay that is available inside a group of system labs at approximately the same time on the same platform. This is not a common occurrence outside of an emergency, but this successful shared verification study can provide a roadmap for such situations as they might present themselves in the future. One additional suggested feature of any future study would be to identify a minimum number of completed, successful dataset submissions prior to allowing any individual lab to go live with the assay.

Conclusion

A coordinated, national verification study within the VA system for the Cepheid SARS CoV-2 assay allowed multiple VA centers to verify the assay using only 6 tests per lab. This strategy allowed labs to go live quickly in an emergency and preserved scarce resources early in the COVID-19 pandemic. The key factor that facilitated and promoted the success of the study was the presence of multiple institutions inside the same system who share the same testing platforms, supply chain sources, and communication venues. This effort provides a guide for potential future shared verification studies, an approach that can be very useful, especially in an emergency scenario.