Assessment of Test Performance and Potential for Environmental Contamination Associated with a Point-of-Care Molecular Assay for Group A Streptococcus in an End User Setting

Although U.S. Food and Drug Administration-approved and CLIA-waived point-of-care (POC) molecular systems are being implemented in routine clinical practice, instrument reliability, test performance in the hands of end users, and the potential for environmental contamination resulting from use of POC molecular systems have not been extensively evaluated.

ABSTRACT

Although U.S. Food and Drug Administration-approved and CLIA-waived point-of-care (POC) molecular systems are being implemented in routine clinical practice, instrument reliability, test performance in the hands of end users, and the potential for environmental contamination resulting from use of POC molecular systems have not been extensively evaluated. We performed a prospective evaluation of the Roche cobas Liat group A streptococcus (GAS) assay compared to routine real-time PCR. We evaluated test accuracy, instrument failure rate, and monitored for environmental contamination when testing was performed by minimally trained end users in an Express Care Clinic environment. The overall concordance of the Liat GAS assay with routine testing was 97.2% (455/468). The average Liat failure rate across three analyzers was 6.6% (33/501) (range, 3.7 to 11.6%), and no environmental contamination was detected during the course of the study. The cobas Liat platform and GAS assay demonstrated reliable performance in the end user setting and may serve as a rapid, POC option for routine diagnostic testing for certain infectious diseases, including GAS.

INTRODUCTION

Accurate laboratory results are critical for patient management in cases of suspected infectious diseases. Traditionally, testing for many infectious diseases has relied on the use of culture techniques, which are often associated with a long turnaround time. Over the past decade, nucleic acid amplification tests (NAATs; e.g., real-time PCR) have become routine in the clinical laboratory for diagnosis of infectious diseases, including Streptococcus pyogenes (i.e., group A streptococci [GAS]). NAATs can typically provide results within hours of sample collection and have demonstrated similar, or improved, sensitivity compared to culture-based tests (1, 2). However, until recently, molecular testing has been classified by the U.S. Food and Drug Administration (FDA) as moderate to highly complex, which in the United States means that samples must be processed and tested by medical technologists in a central laboratory. Point-of-care (POC) methods, either rapid antigen testing or newly developed molecular techniques have certain advantages, including a more rapid turnaround time (<30 min), potentially allowing for more timely patient management decisions. Unfortunately, rapid antigen tests have demonstrated poor sensitivity for certain analytes (e.g., influenza), such that negative samples require confirmatory testing (3, 4). Recently, several POC molecular platforms (e.g., cobas Liat [Roche Diagnostics, Indianapolis, IN]) have been cleared by the FDA and labeled as Clinical Laboratory Improvement Amendments (CLIA)-waived tests. Importantly, these tests have shown similar analytical sensitivity and specificity to routine NAATs performed in central clinical laboratories (1, 5, 6).

Previous studies comparing the performance of POC molecular assays to routine NAATs have most often been conducted in a central laboratory by staff that is skilled in molecular testing and sensitive to the potential for amplicon/specimen contamination (5, 7). Although beneficial, this type of study design evaluates the performance of a platform/assay under optimal conditions. The end users of POC tests are usually health care providers with limited experience in performing molecular tests. Therefore, evaluation of POC molecular tests in the hands of intended end-users (e.g., physicians, nurse practitioners/physician assistants, and nursing staff) is critical to fully assess test performance and the potential for environmental contamination.

In this study, we investigated the performance of the cobas Liat platform and the Liat GAS assay at two retail-based convenience clinics (Mayo Clinic Express Care). Test accuracy compared to a routine real-time PCR assay, as well as the potential for environmental contamination and Liat instrument failure rates, was assessed.

MATERIALS AND METHODS

Study design.

The study was approved by the Mayo Clinic Institution Review Board. The study population (n = 468) was comprised of a community-based cohort, aged 3 to 65 years old (n = 217, age < 18 years; n = 251, age ≥ 18 years), presenting to either of two Mayo Clinic Express Care clinics (sites A and B) in Rochester, MN, between May and August 2017. These clinics are located in retail settings and designed to manage minor acute illnesses. They are staffed exclusively by nurse practitioners (NP) and licensed practical nurses (LPN). All study participants were selected from patients receiving GAS testing as part of their routine clinical care. GAS testing is performed on patients with chief complaint of a sore throat and a Centor score of ≥3 (8). Written consent (adults and children aged 3 to 80 years old) and assent (for children aged 8 to 12 years old) was obtained from all participants.

Two pharyngeal swab specimens (BBL CultureSwab [Becton, Dickinson, Franklin Lakes, NJ] in NB buffer for routine clinical testing; BD ESwab Collection and Transport System for Liat GAS testing) were collected simultaneously from each participant by the NP. The swab collected for routine clinical testing was submitted to the laboratory and tested by real-time PCR as previously described (2). The ESwab was placed in the designated transport media and analyzed immediately on the cobas Liat by either the NP or the LPN according to the manufacturer’s recommendations. In cases of Liat assay failure, the test was repeated using the original sample on a new Liat GAS cartridge. The results of the Liat POC and central laboratory real-time PCR assays were compared and percent concordance calculated. To calculate concordance, only valid results generated from the Liat and routine real-time PCR were used. A total of 27 end users, NPs and LPNs, who rotated between clinics were trained to perform testing on the Liat. Training consisted of reviewing the manufacturer’s instructions for use, the Liat quick reference guide, and observing instrument use.

Environmental monitoring.

Weekly environmental testing was conducted at each study site. A member of the study team (M.J.E.) came to each site in the morning, swabbed the Liat instrument and surrounding benchtop using one swab (ESwab Collection and Transport System; BD). The swab was placed in the transport media and tested immediately by the Liat GAS assay according to the manufacturer’s instructions. The remaining sample was transported to the clinical laboratory for routine testing, which was performed within 4 h. A positive result from the environmental swab tested by the Liat assay would be an indicator of either specimen and/or amplicon contamination, whereas a positive result by the routine real-time PCR test would suggest specimen contamination. During the study period, a total of 26 environmental swabs (13 swabs from site A, 13 swabs from site B) were collected. No special cleaning of the instrument or surrounding benchtop was performed prior to collection of the environmental swabs.

Assessment of assay failure rates.

Assay failures were noted for all Liat instruments (n = 3) used in the study (one instrument at site A; two instruments at site B). Two possible assay failure codes were recorded: (i) an invalid/indeterminate code or (ii) an aborted run code. When either type of assay failure occurred, the Liat test was repeated immediately using a new test cartridge and the original sample. In rare cases, samples required three or four total tests to obtain a valid result (either positive or negative).

RESULTS

Comparison of routine real-time PCR and Liat point-of-care results for group A streptococci.

Among 468 subjects accrued to this study, the results of the Liat GAS POC and routine GAS real-time PCR showed an overall concordance of 97.2% (455/468) (Table 1). Of the 13 discordant results, 5 (38.5%) samples were positive by the Liat but negative by the routine method, while 8 (61.5%) samples were negative by the Liat but positive by the routine real-time PCR (Table 1). This small number of discordant samples was not large enough to demonstrate statistical significance. The relatively high overall positivity rate in our study (45%, 206/459) was attributed to an outbreak of GAS in the local community during the time of this study. The use of a Centor score of ≥3 in routine clinical practice to identify those most likely to be GAS positive likely guided proper test utilization and increased the pretest probability.

TABLE 1.

Comparison of the cobas Liat GAS assay to routine real-time PCR using pharyngeal swab specimens (n = 468)^a

cobas Liat GAS assay result	Routine real-time PCR result (no. of samples)		% Agreement (95% CI)
	Positive	Negative
Positive	206	5	97.2 (95.3–98.4)
Negative	8	249

^aGAS, group A streptococci; CI, confidence interval.

Assessment of Liat GAS assay failure rates.

In cases where the Liat instrument gave an error code, the transport media from the original participant sample was retested. From the 468 participant samples collected, a total of 28 (6.0%) samples (9 pediatric samples, 19 adult samples) required repeat testing after an initial assay failure. For 24 (85.7%) of 28 retested samples, the first repeat gave a valid result of either positive or negative. However, in 3 (10.7%) of the retested samples, a valid result was not achieved until the second repeat, and in 1 (3.6%) sample, three repeats were required to obtain a valid result. Therefore, a total of 501 Liat GAS tests were run during the duration of this study (i.e., 468 initial tests and 33 [6.6%] total repeat tests). There was no discernible pattern to the failure occurrence (failures occurred randomly throughout the study) so failures could not be predicted. All samples requiring more than one repeat analysis came from adult subjects. In all cases, the health care provider was able to obtain a valid result after retesting the original sample on a new test cartridge. All Liat results from initial failed runs were concordant with the central laboratory result.

Assay failure rates were also measured independently for each Liat analyzer used in the study. At site A, a single Liat analyzer (analyzer 1) was utilized for all testing at this location. The assay failure rate for analyzer 1 was determined to be 3.7% (8/219) (Table 2). At site B, the initial Liat analyzer (analyzer 2) displayed an assay failure rate of 11.6% (21/181). Given the higher than expected failure rate of analyzer 2, the instrument was replaced by the manufacturer with analyzer 3 midway through the study. The assay failure rate of analyzer 3 was determined to be 4.0% (4/101) (Table 2).

TABLE 2.

Summary of GAS assay failures using Roche cobas Liat analyzers

Location	Total no. of assays	No. of failed assays	Failure rate (%)
Site A
Analyzer 1	219	8	3.7
Site B
Analyzer 2	181	21	11.6
Analyzer 3	101	4	4.0
Total	501	33	6.6

Determination of environmental contamination rates associated with use of the Liat platform.

Among 26 total environmental swabs collected over the course of the study (13 swabs at each site), all tested negative for GAS by both the Liat (n = 26) and routine real-time PCR (n = 26) assays.

DISCUSSION

In this study, we compared the performance of the cobas Liat GAS assay to routine real-time PCR. Although previous studies have assessed the accuracy of the Liat for detection of influenza and GAS (1, 5, 6), to our knowledge, this is the first study to compare the Liat GAS assay to routine real-time PCR. Furthermore, our study addressed the potential for environmental contamination and Liat instrument failures when testing was performed by health care providers in a POC setting. This is important, since the adoption of rapid, POC molecular tests represents a paradigm shift in the diagnosis of certain infectious diseases, such as GAS. Historically, molecular testing for infectious diseases has been routed to a central clinical laboratory for analysis by medical technologists who are trained in proper workflow and contamination mitigation. The implementation of molecular POC assays will bring about a number of advantages compared to centralized lab testing (e.g., reduced turnaround time), but a number of issues should be carefully considered prior to their widespread use. First, CLIA-waived molecular POC tests will likely be performed by health care providers (i.e., nursing staff and physicians) who spend a minority of their day performing laboratory testing. Healthcare providers have typically not received extensive training on molecular methods, including the potential negative impact that specimen and amplicon contamination can have on test results. Therefore, a primary focus of this study was to assess the potential for environmental contamination when the Liat instruments were used in a POC setting by health care providers. Interestingly, we did not observe evidence of specimen or amplicon contamination during the 4-month study period.

Instrument and assay reliability are important factors when implementing any laboratory test, but they are especially important when evaluating a POC platform that will be used to assist in rapid patient management decisions. To that end, we tracked the failure rate of three different Liat analyzers used during the study. Failures were observed in 33 (6.6%) of 501 total tests, and for all samples with an initial assay failure, a valid result (positive or negative) was eventually achieved with repeat testing of the original sample. A single repeat was needed to obtain a valid result in the majority (86%) of first time failure cases; however, one sample did require three repeat tests to achieve a valid result. Importantly, the valid results obtained from samples with an initial failure were concordant with the central laboratory in all samples indicating that an initial assay failure is not an indication that the subsequent Liat result is more likely to be inaccurate. In routine clinical practice, multiple rounds of repeat testing would not be practical because the analytical time for the Liat GAS assay is ∼15 min. Since a single Liat analyzer can only test one sample at a time, assay failures can have a significant impact on testing throughput in the POC setting. Even with a relatively low failure rate of 3 to 5%, a clinical area with only one Liat analyzer could experience challenges with patient workflows, especially during the peak of GAS or influenza season.

A previous study evaluated the Liat GAS assay and compared its performance to bacterial culture and rapid antigen testing (1). This study demonstrated the Liat GAS assay to have a sensitivity and specificity of 97.7 and 93.3%, respectively, compared to routine culture. Not surprisingly, the Liat showed improved sensitivity (97.7% versus 84.5%) compared to rapid antigen testing. In our study, we compared the Liat GAS assay to routine real-time PCR and observed a high overall concordance (97.2%). Only 13 (2.8%) of 468 samples showed discordant results in our study. Among these discordant findings, the crossing point (C_p) values were able to be reviewed for those samples (n = 8) that tested positive by routine real-time PCR but negative by the Liat. The majority (7/8; 87.5%) of the false-negative Liat results were positive by the routine real-time PCR at a C_p of >30 (out of 40 cycles total). This indicates a small amount of target nucleic acid template in these samples, which may account for the discrepant results.

Our study has the following limitations. First, the Liat and routine real-time PCR tests were performed using different swabs, so sampling issues may account for some of the discrepancies observed. Differences between results obtained using wound fiber swabs versus ESwabs have been noted previously (9–11). Second, the study was conducted over a 4-month period, and the Liat instruments were new and placed in each testing location just prior to the study beginning. Therefore, the results may not account for potential contamination that could occur with prolonged use of the Liat instruments. Lastly, our study only investigated assay performance for GAS, so data cannot be extrapolated for failure rates and environmental contamination of other Liat tests, such as influenza.

Express/urgent care clinics, primary care settings, and emergency departments often experience rapid patient turnover with minimal dedicated space for laboratory testing. Point-of-care molecular systems, such as the Liat, have a relatively small footprint (7.5 in. tall by 7.5 in. deep by 4.5 in. wide) and allow for health care providers to obtain highly sensitive results in <20 min, increasing diagnosis accuracy and expediting treatment plans for patients. However, careful consideration and planning should go into the routine implementation of these systems in POC settings, in order to reduce the potential for contamination that may negatively impact patient results. In summary, this study demonstrated that the Liat PCR system provides accurate GAS results in the clinic setting with no evidence of environmental contamination. The reduced result turnaround time compared to routine real-time PCR (∼15 min versus ∼6 h) would allow for more rapid patient management decisions and appropriate treatment initiation. Therefore, the Liat could be considered an option for POC GAS testing.