Comparison of the Alere i Strep A Test and the BD Veritor System in the Detection of Group A Streptococcus and the Hypothetical Impact of Results on Antibiotic Utilization

ABSTRACT

Rapid detection of group A Streptococcus (GAS) is an integral component of treatment decisions in the clinic, especially in the pediatric population. We prospectively collected 216 specimens from symptomatic, predominantly pediatric patients and evaluated the performance of the Alere i Strep A test (Alere i; Alere Inc., Scarborough, ME) and the BD Veritor system (BD Veritor; Becton, Dickinson and Company, Sparks, MD), with culture results being used as a comparator. Real-time PCR (RT-PCR) was performed as an arbiter in discordant cases. Comprehensive chart review was also done to determine the hypothetical impact of the results on antibiotic use. Alere i had a sensitivity and a specificity of 100% and 91.3%, respectively, and BD Veritor had a sensitivity and a specificity of 76.2% and 93.6%, respectively, when the results were compared to those of GAS culture. Further analysis of discordant results using RT-PCR revealed that while BD Veritor missed 13 confirmed positive cases, Alere i detected 100% (n = 13) of the same cases. Analysis of assay agreement showed that Alere i and BD Veritor had only moderate agreement (agreement = 0.888 [95% confidence interval {CI}, 0.838 to 0.927]; kappa index = 0.689 [95% CI, 0.91 to 0.974]). We also found both the underutilization and the overutilization of antibiotics based on the results of molecular testing. Overall, Alere i showed superior performance over BD Veritor in the detection of GAS pharyngitis and could potentially assist in better antibiotic utilization.

INTRODUCTION

Streptococcus pyogenes (group A Streptococcus [GAS]) is the most significant cause of acute bacterial pharyngitis (1), causing doctor's office visits for sore throat in 20 to 30% of children and 5 to 15% of adults (2, 3) and leading to an estimated 15 million office visits for sickness a year in the United States (3). GAS is also the most frequent diagnosis for which an antibiotic is prescribed for children 6 to 12 years of age in the United States (4). Identification and subsequent treatment of this pathogen are critical in order to avoid serious secondary, more severe complications of infection, such as rheumatic fever, rheumatic heart disease, poststreptococcal glomerulonephritis, and invasive disease (5). Proper identification of GAS is also critical to limit inappropriate antibiotic use (1). While the rates of invasive group A streptococcal disease are low in the United States, there were still over 15,000 cases reported in 2015 alone (6).

Assays to accurately identify GAS are crucial, since clinical signs and symptoms broadly overlap other causes of pharyngitis (7). Culture of throat swabs has traditionally been the standard method for the identification of GAS, but growth and identification can take up to 48 h. Assays that markedly reduce the time to identification, such as rapid antigen tests, have been developed and have high specificities, but they vary greatly in their sensitivities (range, 64.6% to 90%) (8–10).

Recently, rapid molecular assays have also been developed for GAS identification from throat swabs. The Alere i Strep A test (Alere i; Alere Inc., Scarborough, ME) is an FDA-approved, Clinical Laboratory Improvement Amendments (CLIA)-waived rapid molecular point-of-care testing (POCT) platform for the detection of GAS available on the market. It is an isothermal nucleic acid amplification assay that has a turnaround time of approximately 8 min (or less, in the case of a positive result). Our goal was to compare the performance of Alere i to that of our current instrument-read rapid antigen test, the BD Veritor system (BD Veritor; Becton, Dickinson and Company, Sparks, MD), using culture as our comparator and real-time-PCR (RT-PCR) amplification for adjudication of discordant results. A secondary goal was to assess the hypothetical impact of molecular testing for GAS on patient treatment.

MATERIALS AND METHODS

Study design.

With the approval of the Institutional Review Board of the University of Texas Medical Branch, we prospectively tested 216 clinical throat swab samples that were collected during the months of May and June of 2016 for routine testing for strep throat from two predominantly pediatric outpatient clinics within the University of Texas Medical Branch hospital system. These enrolled specimens were additionally tested for the presence of GAS by Alere i, and the results were compared to those of routine patient testing (by BD Veritor with reflex to culture for group A Streptococcus). The inclusion criterion was an order for a test for strep throat by a clinician. Pediatric cases (<18 years of age) accounted for 199 (92.1%) of the specimens, while adults (≥18 years of age) accounted for 17 (7.9%) of the specimens. Each patient was subjected to the collection of two rayon throat (posterior oropharynx) swab specimens as a part of their routine workup for strep throat in the clinic. Only negative BD Veritor results are routinely reflexed to the laboratory for culture confirmation, per American Academy of Pediatrics guidelines (1) and the BD Veritor package insert, but for the purposes of this study, all specimens (both those with positive and those with negative BD Veritor results) were reflexed to the laboratory for additional testing. BD Veritor testing was performed in the clinic where the patients were initially seen. The second swab was then placed in a transport system containing liquid Amies medium and was sent to the laboratory to perform routine culture for strep throat and testing with Alere i. One invalid result obtained by Alere i was not included in the study. The personnel performing Alere i were blind to the culture and BD Veritor results. Culture results were used as the “gold standard” for assay comparison (Table 1), and RT-PCR was performed on all samples with discordant results (see Table S1 in the supplemental material).

TABLE 1.

Sensitivity, specificity, accuracy, and kappa index analysis of Alere i and BD Veritor compared to culture results

Assay	Parameter	Value(s) for specimens with the following culture resulta:
		Positive	Negative	Total
Alere i	No. of samples positive	42	15	57
	No. of samples negative	0	158	158
	Total no. of samples	42	173	215
	% sensitivity (95% CI)			100.0 (91.6–100.0)
	% specificity (95% CI)			91.3 (86.1–95.1)
	% accuracy (95% CI)			93.0 (88.8–96.0)
	Kappa index (95% CI)			0.805 (0.711–0.898)
	Kappa index P value			<0.0001
Veritor	No. of samples positive	32	11	43
	No. of samples negative	10	162	172
	Total no. of samples	42	173	215
	% sensitivity (95% CI)			76.2 (60.5–87.9)
	% specificity (95% CI)			93.6 (88.9–96.8)
	% accuracy (95% CI)			90.2 (85.5–93.9)
	Kappa index (95% CI)			0.692 (0.569–0.815)
	Kappa index P value			<0.0001

^aCulture was the gold standard.

Alere i Strep A test.

The Alere i Strep A test is performed on the Alere i testing platform. It is a molecular assay that uses isothermal nucleic acid amplification for the qualitative detection of GAS from throat swab specimens from symptomatic patients. The assay is a CLIA-waived POCT that utilizes a fluorescently labeled molecular beacon for identification of amplified target sequences. This assay is performed by eluting the sample from the throat swab into elution buffer, followed by transfer of the eluted specimen into the test base. The run time of the assay is approximately 3 min for samples with a strong positive result and 8 min for samples with a negative result. The second throat swab specimen collected from the patient was used to run Alere i, after it was used to streak the bacterial culture plate, which is an off-label use of the assay. A preliminary evaluation of this off-label use, including plate swabbing procedures, was done prior to starting the study to verify that it did not compromise the detection of GAS. After streaking of the plate, the package insert instructions were followed for performing Alere i, and the throat swab was mixed in the sample elution buffer for 10 s and was removed.

BD Veritor system for rapid detection of group A Streptococcus.

The BD Veritor system for the rapid detection of group A Streptococcus is a rapid chromatographic immunoassay used for the qualitative detection of GAS antigen from posterior oropharynx swab specimens obtained from symptomatic patients. It is a CLIA-waived test used in the clinical setting for screening for GAS. The system's automated reader analyzes the assay and interprets the results. The sample from the throat swab is eluted into the reagent bottle, which is then used to aliquot the specimen into the test device well. This test device then gets inserted into the assay reader. BD Veritor requires approximately 10 min to set up and run. This test was performed in the clinic on the first swab specimen following the package insert instructions.

Bacterial culture.

Swabs were inoculated on 5% sheep blood agar and incubated for 24 h at 37°C in 5% CO₂. The plates were examined at 24 h and 48 h for signs of growth and beta-hemolytic colonies. Bacterial cultures were considered positive if GAS was isolated. Cultures were considered negative if there was no growth at 48 h or if organisms other than GAS, including group B Streptococcus, non-group A Streptococcus, and nongroupable Streptococcus, were isolated. Streptococci were grouped using latex agglutination with a Remel PathoDx strep grouping kit (Thermo Fisher Scientific, Waltham, MA).

Real-time PCR analysis.

The results for samples with discordant test results were confirmed by RT-PCR analysis on a 3M integrated cycler using a Simplexa Group A Strep Direct test (Focus Diagnostics, Cypress, CA). This test allows the direct amplification and qualitative detection of GAS DNA from throat swabs. Testing was performed by trained microbiology technologists in the microbiology laboratory as an off-label test from an aliquot collected from Alere i elution buffer, which was then stored at 4°C for ≤24 h until it was tested. This assay functioned as an arbiter between the 2 assays being evaluated (Alere i and BD Veritor) and was performed on all samples with discordant results (including culture) in the study. This assay was chosen as an arbiter since it is an RT-PCR assay that performs >40 cycles of amplification and has a very high sensitivity (97.4%) and specificity (95.2%), per the package insert.

Chart review and hypothetical impact.

Each patient's chart was reviewed in the electronic medical record, including any medications prescribed, their start date, and their duration. The metrics evaluated included the reason for antibiotic therapy (e.g., otitis media, sick contacts), other comorbidities (e.g., influenza status), the antibiotic(s) given at the time of the clinic visit or after the clinic visit, their appropriateness on the basis of laboratory results, and if the antibiotic therapy or a lack thereof was appropriate on the basis of the laboratory test results (BD Veritor and bacterial culture) and the results of the assay being investigated (Alere i). A hypothetical impact was determined on the basis of what results would have been available to the ordering provider if the test being performed in our study (Alere i) was provided and utilized in treatment decisions.

Statistical analysis.

The binomial proportion confidence intervals (CIs) for sensitivity, specificity, accuracy, and overall agreement were calculated using the Clopper-Pearson method. The confidence interval for the kappa index was calculated using the normal distribution method (11). Fisher's exact test with a P value of <0.05 was used to determine significance. SAS (version 9.3) statistical software was used for analysis.

RESULTS

A total of 215 specimens were tested by both assays (Alere i and BD Veritor) and culture. Overall, a total of 62 positive test results were obtained across all 3 assays. Thirty-two of the 62 positive results were detected by all 3 methods (Alere i, BD Veritor, and culture). Alere i detected the highest number of positive results at 57, followed by the BD Veritor at 43 and culture at 42. There were 10 results that were identified by both Alere i and culture but not detected by the BD Veritor. Six results were also detected by Alere i and BD Veritor but not by culture. Both Alere i and BD Veritor also individually detected 9 and 5 positive results, respectively, that were not detected by culture.

We evaluated the sensitivity and specificity of both assays compared to the results of culture, and our findings showed that Alere i had a sensitivity and a specificity of 100% and 91.3%, respectively, and BD Veritor had a sensitivity and a specificity of 76.2% and 93.6%, respectively, when the results were compared to those of culture (Table 1). When overall accuracy was assessed, Alere i showed the highest accuracy at 93.0%, followed by BD Veritor at 90.2%. The kappa index was used to evaluate the level of agreement for each assay compared to the results of culture. Alere i was significant (P < 0.0001) at the strongest level of agreement with a kappa index of 0.805 (95% CI, 0.711 to 0.898) compared to the culture results. BD Veritor was significant for a moderate level of agreement (P < 0.0001) compared to the culture results with a kappa index of 0.692 (95% CI, 0.569 to 0.815) (Table 1). In addition, comparison of Alere i and BD Veritor showed an agreement of 0.888 (95% CI, 0.838 to 0.927) and a kappa index of 0.689 (95% CI, 0.575 to 0.803) with a significant P value of <0.0001, indicating moderate agreement between the two assays (Table 2).

TABLE 2.

Agreement between Alere i and BD Veritor

Alere	Value(s) for samples with the following Veritor result:
	Positive	Negative	Total
No. of samples positive	38	19	57
No. of samples negative	5	153	158
Total no. of samples	43	172	215
Agreement (95% CI)			0.888 (0.838–0.927)
Kappa index (95% CI)			0.689 (0.575–0.803)
P value			<0.0001

In total, there were 30 discordant results in which one or more of the testing results (BD Veritor, Alere i, culture) did not agree. We performed RT-PCR for adjudication of each of these results. When analyzing the 15 results that were positive by Alere i but negative by culture, RT-PCR detected GAS in 14/15 (93.3%) of these specimens. In contrast, BD Veritor detected positive results in 11 specimens that were negative by culture. When these results were analyzed by RT-PCR, it was found that only 5/11 (45.5%) of these results were confirmed to be positive. In each of the 11 cases, the Alere i results agreed with the RT-PCR results (see Table S1 in the supplemental material).

Interestingly, all samples that were detected as positive by more than one of the assays being evaluated (n = 16) were also confirmed to be positive by RT-PCR. Fourteen specimens were detected as positive by only one of the assays, and only 8 (50%) of these results were confirmed with RT-PCR. Zero of 5 (0%) of the results detected solely by BD Veritor were confirmed by RT-PCR, while 8/9 (88.9%) of the results detected only by Alere i were confirmed by RT-PCR (Table S1). In total, 14 specimens that were culture negative were identified as GAS positive by RT-PCR. Alere i was also able to identify 14/14 (100%) of these specimens, while BD Veritor detected only 6/14 (42.9%) of the same specimens (Table S1).

We also performed a chart review on each of the 215 patients included in the study and analyzed antibiotic utilization in each case. In total, 73/215 (34%) patients were given antibiotics at the time of the clinic visit. Out of this group, 26/73 (36%) patients were prescribed antibiotics at the time of the clinic visit, but the treatment was likely inappropriate on the basis of confirmation of negative GAS test results (1). In 20/26 (77%) cases of potentially inappropriate treatment, all tests were negative for GAS, but the patient was prescribed antibiotics. In the case of the 5 patients with BD Veritor-positive results that were negative by all other methods (including RT-PCR), all 5 were treated with antibiotics. This represented 19% (5/26) of inappropriately treated cases. Since the results for these 5 patients were likely false positive, they were included in the inappropriate antibiotic use group.

When analyzing patients that were not prescribed antibiotics at the time of the clinic visit but likely should have been treated, there were 13/215 (6%) cases for which the BD Veritor result was negative and antibiotics were not started at the time of the clinic visit but that were subsequently detected by RT-PCR. In comparison, the Alere i result was positive in 13/13 (100%) of the same cases. In 6/13 (46%) cases, the antibiotics were started 2 to 6 days after the clinic visit. This 2- to 6-day time range includes the time when positive culture results were obtained and the clinicians followed up with the patients in order to prescribe antibiotics. One case also had antibiotics started, even though the BD Veritor and culture results were negative, due to having a history of GAS pharyngitis 4 weeks earlier. The same specimen tested positive by Alere i and was confirmed to be positive by RT-PCR. Our chart review did not reveal any clearly documented instances of adverse outcomes associated with treatment differences.

DISCUSSION

The availability of rapid molecular POCTs for the diagnosis of common infectious diseases, like GAS infection, is becoming greater. Studies comparing these tests to those currently in use will greatly assist clinics in their decision whether to switch to these new tests. Our current study compared one of these new molecular tests, Alere i, to our current instrument-read rapid antigen POCT method, BD Veritor. Culture was used as the gold standard of comparison, with RT-PCR also being performed for adjudication in cases where the results of all 3 tests (Alere i, BD Veritor, culture) did not agree. While other studies comparing molecular POCT methods to a mixed group of rapid antigen tests, culture (12), and a lab-developed test (13) and Alere i to culture (14) have been performed, to our knowledge, this is the first study comparing Alere i and a rapid antigen test head-to-head and analyzing the hypothetical impact on antibiotic utilization based on the test results.

We found that Alere i had 100% sensitivity compared to the results of culture, while BD Veritor reached only 76.2% sensitivity. Statistical analysis also showed that BD Veritor and Alere i showed only moderate agreement. In the discordant results group, 14 specimens were found to be positive by only one of the assays. Interestingly, the results for none of the 5 positive specimens (0%) detected only by BD Veritor were confirmed by RT-PCR, while the results for 8/9 (88.9%) positive specimens detected by Alere i were confirmed by RT-PCR. In the case of BD Veritor, it is very unlikely that these results were true positives due to the fact that these samples were tested by a more sensitive assay (the Focus Diagnostics RT-PCR assay, by which 97.4% of the specimens were positive, per the FDA-approved package insert) and were negative. It is also possible that the one positive specimen that was detected by Alere i but that was not detected by RT-PCR was false negative, but the stated sensitivity for Alere i is 95.9% and therefore lower than that of the Focus Diagnostics RT-PCR assay. Alere i was also able to identify 14/14 (100%) specimens that were missed by culture, while BD Veritor picked up only 6/14 (42.9%) of the same positive specimens. Of interest, there was one case of an untypeable beta-hemolytic Streptococcus sp. (not group A, B, C, F, or G) documented, and this specimen was identified to be GAS positive by BD Veritor but GAS negative by all other testing methods, suggesting a possibility of cross-reactivity.

Along with our instrument comparisons, we also performed a comprehensive chart review for each patient to determine tbe antibiotic usage resulting from the clinic visit. The Infectious Diseases Society of America (IDSA) guidelines for the diagnosis and management of GAS pharyngitis state that the inappropriate administration of antibiotics should be avoided if GAS is excluded in the diagnosis (1). Proper antibiotic utilization in GAS infection is especially critical in the pediatric population, where the start of therapy earlier can limit the spread of disease, reduce symptom duration, limit follow-up losses, allow resumption of normal activity, and limit the development of more serious secondary conditions, such as rheumatic heart disease.

In our analysis, we found that 36% (n = 26) of the patients were given unnecessary antibiotics at the time of their visit, based on confirmed negative test results for GAS. Of these patients, 77% (n = 20) had negative test results by all assays. All 5 of the patients that had false-positive results for GAS from BD Veritor were also given antibiotics; these represent 19% of patients that received unnecessary antibiotics. This is the test currently utilized in our clinics, and as such, treatment decisions are based on its results. Since these were positive results, they were not reflexed for culture per IDSA guidelines (1) and the FDA-approved BD Veritor package insert and would have typically been missed. Conversely, 6% (n = 13) of the positive cases were missed by BD Veritor, while Alere i detected all 13 (100%) of these cases. Antibiotics were started 2 to 6 days after the visit in 6 (46%) cases, with one patient being lost to documented follow-up. The remaining 6 (46%) patients were culture negative and were therefore not treated but were confirmed to be positive by RT-PCR. The use of Alere i could have potentially led to these 6 (100%) missed patients being treated.

One recent study analyzing POCT for respiratory viruses reported that inappropriate antibiotic use was not curtailed, but this was due to patients being started on antibiotics before POCT results (which take at least an hour to receive on the platform used in that study) were available (15). This scenario would likely not occur for POCT for GAS done in the clinic, since Alere i takes approximately 8 min from start to finish. Some limitations are that the study was not able to be randomized to prevent bias in order to avoid an impact on routine patient testing. Also, Alere i was performed from the same swab specimen used to inoculate the agar for the detection of strep throat. With these caveats in mind, the superior performance of Alere i compared to BD Veritor, which had the advantage of a dedicated swab for testing, is striking.

In this study, we showed that Alere i had a performance superior to that of BD Veritor when they were used to diagnose GAS infections, which could assist in the better utilization of antibiotics in real time. This new molecular platform should be considered a viable alternative POCT device for the diagnosis of GAS pharyngitis.