Sensitivity and specificity of rapid influenza testing of children in a community setting

Abstract

Please cite this paper as: Stebbins et al. (2011) Sensitivity and specificity of rapid influenza testing of children in a community setting. Influenza and Other Respiratory Viruses 5(2), 104–109.

Introduction  Rapid influenza testing (RFT) allows for a rapid point‐of‐care diagnosis of influenza. The Quidel QuickVue^® Influenza A+B test (QuickVue) has a reported manufacturer’s sensitivity and specificity of 73% and 96%, respectively, with nasal swabs. However, investigators have shown sensitivities ranging from 22% to 77% in community settings.

Methods  The QuickVue rapid influenza test was evaluated in a population of elementary (K‐5) school children, using testing in the home, as part of the Pittsburgh Influenza Prevention Project during the 2007–2008 influenza season. The QuickVue test was performed with nasal swab in full accordance with package instructions and compared with the results of nasal swab semi‐quantitative RT‐PCR.

Results  Sensitivity of the QuickVue was found to be 27% in this sample. There was no statistically valid correlation between the semi‐quantitative PCR result and the QuickVue result.

Conclusions  This study is consistent with the low sensitivity of the QuickVue test also reported by others. Viral load, technique, and the use of nasal swabs were examined as contributing factors but were not found to be explanations for this result. Community testing includes patients who are on the lower spectrum of illness which would not be the case in hospital or clinic samples. This suggests that RFT is less sensitive for patients at the lower spectrum of illness, with less severe disease.

Introduction

Rapid influenza testing (RFT) allows for a rapid point‐of‐care diagnosis of influenza, which can improve medical management by guiding the appropriate use of antivirals and antibiotics. ¹ , ²

The performance of the QuickVue rapid influenza test, a Clinical Laboratory Improvement Amendments (CLIA)‐waived, point‐of‐care immunoassay, was evaluated as a part of the Pittsburgh Influenza Prevention Project (PIPP). The QuickVue test was evaluated in a population of elementary (K‐5) school children from 10 different schools in the Pittsburgh metropolitan area. Students with an influenza‐like illness (ILI) were enrolled in the study during the 2007‐2007 influenza season and tested at home.

The Quidel QuickVue^® Influenza A+B test (QuickVue) has a reported manufacturer’s sensitivity and specificity of 73% and 96%, respectively, when nasal swabs are utilized. ³ Several studies have noted a similar performance of this test in emergency room and physician office settings, with sensitivities of 74% and 82%. ⁴ , ⁵ However, investigators have shown sensitivities ranging from 22% to 77% (22%, 77%, 29%, and 27%) when the test was utilized in community settings. ⁶ , ⁷ , ⁸ , ⁹

The PIPP study population consisted entirely of community elementary school age children. This offered the opportunity to assess the performance of rapid influenza tests in this population and in the home setting.

Methods

Study protocol

The Pittsburgh Influenza Prevention Project (PIPP), a cluster‐randomized trial, was designed to assess the impact of non‐pharmaceutical interventions (NPIs) on the incidence of laboratory‐confirmed influenza A/B infections among children in ten Pittsburgh K‐5 elementary schools. A cluster design was adopted because of logistical constraints of performing the intervention within smaller units of each school and to reduce intervention‐control crossover.

Thirteen Pittsburgh elementary schools initially expressed interest in the study; the largest 10 were selected and assigned to either intervention or control arms by a constrained randomization algorithm using data from the 2006–2007 school year. Five schools were randomized to each group, balancing covariates that might be associated with the primary outcome while maintaining an unbiased and valid randomization. All enrolled students were considered eligible for the study. Parents and guardians were educated about the study at the beginning of the school year and were given the opportunity to decline participation. ¹⁰

The evaluation period for this study spanned a 4 month period from January 7 through April 25, 2008 which corresponded to the beginning and end of detectable influenza activity in Southwestern Pennsylvania during the 2007‐2008 influenza season.

Training of PIPP project specialists

A unique feature of PIPP was that all of the sample collection, transport, and testing was performed by trained influenza project specialists who were not certified health care providers (i.e., physicians, laboratory technicians, nurses, etc.). Project specialists received a formal 2‐week class on all aspects of their job. This included one full day of detailed instruction by a physician on the collection of nasal swab samples for influenza testing. Further mock home visits and accompanied home visits were also conducted with each staff member individually to assure proper sample collection technique.

Influenza testing procedure

During the influenza season, families of children reported absent from school were contacted by telephone. A screening questionnaire was administered to determine if the reason for absence was an ILI, which was defined using the standard US Centers for Disease Control and Prevention (CDC) case definition of acute onset of fever (>100°F), and cough and/or sore throat. ¹¹ Reported ILIs were followed up with a home visit which included informed consent by the parent/guardian and assent by the child.

Two nasal swabs were collected at the home visit from the child with reported ILI. The first swab was used for the QuickVue test (performed immediately in the home), and the second swab was placed in M4 viral transport medium and transported in a cold pack box to the University of Pittsburgh Viral Laboratory for confirmatory RT‐PCR testing.

Quickvue test

The QuickVue test detects influenza antigens in clinical specimens and provides results in 10 min. This test is a lateral‐flow immunoassay that uses monoclonal antibodies specific for influenza viral nucleoprotein antigens and detects and differentiates between influenza A and B. The test uses an extraction reagent to disrupt virus particles in the clinical specimen. Viral nucleoproteins are subsequently exposed to and react with an antibody‐coated strip. If influenza virus is present, a pink to red test line in either the “A” or “B” position, along with a blue procedural control line, appears on the test strip. The presence of a clearly demarcated blue line provides a built‐in control.

QuickVue testing was performed and interpreted according to the manufacturer’s instructions (package insert). ³ A physician verified staff results within 24 h. In the few instances where a result was ambiguous, the physician was contacted to make a determination.

PCR test

The diagnosis of influenza was confirmed by real‐time, reverse transcriptase polymerase chain reaction (RT‐PCR) at the University of Pittsburgh Virology Laboratory, Pittsburgh PA. Isolation of viral nucleic acid from control material and patient specimens was carried out using an EasyMag automated extractor (bioMérieux, Durham, NC, USA) according to the manufacturer’s instructions. A volume of 200 ul of specimen was added to lysis buffer (bioMerieux) for 15 min, to which a fixed volume and concentration of equine arteritis virus (EAV; 10^6·25 TCID₅₀/ml) was added as internal control for extraction and amplification, followed by addition of magnetic silica as per the manufacturer’s instructions. For RT‐PCR, influenza A, influenza B, and influenza A H1 and H3 primer and probe sequences used were made available by CDC to the Laboratory Response Network. ¹² [NOTE: The primer details are available upon request]. Each PCR reaction consisted of 25 ul volume with 12·5 ul of 2 ×  Quantitect multiplex RT‐PCR NR (Qiagen, Valencia, CA) master mixes with RT‐enzyme (Qiagen) for RNA viral targets and 2 ×  Quantitect multiplex PCR NR (Qiagen) master mixes for DNA viral targets. Primers and probe concentration for viral targets in each panel were optimized for optimal sensitivity. PCR thermal cycling and detection was performed using the ABI 7500 real‐time PCR instrument.

IRB

Institutional Review Board approval for the study was obtained from the University of Pittsburgh IRB, Pittsburgh Public Schools IRB, and the CDC IRB. The researcher from Johns Hopkins University (Cummings) did not have access to personally identified data and did not engage in human subject research.

Statistical analyses

The relationship between a positive result with QuickVue and RT‐PCR testing was examined. RT‐PCR testing was assessed and reported as a semi‐quantitative value of 1, 2, or 3, for positive influenza A or B. This was based on the cycle threshold (CT) value, which measures the strength of the result. The CT value is the cycle number at which the fluorescence from a PCR crosses the threshold. This inversely correlates (logarithmically) to the initial copy number or original template in the sample. The CT value assigned to a particular sample is the point during the PCR at which a sufficient number of amplicons have accumulated. This scale gives a semi‐quantitative indication of the strength of the PCR signal detected per sample. The reported scale corresponds with positive influenza A/B RNA template values of: ≤25 = (3); >25–35 = (2); and ≥36 = (1). A lower CT value indicates detection of RNA template at an earlier cycle, therefore, more virus/viral particles in the original sample. If there is a higher CT value, there is less RNA or virus in the original sample.

Sensitivity and specificity analyses were performed for RFT using the RT‐PCR as the gold standard. Fisher’s exact test was used to determine the association between RFT and semi‐quantitative RT‐PCR results. The Fisher’s exact test was used because of cell sizes <5. To test the strength of association between an increasing value of the semi‐quantitative RT‐PCR and the RFT outcome, a logistic regression was performed using the RFT outcome as the binary outcome. For these analyses, statistical significance was assessed at the 0·05 level. All analyses were performed using the R software (R version 2.8.1). ¹³

Results

During the study period, 278 pairs of swabs were collected from children during home visits. All samples were collected from children in grades K through 5. A total of 104 samples were positive by RT‐PCR test (Table 1). Of the 104 flu positive, 54/104 (52%) were influenza A and 50/104 (48%) were influenza B, and 28/104 (27%) of these influenza‐positive samples were detected as positive by rapid influenza testing. Influenza A accounted for 17 (61%) of the 28 positive tests. (Table 2). The remaining 11 (39%) were influenza B.

Table 1.

 Results of Influenza testing using QuickVue and RT‐PCR (n = 278)

Number of specimens	RT‐PCR
Rapid test	Positive	Negative	Total
Positive	28	6	34
Negative	76	168	244
	104	174	278

Table 2.

 QuickVue and RT‐PCR results by influenza subtype

RT‐PCR
RFT	Pos Flu A	Pos Flu B	Neg	Total
Pos	17	11	6	34
Neg	37	39	168	244
Total	54	50	174	278

Compared to RT‐PCR, the sensitivity and specificity of rapid influenza testing using QuickVue in the students with ILI who were tested was 27% (28/104) and 97% (168/174), respectively. The overall positive predictive value and negative predictive values were 82% (28/34) and 69% (168/244), respectively. The likelihood ratio for a positive test (LRP) and the likelihood ratio of a negative test (LRN) were 7·81 ([18/104]/[1‐(168/174)]) and 0·76 ([1‐(28/104)]/[168/174]), respectively. The sensitivity of rapid flu testing for influenza by type was 31% and 22% for influenza A and B, respectively.

A semi‐quantitative measure of viral load was obtained for all positive RT‐PCR influenza tests, as described above. 3, 4, 5 display the overall results and the individual results for influenza A and B.

Table 3.

 Semi‐quantitative RT‐PCR value by result for all positive flu tests for Influenza RT‐PCR RFT positives

RFT result	Semi‐quantitative RT‐PCR value
	1	2	3
Number of positive RT‐PCR tests	14	76	14
Number of positive RFT tests (%)	3 (21)	19 (25)	6 (43)

Table 4.

 Semi‐quantitative RT‐PCR value by result for positive flu tests for Influenza A RT‐PCR positives

RFT result	Semi‐quantitative RT‐PCR value
	1	2	3
Number of positive RFT tests	3	11	3
Number of negative RFT tests	7	28	2

Table 5.

 Semi‐quantitative RT‐PCR value by result for positive flu tests for Influenza B RT‐PCR positives

RFT result	Semi‐quantitative RT‐PCR value
	1	2	3
Number of positive RFT tests	0	8	3
Number of negative RFT tests	4	29	6

Two analyses were performed, both including and excluding RT‐PCR negative results. For analyses including the RT‐PCR negative tests, there was a statistical significant association between RFT and RT‐PCR (Fisher’s exact test P < 0·001). When only the PCR‐positive tests are used, there was no statistical association between RFT and PCR (Fisher’s exact test P < 0·37).

A logistic regression using the RFT outcome as the binary outcome and the semi‐quantitative results for each strain as the explanatory variables was performed. Negative RT‐PCR results were included and coded as 0. There was a statistically significant association between both semi‐quantitative RT‐PCR influenza A (OR 3·3 95% CI 2·1, 5·2) and semi‐quantitative RT‐PCR influenza B (OR 2·4 95% CI 1·6, 3·8). These results suggest that for every unit increase in semi‐quantitative result for influenza A there is a 3·3‐fold increase in the odds of RFT being positive. These results are dominated by the inclusion of the negative results (63%) and the increased odds of positive RFT when RT‐PCR is positive. When the negative RT‐PCR is excluded, logistic regression shows a non‐statistical significant association with both influenza A semiquantitative scores (OR 1·7 95% CI 0·54, 5·7) and influenza B semiquantitative scores (2·4 95% CI 0·64, 10·10). The results of both the Fisher’s exact test and the logistical regression are consistent.

Discussion

Viral cell culture and more recently PCR assays for viral RNA serve as the gold standard tests for national influenza surveillance. However, these techniques are labor‐intensive and require specialized laboratory skills. This limits their widespread use, especially in community settings ¹⁴ where the vast majority of influenza cases occur. A rapid flu test with good sensitivity and specificity would improve surveillance and point‐of‐care decision making. For Quidel’s QuickVue, the reported sensitivity and specificity from the manufacturer’s package insert and website for nasal swabs is 73% and 96%, respectively. ³ In controlled laboratory trials and in medical practices, rapid influenza tests have generally demonstrated moderate sensitivity (45—90%) and good specificity (86%–100%) to detect influenza virus infection. ¹⁵ , ¹⁶

Analysis of semi‐quantitative RT‐PCR values and RFT outcomes finds no relationship between these values. This suggests that there is no relationship between the viral antigen and the outcome of the RFT, within the range that was tested, in this community setting.

The specificity of QuickVue rapid influenza testing in the study sample was consistent with the manufacturer’s literature and with previous studies (97%). However, the sensitivity of rapid influenza testing using QuickVue in a community setting with school children who have ILI compared with RT‐PCR was consistent with numerous other community‐based studies where similar sensitivities (i.e., 22%–29%) have been reported. ¹⁷ , ¹⁸ , ¹⁹ It was much lower when compared to other studies including notably Simmerman et al. ¹⁴ which reported a sensitivity of 77%. Simmerman’s work was conducted in rural Thailand, and the authors considered that this would present conditions similar to other studies reporting low sensitivity. However, it was carried out in the outpatient department of rural hospitals and by research nurses, and so may be more like other studies focusing on patients seen in medical settings.

The lower sensitivity profile in our results could be because of the use of nasal swabs instead of nasal aspirates/washes and the innate superiority of RT‐PCR compared to conventional viral culture as gold standard. ²⁰ However, all tests were carried out according to the manufacturer’s instructions. The manufacturer specifies slightly lower values for nasal swab versus nasal aspirates – 73% versus 81%, but these previous results are in stark contrast to the sensitivity of the test found in our study. The high values of the semi‐quantitative RT‐PCR tests for samples obtained in this study suggest that samples obtained by nasal swab can yield strongly reactive samples and successful test results.

Testing by non‐medical project specialists in households raises the possibility that operator error may have contributed to the low sensitivity. This is felt to be unlikely for the following reasons: (i) Staff received a full day of training by a physician, performed this test several hundred times, and were followed closely during influenza season and (ii) This is a CLIA‐waived test, defined by FDA regulations (42 CFR part 392) “as simple laboratory examinations” that “employ methodologies that are so simple and accurate as to render the likelihood of erroneous results negligible”. ²¹

One systematic error was identified and corrected early in the study as noted below.

Out of a total of six false‐positive results (positive RFT and negative RT‐PCR), four occurred during the first month of operation and three were traced back to the same school and were submitted by the same project specialist. These appeared to result from misinterpretation of a pink hazy color as a positive test, as opposed to the sharply defined pink line that the manufacturer required. It was resolved that all results in the absence of a sharply defined pink line would be called negatives. A physician confirmed all results, and this problem did not occur again. It is worth noting that this error would not change the sensitivity in any event, even if repeated.

Conclusion

These results verify the findings of Rashid et al., Stein et al., and Uyeki et al., with regard to low sensitivity of the QuickVue RFT. This study goes beyond this, with the semi‐quantitative PCR results showing that swabs taken by trained project staff in community and home settings had sufficient viral load to invoke a positive test – in other words, capture flu virus.

The populations tested may explain the difference in sensitivity between this community use of the RFT and that reported by the manufacturer and in clinical use. Influenza infection exhibits a broad difference in severity of symptoms. Community testing includes patients who represent the entire spectrum of illness, with the majority exhibiting mild to moderate symptoms and not requiring medical attention. Those reporting to hospitals and clinics are MORE COMMONLY significantly or seriously ill. None of the subjects of this study were hospitalized. This suggests that RFT are less sensitive at the lower spectrum of illness outside of the few flu cases that come to the attention of medical professionals and receive appropriate testing. Future study that quantified viral load and RFT in hospital patients would be of interest to verify this observation.

This study suggests that the QuickVue test (and likely the current generation of rapid influenza tests) has multiple limitations. First, the manufacturer’s reported sensitivity is overly optimistic when considering the entire population, including persons with mild symptoms who may not present to a healthcare facility for evaluation. Second, the low sensitivity of this test limits its use as a “point‐of‐care” diagnostic tool, especially in non‐clinical care settings, and third, the public health system continues to lack reliable ways to identify most cases of influenza. If tests with higher sensitivities in community settings (such as schools, day cares, homes, and pharmacies) were available, they could be used as a part of an improved surveillance and “early warning” system for influenza activity and RAPID CONFIRMATION OF OUTBREAKS.