EVALUATION OF A RAPID ANALYTE MEASUREMENT PLATFORM AND REAL-TIME REVERSE-TRANSCRIPTASE POLYMERASE CHAIN REACTION ASSAY WEST NILE VIRUS DETECTION SYSTEM IN MOSQUITO POOLS

Abstract

We evaluated the commercially available Rapid Analyte Measurement Platform (RAMP^®) West Nile Virus (WNV) antigen detection test for sensitivity and consistency with real-time reverse transcriptase polymerase chain reaction (RT-PCR) confirmation testing. Panels of samples consisting of WNV-spiked mosquito pools and negative control pools were sent to 20 mosquito abatement districts (MADs) that processed the pools using the RAMP assay. The samples were then sent to the reference laboratories used by the MADs for confirmation by real-time RT-PCR. Positive pools with virus titers of roughly 1-3 log₁₀ PFU/ml had RAMP scores above the RAMP test positive cutoff score of 30 RAMP units, but these virus-positive samples could not be reliably confirmed by real-time RT-PCR testing. Pools with virus titers ≥ 4 log₁₀ PFU/ml scored ≥ 50 RAMP Units. Real-time RT-PCR results varied among the confirmation laboratories. With few exceptions, pools returning a RAMP score of ≥ 100 were confirmed with real-time RT-PCR, while pools returning a RAMP score of 50-99 appeared to be at the limit of real-time RT-PCR detection. Therefore, we recommend using a positive cut-off of 50 RAMP units with no real-time RT-PCR confirmation to maximize speed, efficiency, and economy of the RAMP assay. A more conservative approach would be to implement a “grey zone” range of 50-100 RAMP units. Pools scoring within the grey zone could be submitted for real-time RT-PCR confirmation with the understanding that positive pools may not confirm due to the inhibitory effect of the RAMP buffer on the real-time RT-PCR assay. We also conducted a series of experiments using laboratory-prepared mosquito pools spiked with WNV to compare mosquito homogenization buffers, pool sizes, and grinding methods in order to determine how these variables affect the RAMP and real-time RT-PCR assay results.

Introduction

Since the 1999 introduction of West Nile Virus (WNV) in the United States and subsequent expansion of the virus across the country, health departments and mosquito abatement districts (MADs) have established or intensified existing mosquito-based arbovirus surveillance programs. Traditional methods for arbovirus surveillance include reverse-transcriptase polymerase chain reaction (RT-PCR) assays (Kaufman et al. 2003), cell culture plaque assays (Beaty et al. 1995), and enzyme immunoassays (Hunt et al. 2002). Rapid procedures, such as automated high-throughput arboviral RNA extraction protocols and real-time RT-PCR assays (Lanciotti et al. 2000), were developed and utilized to ensure efficient virus detection when testing the large numbers of mosquito pools required to accurately estimate the infection rate in mosquito populations (Gubler 2000). While these assays are excellent tools for arbovirus surveillance, they require significant monetary resources, technical expertise, and appropriate laboratory equipment and space.

Aided by Epidemiology and Laboratory Capacity (ELC) funding from the Federal government, state and local health departments expanded arbovirus surveillance. However, due to budget reductions over the years, many laboratories limited testing capacity or ceased mosquito pool testing altogether. In addition to the budget reductions, the time between submitting samples for testing and receiving the results was viewed as a significant limitation to using central laboratories for mosquito pool testing. Due to sample transport, implementation of weekly batch-processing schedules by the labs, and the time required to run the test, the lag from collection to test results can easily be a week or longer.

Since MADs often make mosquito control decisions using mosquito-based arbovirus surveillance data, these limitations were troublesome. Commercial companies responded by developing WNV antigen detection assays that are accurate, rapid, and available to entities with limited technical and laboratory resources. One such assay, the Rapid Analyte Measurement Platform (RAMP^®) test developed by Response Biomedical (Response Biomedical Corp., Burnaby, British Columbia, Canada), was adopted by many MADs across the US. Briefly, up to 50 mosquitoes are homogenized in a proprietary RAMP grinding buffer and centrifuged. An aliquot of the resulting supernatant is mixed with a conjugated-antibody complex and added to the test cartridge that contains an immunochromatographic test strip. During a 90-minute incubation period the sample migrates along the strip to the detection and internal control zones. The antigen-bound antibody complex is captured at the detection zone, and excess antibody conjugate is captured at the internal control zone. The cartridge is inserted into the RAMP reader which reads and calculates the ratio between the fluorescence emitted at each zone to produce the results in RAMP units between 0 and 640. Mosquito pools producing ≥ 30 RAMP units are considered WNV positive according to the RAMP kit insert. Most MADs then send positive RAMP samples to laboratories for real-time RT-PCR confirmation.

While many MADs incorporated the RAMP assay into their surveillance programs successfully, some programs observed that samples which were RAMP positive in their labs produced negative results when tested with real-time RT-PCR in confirmation labs and sought advice from the manufacturer and the Centers for Disease Control and Prevention (CDC). Previously we had suspected that real-time RT-PCR may be inhibited by the grinding solution used for the RAMP assay and suggested a modified extraction protocol incorporating extra washes (Burkhalter et al. 2006), but due to the wide variability in sample handling, storage, and processing protocols among MADs and confirmation laboratories, we were unable to definitively determine the cause of the incongruent RAMP and real-time RT-PCR results. To address these discrepancies, we conducted several experiments using laboratory-prepared mosquito pools. To examine the WNV detection and confirmation system frequently incorporated by MADs and laboratories, and to determine a useful RAMP positive cut-off score, we created proficiency panels consisting of pools of whole mosquitoes spiked with known quantities of WNV and negative control pools to be tested by MADs and their confirmation labs by the RAMP and real-time RT-PCR assays. In addition, we conducted a series of in-house experiments at the CDC on laboratory-prepared mosquito pools spiked with WNV to compare grinding buffers, pool sizes, and grinding methods in order to determine how these variables affect the RAMP and real-time RT-PCR assay results.

Materials and Methods

Proficiency Panel Preparation

Each proficiency panel consisted of 10 pools of 50 laboratory-reared Culex quinquefasciatus Say, each containing an aliquot of medium that was either positive or negative for WNV. Ten-fold serial dilutions of high-titered stock WNV strain CO08-10517 harvested from a pool of field-collected mosquitoes were prepared in RAMP buffer. These 7 dilutions contained virus titers of approximately 1-7 log₁₀ PFU/ml. Aliquots of 100 μl of each virus dilution were added to 7 mosquito pools to generate the positive pools for each panel. Aliquots of 100 μl of uninfected RAMP buffer were added to 3 pools of each panel to serve as negative controls. Panels were blind-coded and stored at −70 °C until shipped on dry ice to 20 participating MADs in Idaho, Illinois, Washington, California, South Carolina, Virginia and Utah. An additional 5 panels were tested in the CDC laboratory in Ft. Collins, Colorado to provide a standard for comparison with the MAD and reference laboratory results.

CDC Proficiency Panel Testing Methods

The following procedures were used to test the 5 sample panels evaluated by RAMP and real-time RT-PCR at the CDC lab. One ml RAMP buffer and 1 copper-coated BB were added to each pool. Pools were homogenized for 4 min at 25 cycles/sec in a Qiagen Mixer Mill MM 300 (QIAGEN, Inc., Valencia CA), and centrifuged for 3 min at 4000 rpm in a refrigerated centrifuge. Each pool was tested using the RAMP assay according to the manufacturer’s protocol in triplicate to determine the variation in RAMP scores for a given sample.

After processing the pools for the RAMP assay, RNA was extracted from two replicate aliquots from each pool and each extract was tested for WNV by real-time RT-PCR with two primer sets in duplicate (resulting in 8 real-time RT-PCR results for each pool of each panel). RNA was extracted from a 100-μl aliquot of supernatant from each pool using Qiagen’s QIAmp Virus BioRobot 9604 kit on a Qiagen BioRobot 9604 according to the manufacturer’s protocol. Five μl of each extraction were added to primers and probes specific to the 3’ non-coding (NC) and envelope (ENV) regions of the WNV genome and reagents of Qiagen’s Quantitect Probe RT-PCR kit, and subjected to 45 amplification cycles in the Bio-Rad Icycler IQ™ Real-time Detection system (Bio-Rad, Hercules, CA) according to previously described cycling conditions (Lanciotti et al. 2000). Results are reported as the Cycle Threshold (CT) value for each real-time RT-PCR run (pools are considered positive if the CT score is ≤ 37).

MAD proficiency panel testing methods

We requested that each MAD perform the same processing and RAMP assay protocols on the panel samples as they would on their regular field-collected mosquito pools. MADs recorded protocol data and test results in a data collection form included with each panel. With one exception, all MADs homogenized the pools by vortexing the samples with 2 BBs in 1 ml RAMP buffer (vortexing intervals ranged from 1.5 – 12 minutes), and performed the RAMP assay according to the manufacturer’s protocol. One MAD, according to their state’s established protocol, ground their pools in a virus-transport medium provided by their state’s health department and added 60 μl of supernatant to 60 μl RAMP buffer, and used this 120 μl aliquot to perform the RAMP assay as described above. All MADs forwarded the processed pools to their respective confirmation labs; MADs within a state sent their panels to the same confirmation lab.

Confirmation Lab Proficiency Panel Testing Methods

The confirmation labs performed RNA extractions on each pool using comparable RNA extraction kits by Qiagen^® and Ambion^® (Life Technologies, Grand Island, NY). Each lab used primers and probes specific to either the 3’ non-coding (NC) or envelope (ENV) regions of the WNV genome to perform real-time RT-PCR on each pool (Lanciotti et al. 2000); some labs tested each pool with both primer sets. The confirmation labs used similar real-time RT-PCR kits from Qiagen^®, Ambion^® and Invitrogen™ (Life Technologies, Grand Island, NY), and the thermocycler protocols used were nearly identical and as described previously (Lanciotti et al. 2000).

Processing Variables Experiment Methods

Samples of varying pool sizes (i.e., number of mosquitoes in the pool) were processed at the CDC by two grinding methods (vortex for 1 min, Mixer Mill for 4 min at 25 cycles/sec) in two buffers (BA-1 cell culture medium, RAMP buffer) to evaluate the effects of these variables on RAMP and real-time RT-PCR detection of WNV.

Ten-fold serial dilutions were made with high-titered stock WNV strain CO08–10142 containing virus titers of approximately 3.5 – 6.8 log₁₀ PFU/ml and 100 μl of each dilution was added to pools containing 1, 3, 5, 10, 15, 20, 25, and 50 lab-reared Cx. quinquefasciatus. Aliquots of 100 μl of RAMP buffer without virus were added to additional pools that served as negative controls. One copper-clad BB and 900 μl RAMP buffer were added to pools of each viral titer and the negative controls. Half the pools of each titer were ground by Vortex (1 min) and the other half were ground by the Qiagen Mixer Mill MM 300 (4 min at 25 cycles/sec). After homogenization, all pools were centrifuged for 3 min at 4000 rpm in a refrigerated centrifuge and aliquots of supernatant were removed for RAMP and real-time RT-PCR testing performed according to CDC’s proficiency panel processing methods described above.

Ten-fold serial dilutions were made with high-titered stock WNV strain CO08–10142 containing virus titers of approximately 3.5 – 6.5 log₁₀ PFU/ml and 100 μl of each dilution was added to pools containing 1, 5, 10, 25, and 50 lab-reared Cx. quinquefasciatus Say. Aliquots of 100 μl of BA-1 cell culture medium without virus were added to additional pools that served as negative controls. One copper-clad BB and 900 μl BA-1 were added to pools of each viral titer and the negative controls. Half the pools of each titer were ground by Vortex (1 min) and the other half were ground by the Qiagen Mixer Mill MM 300 (4 min at 25 cycles/sec). After homogenization, all pools were centrifuged for 3 min at 4000 rpm in a refrigerated centrifuge and aliquots of supernatant were removed for subsequent real-time RT-PCR testing as described above in the CDC’s proficiency panel processing methods.

Statistical Analysis Methods

A linear model was used for the RAMP assay analysis, including estimated titer, grinding method and mosquito pool size as predictors, the logarithm of the RAMP values as the response variable, and the variance was modeled as a power function of the estimated titer for different pool sizes (Pinhiero and Bates 2000). This model was fitted using generalized least squares. A linear model was used for the real time RT-PCR assay analysis, including the same predictors as in the RAMP assay analysis with the addition of buffer type, and the real-time RT-PCR CT values as a response. The best fitting model was chosen using restricted likelihood ratio tests (RLRT) to compare models. Diagnostic plots were created to ascertain appropriateness of model assumptions and fit. Once a best fitting model was selected, to characterize results using a measure of relevant practical application, we estimated the sensitivities of the assays by estimating the proportion of RAMP scores expected to exceed 50 and real-time RT-PCR CTs expected to be ≤ 37, for various estimated titer and pool size combinations. Estimates of the sensitivities and 95% CIs were computed using estimates from the regression models described, as detailed in the appendix. Analyses were performed in R (www.r-project.org, ver. 2.15.3) and TIBCO Spotfire S+ (version 8.2.0, TIBCO Software, Inc., Somerville, MA).

Results

Proficiency Panel Results – RAMP Assay

Results of the Proficiency Panel RAMP assays are presented in Fig. 1. Data from one MAD which performed the alternate processing and RAMP protocol described above were excluded from the overall RAMP score analysis. All samples containing ≥ 4 log₁₀ PFU/ml WNV produced results > 50 RAMP units. Five of the 24 panel samples containing ~3 log₁₀ PFU/ml WNV produced RAMP results > 30; the rest scored < 30. No samples containing ~3 log₁₀ PFU/ml scored > 50. With 2 exceptions, negative pools and pools containing approximately 1-2 log₁₀ PFU/ml WNV did not produce RAMP scores > 30. One negative pool (of 72 total) produced results of 46.5 and 33.0 in 2 of the 3 replicates (the third repeat scored < 30). One pool containing ~2 log₁₀ PFU/ml produced a result of 640. None of the negative control pools produced a result ≥ 50.

Fig. 1.

RAMP assay results from the Centers for Disease Control and Prevention (CDC) and mosquito abatement district (MAD) participants in the Proficiency Panel evaluation. Results are shown in RAMP units averaged from each sample if repeat tests were performed. The manufacturer’s recommended RAMP score of ≥ 30 RAMP units as the positive cut-off for mosquito pools is represented by the dashed line; our proposed cut-off of ≥ 50 RAMP units is represented by the solid line. CDC RAMP results are shown in black; MAD RAMP results are shown in gray.

Proficiency Panel Results – Real-time RT-PCR Assay

Real-time RT-PCR results varied between labs, as seen in Fig. 2. Using a CT value of > 37 as the negative real-time RT-PCR cut-off (Lanciotti et al. 2000), CDC and 4 confirmation labs were unable to detect WNV in negative pools and pools containing ≤ 3 log₁₀ PFU/ml. The other three confirmation labs reported equivocal or positive results for some of the negative pools; the same confirmation labs also reported equivocal or positive results in some pools containing ≤ 3 log₁₀ PFU/ml. CDC and 4 confirmation labs were able to detect WNV in pools containing ≥ 4 log₁₀ PFU/ml. The other three confirmation labs returned equivocal or negative results for pools containing ~4 log₁₀ PFU/ml and 2 of these labs returned equivocal or negative results for pools containing ~5 log₁₀ PFU/ml. All labs were able to detect WNV in pools containing ~6 log₁₀ PFU/ml.

Fig. 2.

RAMP and real-time reverse transcriptase polymerase chain reaction (RT-PCR) results from the Centers for Disease Control and Prevention (CDC), mosquito abatement district (MAD) and confirmation laboratory participants in the Proficiency Panel evaluation. Each West Nile virus (WNV) sample titer is graphed on a separate panel, showing the comparison of mean RAMP results and mean real-time RT-PCR results. Real-time RT-PCR cycle threshold (CT) values ≤ 37 are considered positive (represented by the vertical dashed line in each panel). The manufacturer’s recommended RAMP score of ≥ 30 RAMP units as the positive cut-off for mosquito pools is represented by the horizontal dashed line; our proposed cut-off of ≥ 50 RAMP units is represented by the horizontal solid line. CDC results are shown as black diamonds (◆); MAD and confirmation labs’ results are shown as gray diamonds (◆).

The comparison of RAMP results and CT values is also depicted in Fig. 2. With few exceptions, pools scoring ≥ 100 RAMP units contained ≥ 4 log₁₀ PFU/ml and were confirmed with real-time RT-PCR. Three pools scoring between 50 and 100 RAMP units were confirmed by real-time RT-PCR; however, the RAMP results were closer to 100 than 50 (81.3, 86.6, and 93.7) and pools scoring less than these were not confirmed by real-time RT-PCR in the proficiency panel evaluation.

Processing Variables Experiment Results – RAMP assay

The RAMP results for the evaluation comparing grinding methods and pool sizes conducted at CDC are shown in Fig. 3. For samples containing 3.8 log₁₀ PFU/ml, pool size must be reduced to ≤ 20 to affect qualitative RAMP interpretation (pos/neg); pools containing ≥ 25 mosquitoes are qualitatively similar. Samples with titers > 4 log₁₀ PFU/ml were within the positive range (> 50 RAMP units) even though the RAMP scores were lower as pool size increased; all samples with titers < 3.6 log₁₀ PFU/ml were negative (< 50 RAMP units).

Fig. 3.

RAMP results from 2 repeats (pool sizes containing 1, 5, 25, 50 and 1, 3, 5, 10, 15, 20, 25 and 50 individuals) of the processing variables evaluation of samples homogenized in RAMP buffer by Vortex (1 min) or Mixer Mill (4 min at 25 cycles/sec). Each West Nile virus (WNV) sample titer is graphed on its own panel. The manufacturer’s recommended RAMP score of ≥ 30 RAMP units as the positive cut-off for mosquito pools is represented by the dashed line; our proposed cut-off of ≥ 50 RAMP units is represented by the solid line. Samples that were vortexed are represented by black diamonds (◆); samples processed by Mixer Mill are shown as open diamonds (◇).

Results of the linear model including mosquito pool size and estimated titer indicated statistically significant interaction effects among these predictors and grinding method (RLRT p < 0.001), complicating interpretation. We describe the comparisons relevant to the utility of the RAMP assay.

While mean RAMP score did significantly depend on the grinding method’s interaction with mosquito pool size and estimated titer, this dependence was not qualitatively important when evaluating RAMP assay sensitivity (i.e., the probability that a sample of a particular mosquito pool size and estimated titer will produce ≥ 50 RAMP units). We therefore used a model excluding grinding method but retaining the mosquito pool size and estimated titer main effects and interaction for estimation of RAMP assay sensitivity; this in effect estimates the average sensitivity over the grinding method effect at each combination of mosquito pool size and estimated titer. As inspection of the RAMP scores in Fig. 3 suggests, reactivity of the RAMP assay was dynamic over the range of estimated titers used, with the estimated titer 3.8 log₁₀ PFU/ml yielding the range of qualitative differences in sensitivity of interest when using a positivity cutoff of 50 for the RAMP score. Indeed, sensitivities using this cutoff for estimated titers below 2.8 log₁₀ PFU/ml were nearly 0, for those at 3.6 log₁₀ PFU/ml were all well below 0.5, and for those at least 4.5 log₁₀ PFU/ml were nearly 1. In our evaluation, the sensitivities and 95% CIs for estimated titer 3.8 log₁₀ PFU/ml for the different pool sizes evaluated are given in Table 1. For estimated titer 3.8 log₁₀ PFU/ml and all mosquito pool sizes tested, sensitivities are at least 50%; while for samples with 15 or fewer individuals in the pools the sensitivities are generally above 95%.

Table 1.

Sensitivities (i.e., the probability that a positive sample will produce a positive result in the RAMP or real-time reverse-transcriptase polymerase chain reaction (RT-PCR) assays) and 95% confidence intervals (CI) of samples containing 3.8 log₁₀ PFU/ml West Nile Virus (WNV) using a positivity cutoff of ≥ 50 for RAMP units and ≤ 37 for real-time RT-PCR cycle threshold (CT) values.

		RAMP Scores ≥ 50 RAMP units		Real-time RT-PCR CT Values ≤ 37
Pool Size	Number of Pools	Sensitivity (%)	95% CI (%)	Sensitivity (%)	95% CI (%)
50	10	56.5	51.3, 61.8	55.3	45.6, 65.1
25	10	67.8	62.7, 72.9	79.2	71.0, 87.0
20	10	87.6	83.9, 90.9	NT	NT
15	10	99.9	99.9, 1.00	NT	NT
10	9	88.4	84.9, 91.7	NT	NT
5	9	99.8	99.6, 99.9	59.8	50.2, 69.4
3	10	96.3	94.4, 97.8	NT	NT
1	10	97.6	96.2, 98.7	NT	NT

NT = not tested

Processing Variables Experiment Results – Real-time RT-PCR assay

All pools processed in RAMP buffer produced significantly higher real-time RT-PCR CT values when compared to pools of the same titer, size, and grinding protocols that were processed in BA-1 (Fig 4). Linear models fit to real-time RT-PCR CT showed statistically significant differences in average CT between the two buffers, the magnitude of which depended on the numbers of mosquitoes (p < 0.01), while adjusting for estimated titer. Given that the RAMP assay and subsequent real-time RT-PCR confirmation are performed on samples processed in RAMP buffer and not BA-1, we consider only the RAMP buffer variable as we further analyze CT.

Fig. 4.

Real-time reverse transcriptase polymerase chain reaction (RT-PCR) results from the processing variables evaluation of mosquito pools containing 1, 5, 25, and 50 individuals homogenized in BA-1 cell culture medium by Vortex (1 min) or Mixer Mill (4 min at 25 cycles/sec). CT values ≤ 37 are considered positive (represented by the dashed line in each panel). Samples that were processed in BA-1 are represented by black diamonds (◆); samples processed in RAMP buffer are shown as gray diamonds (◆).

Regression of CT on mosquito pool size, grinding method, and estimated titer resulted in the best-fitting model retaining number of mosquitoes and estimated titer (with both linear and squared terms) in the model (p < 0.05 for each); differences in average CT did not differ by grinding method when accounting for these variables (p = 0.11). Dependence of CT on pool size and estimated titer resulted in different sensitivity estimates (and 95% CIs) by these values, and we report in Table 1 the estimated sensitivities for the observed mosquito pool sizes 5, 25, and 50, at estimated titer 3.8 log₁₀ PFU/ml, values for which were estimated using the model. As expected, sensitivity increases with increasing estimated titer, and sensitivities were estimated at essentially 100% for estimated titers at least 4.5 log₁₀ PFU/ml.

Discussion

The development of commercial WNV detection assays has allowed MADs across the US to expand WNV surveillance in mosquitoes without the need for developing a costly virus isolation or PCR laboratory, enabling MADs to use surveillance data to guide abatement decisions and determine the effectiveness of mosquito control strategies. However, conflicting results from RAMP and real-time RT-PCR assays have been frustrating for MADs who have encountered them. One study that systematically examined RAMP and real-time RT-PCR data generated from field-collected mosquito pools described incongruent results between the two assays, the most troubling being cases where samples were RAMP-positive but real-time RT-PCR negative (Kesavaraju et al. 2012). These samples were described as “false-positive,” since detection of WNV by real-time RT-PCR is considered more sensitive than antigen detection (Kesavaraju et al. 2012). We have shown in this evaluation that real-time RT-PCR performed on samples processed in RAMP buffer is inhibited by the buffer and does not detect RNA in lower-titered pools. Our experiments used laboratory-generated WNV positive and negative mosquito pools to serve as controls so we could accurately gauge the performance of the RAMP and real-time RT-PCR assays together as part of a WNV detection system utilized by many states, and to subject RAMP and real-time RT-PCR to varying conditions to determine their effects on each assay.

Almost all of the RAMP results generated from the proficiency panels fell within an expected range of RAMP units for each titer (Fig. 1). Using the manufacturer’s suggested RAMP positive cutoff of ≥ 30, the RAMP assay produced false negatives for all but 4 samples with titers of ≤3 log₁₀ PFU/ml; however this was expected based on previous sensitivity evaluations (Burkhalter et al. 2006, Sutherland and Nasci 2007). One MAD’s sample containing 2 log₁₀ PFU/ml WNV produced an unexpected RAMP result of 640 while all other samples with that titer produced negative RAMP results. The reason for this discrepancy is unknown. All but one negative control produced a result < 30 RAMP units. Changing the RAMP positive cutoff from 30 to 50 RAMP units would not significantly impact the sensitivity limits of the RAMP assay as determined previously (Burkhalter et al. 2006) since most of the pools in the panel that scored less than 50 also scored less than 30 and thus would be interpreted as negative using either cut-off. However raising the cut-off to 50 eliminates the chance that false positives will be produced by negative pools. No negative control pools in this evaluation or any we have conducted thus far have produced RAMP results ≥ 50.

Factors that may account for the inability to obtain RT-PCR confirmation of RAMP positive samples in field-collected mosquito pools may be the improper utilization of the RAMP, mishandling of samples before or after RAMP testing and during transit to confirmation labs, or the effect of the RAMP buffer on the RT-PCR reaction. In our assessment, it appears unlikely that the RAMP assay was improperly conducted as we were able to ascertain from protocol data reported as part of the proficiency panel evaluation by the MADs that the RAMP assay protocol was followed satisfactorily. To promote proper use of the RAMP kit, Response Biomedical (2010) has published technical documents providing guidelines for handling, processing, and testing samples beyond what is described in the RAMP assay kit insert. It is possible that sample degradation due to high temperatures or repeated freeze-thaws in shipping and handling may render WNV RNA in the samples undetectable in real-time RT-PCR. In our proficiency panel evaluation frozen panels were sent to MADs on dry ice and samples were kept cold before being ground and tested with the RAMP assay. In the processing variables evaluation mosquito pool samples were kept frozen at −70 °C until ground and tested with the RAMP and real-time RT-PCR assays.

A more likely cause for incongruent RAMP and real-time RT-PCR results appears to be the effect of the RAMP buffer on real-time RT-PCR’s ability to detect WNV RNA. In both the proficiency panel and processing variables comparison evaluations, RAMP samples containing ≤ 3 log₁₀ PFU/ml were undetectable by real-time RT-PCR. By comparing real-time RT-PCR results of WNV positive samples ground in RAMP buffer and BA-1, we were able to show a significant increase in real-time RT-PCR CT values of the samples ground in RAMP buffer. Beyond general reduction in sensitivity due to inhibition by RAMP buffer, however, we found that real-time RT-PCR results and sensitivity limits varied among labs that participated in the proficiency panel evaluation; in some cases even samples with WNV titers of ≥ 4 log₁₀ PFU/ml were not confirmed. Determining the causes of the lower sensitivity in these labs was beyond the scope of this evaluation and not investigated further. Response Biomedical (2008) recommends freezing an aliquot of supernatant from samples to be tested with real-time RT-PCR at −70 °C as soon as possible to reduce RNA degradation. However, the best results from real-time RT-PCR testing of pools spiked with WNV were produced when the supernatant was extracted for RNA before the sample was frozen and little time had passed between processing and RNA extraction (Response Biomedical 2008).

We also tested pools of various sizes ground in RAMP buffer using 2 different grinding methods to determine the effects of these variables on RAMP and real-time RT-PCR results. Even though RAMP scores decreased as pool size increased the results remained in the positive RAMP unit range for pools containing ≥ 4 log₁₀ PFU/ml. Decreasing the pool size did not affect the qualitative negative results of samples containing ≤ 3 log₁₀ PFU/ml when using a RAMP cut-off of ≤ 50. Operationally, testing the maximum pool size of 50 individuals does not diminish the RAMP’s ability to detect WNV in samples containing ≥ 4 log₁₀ PFU/ml, nor does decreasing the pool size to < 50 enhance RAMP’s ability to detect WNV in samples containing ≤ 3 log₁₀ PFU/ml, regardless of grinding method used. For pools containing 3.8 log₁₀ PFU/ml, reducing pool size to ≤ 20 individuals increases the number of pools that would be expected to produce ≥ 50 RAMP units, and vortexed pools containing ≤ 20 mosquitoes produced higher RAMP results than pools of the same size ground by Mixer Mill (Table 1). However, we suspect that the few numbers of previously undetected samples in this narrow titer range that would be detected by reducing pool sizes does not warrant incurring the additional cost of testing more pools.

Based on the data generated by our evaluations, we offer two recommendations for interpreting RAMP assay data. To maximize the speed, efficiency, and economy of testing mosquito pools with the RAMP assay, the RAMP positive threshold could be set at 50 RAMP units, with no real-time RT-PCR confirmation testing required. In this evaluation samples producing RAMP scores ≥ 100 contained approximately ≥ 4 log₁₀ PFU/ml and were within the sensitivity limits of real-time RT-PCR confirmation of samples processed in RAMP buffer. A more conservative approach would be to set the RAMP positive threshold at 100 above which samples do not require real-time RT-PCR to be considered positive and to implement a “gray zone” of 50-99 RAMP units. Samples that produce scores within the established “gray zone” would be confirmed by real-time RT-PCR to be considered positive. In the CDC processing variables evaluation, pools of 25 and 50 individuals containing between 3.6 – 4 log₁₀ PFU/ml produced RAMP results between 50 and 100 and most were confirmed by real-time RT-PCR. In the proficiency panel evaluation, a few pools of 50 mosquitoes containing approximately 4 log₁₀ PFU/ml produced RAMP scores between 80 and 100 and all were confirmed by real-time RT-PCR. These data support the cautionary warning that mosquito pools processed in RAMP buffer producing scores within the range of 50-100 RAMP units are at the limit of real-time RT-PCR detection and may or may not confirm even if positive. Implementing either recommendation would reduce the number of RAMP positive samples needing to be confirmed using real-time RT-PCR.

In the proficiency panel evaluation we were limited to evaluating pools of one mosquito species and pool size, and all mosquitoes were deplete (not blood fed). These constants, as well as the standard processing procedures and assay protocols used in this study do not account for the wide variability in sample collecting, handling, processing and testing that field-collected mosquitoes may be subjected to across the country by MADs and confirmation laboratories. We recommend that samples are handled carefully and maintained in a cold chain, and that the guidelines from the manufacturer are heeded to ensure the best results from the RAMP assay and greatest likelihood of correlating real-time RT-PCR results. When the RAMP assay is being performed correctly by the MAD, careful considerations should be made to determine whether sending pools for additional confirmation by real-time RT-PCR is necessary when pools score above a RAMP cutoff of 50 or 100. Results from this evaluation suggest that from an operational standpoint, the RAMP assay is a moderately sensitive and accurate tool for MADs to incorporate in their surveillance programs in lieu of molecular detection capabilities.

Appendix: Computation of sensitivity estimates and confidence intervals

For a random sample X₁, X₂, … , X_n from a normal distribution with mean μ and variance σ² , let $\widehat{\mu }$ and ${\widehat{\sigma }}^2$ be the usual unbiased estimators of the mean and variance, respectively, and note that $\widehat{\mu }$ is normally distributed with mean μ and variance σ²/n. Let θ be a fixed value that we use to represent a cutoff value for observations from the common distribution of the {X_i}, such that values for which X_i > θ are considered “positive,” and otherwise “negative.” We are interested in computing the probability that future values of X from this distribution are greater than θ and we call this probability the sensitivity. Let $W=(\theta -\widehat{\mu })∕\widehat{\sigma }$ and P = 1 − Φ(W). The value P is therefore the sensitivity, and a point estimate for P is obtained from the observed value for W via the specification for θ and the observed values for $\widehat{\mu }$ and ${\widehat{\sigma }}^2$. Confidence intervals for the sensitivity P can be obtained from its sampling distribution, the CDF for which is given by

$$\begin{gathered} P[P\le p]=P[1-\Phi (W)\le p]=P[\widehat{\mu }\le \theta -\widehat{\sigma }{\Phi }^{-1}(1-p)] \\ ={\int }_0^{\infty }P\left[\widehat{\mu }\le \theta -\sqrt{s^2}{\Phi }^{-1}(1-p)\right]f(s^2;{\sigma }^2)d{s}^2 \\ ={\int }_0^{\infty }\Phi \left(\theta -\sqrt{s^2}{\Phi }^{-1}(1-p);\mu ,\frac{{\sigma }^2}{n}\right)f(s^2;{\sigma }^2)d{s}^2 \end{gathered}$$

where f(s²; σ²) is the density of ${\widehat{\sigma }}^2\sim \frac{{\sigma }^2}{n}{\chi }^2$, a scaled chi-squared distribution with n – 1 degrees of freedom. From this CDF, a 100(1 – α)% CI (p_L, p_U) may be found by solving $P[P\le p_L]=\frac{\alpha }{2}$ and $P[P\le p_U]=1-\frac{\alpha }{2}$ for p_L and p_U,, which must be accomplished numerically. Computation of these expressions requires estimates for parameter values from a particular application.

In the regression modeling here, we assumed a normal regression model for the responses (log RAMP scores or real-time RT-PCR CT as a function of covariates estimated titer, mosquito pool size, and grinding method. We also modeled the variance as a function of the estimated titer. Estimates of the regression parameters from these fits provide estimates for the mean, at specified values of the covariates, and variance function and degrees of freedom, at specific values of estimated titer, for use in the above formula for P[P ≤ p].