Optimization of the Cutoff Value for a Commercial Anti-Dengue Virus IgG Immunoassay

Abstract

A commercial anti-dengue virus (anti-DENV) indirect IgG enzyme-linked immunosorbent assay (ELISA) for serological diagnosis was evaluated for its utility in determining previous DENV exposure in U.S. travelers. The Boston Area Travel Medicine Network clinics used Focus Diagnostics anti-DENV IgG ELISA to measure anti-DENV IgG antibodies in 591 pretravel specimens from U.S. residents who had traveled to countries where dengue is endemic. When using the manufacturer's index cutoff value for this ELISA, false-positive results were observed that overestimated the perceived past DENV exposure in U.S. travelers. Validation of 121 of these anti-DENV IgG results by plaque reduction neutralization test (PRNT) was used for receiver operating characteristic (ROC) curve optimization of the index cutoff value from 1 to 3.0, improving the specificity of the anti-DENV IgG ELISA from 24% to 95.7%. Additionally, previous vaccination with yellow fever virus contributed to 52.8% of the false-positive rate in the anti-DENV IgG ELISA results. Optimization of the cutoff value of the anti-DENV IgG ELISA provided better interpretation and confidence in the results and eliminated the need for confirmation by PRNT. The travel history of U.S. travelers was also useful for categorizing these travelers into groups for analysis of previous DENV exposure.

INTRODUCTION

Dengue is a major, global health problem, with an estimated 2.5 billion people at risk of contracting the disease and over 50 million infections annually (1). Dengue virus (family Flaviviridae, genus Flavivirus) is mosquito-borne with four antigenically distinct serotypes (DENV-1 to -4). There is sufficient antigenic homology among the four DENV serotypes and within the genus Flavivirus to create cross-reactivity in many immunoassays. This cross-reactivity is a diagnostic challenge when testing samples in regions where multiple DENV serotypes circulate and other flaviviruses cocirculate. Hence, immunoassays, such as the IgM enzyme-linked immunosorbent assay (ELISA) and the IgG ELISA, often require testing using a second method, such as a plaque reduction neutralization test (PRNT), to confirm the specificity of the antibody response (2).

Anti-DENV IgG ELISAs are used to measure prevalence of previous infection by DENV and for conducting vaccine studies. The Boston Area Travel Medicine Network (BATMN) clinics used commercial anti-DENV IgG ELISAs to determine exposure to DENV in U.S. travelers; dengue is the leading cause of acute febrile illness in U.S. travelers returning from countries where DENV is endemic, as observed by GeoSentinal clinics (3). Among these patients, the Focus Diagnostics anti-DENV IgG ELISA had low specificity (24%) and high sensitivity (100%), as determined by the PRNT confirmatory test. Further evaluation of the cutoff value of the anti-DENV IgG ELISA was conducted in an attempt to increase the specificity of the test.

MATERIALS AND METHODS

Study.

Travelers were enrolled from five travel clinics from the BATMN group in Boston, MA, from August 2008 through June 2009. Approximately 7,500 travelers per year visit these five clinics. Travel history questionnaires were collected from each individual (4). Single blood samples and travel histories were obtained from 591 pretravelers, ranging in age from 11 to 86 years old, who planned to visit countries where dengue is endemic for at least 2 weeks. Collected sera were stored at −70°C prior to anti-DENV IgG ELISA testing.

Study participants were also asked their place of birth and vaccination history, particularly vaccination against yellow fever (YF) and Japanese encephalitis (JE) viruses. Based on their responses, the participants were classified into three groups: (i) travelers born in countries where dengue is endemic, (ii) travelers born in countries where dengue is nonendemic but lived in countries where dengue is endemic for 1 year or more, and (iii) travelers born in countries where dengue is nonendemic but visited countries where dengue is endemic for at least 2 weeks but <1 year. The study was approved by the Institutional Review Boards of all participating institutions and of the Centers for Disease Control and Prevention.

Indirect IgG anti-DENV ELISA.

Focus Diagnostics anti-DENV IgG ELISA EL1500G (Focus Diagnostics, Cypress, CA,) was used to determine DENV exposure by qualitative testing of pretravel specimens. The Focus ELISA kit includes a cutoff calibrator and two controls. After subtracting the blank wells, the target optical density (OD) of the cutoff calibrator should be approximately 0.250 OD units with an acceptable range of 0.10 to 0.50 OD units according to the manufacturer. Results are reported as index values relative to the cutoff calibrator. To calculate the index values, the specimen's OD unit is divided by the mean of the cutoff calibrator OD unit providing the index value per specimen. The manufacturer's suggested index cutoff value of >1.00 was used to classify specimens as positive. The IgG testing was performed in duplicate, following the manufacturer's test procedure (5). A subset of 121 specimens (109 positive and 12 negative) was tested by anti-DENV IgG ELISA to represent all clinic sites, and a range of destinations were selected among the 591 samples collected by the 5 BATMN travel clinics for analysis and validation by PRNT.

PRNT.

The PRNT is used as the confirmatory test for anti-DENV IgG-positive samples to determine antibody and serotype specificity (6). The presence of neutralizing antibodies to DENV-1 to -4 in the 121 pretravel specimens, tested by anti-DENV IgG ELISA, was determined by PRNT. Briefly, serotype-specific DENV and patient serum were mixed in vitro and applied to a confluent monolayer of susceptible Vero cells (African green monkey kidney cells; ATCC, Rockville, MD). Laboratory viral strains used for the assay, including DENV-1 (Hawaii), DENV-2 (New Guinea-C), DENV-3 (H87), and DENV-4 (H241), were obtained from the Centers for Disease Control, Dengue Branch, virus reference collection (San Juan, Puerto Rico) (7). Serum samples were filtered through a 0.2-μm filter and heat inactivated at 56°C for 30 min, and serial 2-fold dilutions were made in phosphate-buffered saline (PBS) containing 30% fetal bovine serum (FBS) starting at a dilution of 1:20 and ending at a dilution of 1:640. Diluted patient serum (100 μl) was mixed with an equal volume of standard diluent containing 200 PFU of viral seed and incubated for 2 h at room temperature (RT). After incubation, 100 μl of the serum-virus mixture was inoculated, in duplicate, onto confluent monolayers of Vero cells in Costar 12-well plates at RT for 60 min, and the plate was rocked every 15 min. A back-titration of virus without serum was included to determine the 90% reduction in plaques. Following incubation, 3 ml of Ye-Lah overlay medium (Earle's balanced salt solution, containing 20 g/liter yeast extract, 100 g/liter lactalbumin hydrolysate, 25% FBS, 1% Fungizone-gentamicin, and 0.5% agarose) was added to each well and placed in an incubator at 37°C, with 10% CO₂, for 5 days. Plates were stained by adding 1 ml/well of 3.2% neutral red (Sigma, St. Louis, MO), in PBS, on day 5 postinfection and incubated overnight at 37°C in 10% CO₂. Plaques were counted on the following day to obtain neutralizing antibody titers.

Serotype-specific, neutralizing anti-DENV will neutralize the virus and prevent cell infection. Nonneutralized DENV infects the Vero cells forming plaques in the cell monolayer (8). The PRNT endpoint titer was reported as the reciprocal of the titer in which there is a reduction of plaques by 90% compared to the virus control for each sample (8).

A positive PRNT result was defined as a neutralizing titer of ≥1:20 to any DENV serotype. A 4-fold or higher increase in PRNT titer for one DENV serotype than in those of the other serotypes was considered to be the predominant infecting DENV serotype. Samples with reactivity to only one serotype were defined as primary DENV infections. Samples with neutralizing antibodies to at least 2 or more DENV serotypes were defined as secondary DENV infections.

Statistical methods.

Simple associations between ELISA and PRNT were analyzed. ELISA screening capabilities were evaluated using PRNT as the confirmatory test, and the ELISA results were evaluated using a chi-square test, Fisher's exact test (when applicable), logistic regression analysis, and a receiver operating characteristic (ROC) curve to determine the best optical density cutoff value for the anti-DENV IgG ELISA. Furthermore, the effect of each patient's vaccination history was also evaluated, using logistic regression analysis, with positive responses to both previously mentioned tests to indicate the interactions between vaccination and lifetime exposure to DENV. Statistical analyses were performed using SAS software version 9.3 (SAS, Inc., Cary, NC), and significance was achieved when P was <0.05.

RESULTS

Indirect IgG ELISA results confirmed by PRNT.

Of the 121 specimens tested by both anti-DENV IgG ELISA and PRNT, 109 were ELISA positive and 12 were negative. From the 109 anti-DENV IgG-positive specimens, 71 were positive and 38 were negative by PRNT. All of the 12 specimens negative for anti-DENV IgG by ELISA were negative by PRNT (Table 1). The comparison of the 121 traveler specimens tested by both anti-DENV IgG ELISA and PRNT showed that the anti-DENV IgG ELISA had a sensitivity of 100% and a specificity of 24%, which differed from the manufacturer's technical summary report of 95% sensitivity and 99% specificity (for regions of nonendemicity). However, the Focus performance characteristics data sheet reported a sensitivity of 39% for past dengue infections compared to the positive results of the hemagglutination inhibition (HI) assay and a specificity of 99% for asymptomatic individuals from areas of nonendemicity and 86% for asymptomatic individuals from areas of endemicity (9).

Table 1.

Pretravel specimens analyzed using the Focus anti-DENV IgG ELISA^a

Virus	PRNT specimens				Indirect ELISA results
	Primary	Secondary	Total	% of total	Positive IgG index > 1	Positive IgG index ≥ 3.0	Negative IgG index < 1
DENV-1	7	3	10	8	10	7	0
DENV-2	8	3	11	9	11	7	0
DENV-3	7	0	7	6	7	4	0
DENV-4	1	0	1	<1	1	1	0
≥2 serotypes	NA	42	42	35	42	40	0
Negative	NA	NA	50	41	38	3	12
Total	23	48	121	100	109	62	12

^aFocus anti-DENV IgG ELISA was performed with the manufacturer's cutoff index value and the adjusted cutoff index value confirmed by PRNT. NA, not applicable.

The results from this study were analyzed using a ROC curve to establish an index cutoff value for the anti-DENV IgG ELISA that would yield a positive PRNT result. The ROC curve determined that the optimal index cutoff value for the anti-DENV IgG ELISA was 3.0, with a corresponding specificity of 95.7% and a sensitivity of 85.3% (Fig. 1 and Table 1). The area under the ROC curve (AUC) value was 0.95.

Fig 1.

Receiver operating characteristic (ROC) curve indicating the optimal cutoff value of 3.0, yielding a specificity of 95.7% and corresponding sensitivity of 85.3% for the Focus anti-DENV IgG ELISA compared against the PRNT as the gold standard confirmatory test.

Cross-reactivity in the anti-DENV IgG ELISA due to previous vaccination with YF and JE.

The YF and JE vaccination histories of the subjects were used to determine the proportion of false-positive results obtained from the anti-DENV IgG ELISAs due to preexisting flavivirus antibodies. Of the 121 individuals, 44 had previously been vaccinated against YF only, 3 were travelers previously vaccinated against JE only, and 4 were individuals previously vaccinated against both YF and JE. Previous vaccination against YF only yielded a false-positive rate of 52.8% in the anti-DENV IgG ELISA using the original cutoff value of the assay (>1) and was reduced to 6.7% when the cutoff value was adjusted (≥3.0) (Table 2). Although the result for previous JE vaccination was statistically significant, the study was underpowered to address the outcome of the ELISA, based on JE vaccination, due to an insufficient sample size (Table 2). Most of the false positives for both cutoff values (>1 and ≥3) that were observed in the ELISAs were found in serum samples from travelers who visited countries where DENV is endemic for at least 2 weeks but for no more than a year (75% and 14.3%, respectively).

Table 2.

Comparison of positivity rates between ELISA and PRNT among travelers with previous flavivirus vaccinations

Travelers	ELISA IgG positivity rate (%)		False-positive rate (%)a
	Cutoff >1	Cutoff ≥ 3.0	Cutoff > 1	Cutoff ≥ 3.0
Flavivirus vaccines
None (>n = 70)	66 (94.3)	44 (62.9)	15/66 (22.7)	1/44 (2.3)
YF only (n = 44)	36 (81.8)	15 (34.1)	19/36 (52.8)	1/15 (6.7)
JE only (n = 3)	3 (100.0)	1 (33.3)	2/3 (66.67)b	0
Both (n = 4)	4 (100)	2 (50.0)	2/4 (50.0)	1/2 (50.0)
By group
Group 1 (n = 72)c	69 (95.8)	50 (69.4)	12/69 (17.4)	2/50 (4.0)
Group 2 (n = 18)d	16 (88.9)	5 (27.8)	8/16 (50.0)	0
Group 3 (n = 31)e	24 (77.4)	7 (22.6)	18/24 (75.0)	1/7 (14.3)

^aFalse positive defined using PRNT as the true positive.

^bInadequate sample size for analysis.

^cU.S. travelers born in countries where dengue is endemic.

^dU.S. travelers born in countries where dengue is nonendemic but lived in countries where dengue is endemic for ≥1 year.

^eU.S. travelers born in countries where dengue is nonendemic but visited countries where dengue is endemic for at least 2 weeks but <1 year.

DISCUSSION

The presence of anti-DENV IgG antibodies is an indication of long-term, acquired immunity from a past DENV infection. Moreover, IgG antibody titer differences can identify an acute dengue infection when paired serum samples, consisting of acute (0 to 5 days after onset of illness) and convalescent (>5 days after onset of illness) samples, are available. However, an anti-DENV IgG result in a single sample cannot clinically diagnose an acute infection. The Focus anti-DENV IgG ELISA (catalog no. EL1500G) used in this study is an indirect ELISA, which immobilizes DENV antigens on a solid surface. The DENV antigen is then recognized by anti-DENV IgG antibodies in the tested sample. A detector antibody that is conjugated to horseradish peroxidase binds nonspecifically to human IgG and amplifies the signal for detection. The indirect anti-DENV IgG ELISA is often used for epidemiologic studies to determine past DENV exposure because of its improved sensitivity and specificity compared to the direct ELISA method. The direct IgG ELISA requires purified antigen, and this reagent is not always available.

The anti-DENV IgG antibodies from infected patients are primarily targeted against the viral structural genes, particularly envelope protein E. Because of the antigenic similarities of the E proteins within the flavivirus family, anti-DENV IgG antibodies may have cross-reactive epitopes, requiring the PRNT confirmatory diagnostic test to assess antibody specificity (10). The ELISA is usually developed to detect all DENV serotypes equally, but there may be differences in sensitivities by serotype that could not be measured in this study. The PRNT is the gold standard confirmatory test for serologically positive DENV specimens (11). Unfortunately, the PRNT is a costly, time-consuming, and technically complicated test to perform and few laboratories have the capacity and technical expertise to properly perform this test. Hence, many DENV studies rely solely on the anti-DENV IgG ELISA results to measure prevalence of DENV infection in a population. One limitation of the PRNT is that it measures the neutralization capacity of the antibody response, whereas the IgG ELISA measures total anti-DENV antibodies; hence, the IgG ELISA will also detect the presence of nonneutralizing antibodies.

The Focus anti-DENV IgG ELISA used in this study has a reported sensitivity of 95% and a specificity of 99% (for regions of nonendemicity), based on the technical summary sheet from previous studies performed by the manufacturer (9). Further characterization of the assay was summarized in the Focus performance characteristics data sheet in which the sensitivity of past dengue infections compared to HI was 39% and specificity of the test varied dependent on whether the individual was from a region of nonendemicity (99%) or a region of endemicity (86%). In the present study, using specimens from U.S. travelers visiting the BATMN clinics, the sensitivity and specificity of this ELISA were 100% and 24%, respectively, compared to the PRNT results. The results of this study indicated that the manufacturer's suggested index cutoff value for the ELISA caused substantial misclassification that increased the prevalence of prior DENV infection among U.S. travelers compared to a more specific test, namely, the PRNT. This misclassification rate in regions where dengue is nonendemic may be due to the positive and negative predictive values (PPV and NPV) of the test, based on the prevalence of the infection in the population. For example, in Key West, Florida, a seroprevalence study conducted following an outbreak of dengue due to DENV-1 in 2009 indicated that the anti-DENV IgG ELISA, without PRNT confirmation, falsely elevated the prevalence of prior dengue (12). This may be explained by the previous exposure of this population to other known circulating flaviviruses (St. Louis encephalitis virus and/or West Nile virus [WNV]) or previous YF and/or JE vaccinations. Conversely, in an area where dengue is endemic, such as Puerto Rico, the prevalence of DENV infection is 95%; hence, most of the anti-DENV IgG ELISA-positive results would also be PRNT positive since DENV is the predominant circulating flavivirus in Puerto Rico and WNV transmission is only sporadically introduced and not maintained (13, 14).

This study tested samples obtained from U.S. travelers with unknown exposure to multiple flaviviruses. Characterizing the samples by group allowed for further analyses based on potential exposure to DENV. Additionally, since YF virus and JE virus are flaviviruses, there exists a possibility for cross-reactivity due to prior vaccination (15). The present results indicated that 52.8% of the false positives were due to YF vaccine, which was similar to the Focus performance characteristics data sheet, indicating a specificity of 50% for YF vaccination in 3 out of the 6 total patients studied. Similar cross-reactivity were observed in a previous study where the YF vaccination group was tested with Panbio indirect anti-DENV IgG ELISA. Schwartz et al., identified high rates of cross-reactivity in the anti-DENV Panbio IgG ELISA when the patients were previously vaccinated against YF and JE, overestimating past DENV exposure when using this assay (16). Antibody cross-reactivity may depend on the time frame of vaccination in the context of testing for anti-DENV IgG (17).

In order to improve performance of the anti-DENV IgG ELISA for the detection of past DENV infection dengue exposure in U.S. travelers, the current investigation reevaluated the ELISA, according to groups, and established a new overall index cutoff value. The optimal index cutoff value, determined by ROC analysis (3.0), improves the specificity of the assay to 95.7%, providing a more accurate assessment of past DENV exposure. Additionally, the new cutoff value decreases the false-positive rates in YF vaccination individuals from 52.8% to 6.7%. Using the optimal index cutoff value eliminates the need for PRNT confirmatory tests and increases overall confidence in the results of this anti-DENV IgG ELISA in the context of this population. Previous studies have also indicated that an improvement in the performance of commercial ELISAs may be achieved after adjusting the cutoff value, under appropriate validation, in the context of the study and sample types that fit the model (18).

The limitations of the new index cutoff value for the Focus anti-DENV IgG ELISA include a decrease in sensitivity from 100% to 85.3%. Additionally, the PPV may vary depending on whether the samples are from a region where DENV is endemic or nonendemic. By adjusting the index cutoff value, the Focus anti-DENV IgG ELISA provides a useful tool for epidemiological studies to evaluate previous DENV exposure in U.S. travelers. This study lacked sufficient travelers' samples vaccinated against JE to assess the effects of this vaccination on the ELISA. In addition, this study lacked sufficient samples in target group 3, U.S. travelers not born in countries where dengue is endemic. Finally, all positive samples and only a small subset of negative samples were chosen for the PRNT, thus leading to potential bias toward low specificity.

In conclusion, adjusting the cutoff value for a commercial assay requires that the study be properly designed but may not always be appropriate—depending on the population in which the assay is intended to be used—and cannot be used across all populations. Further considerations in the design of a study to adjust cutoff values include geographical location of the study, age of the study participants, previous exposure of the participants to other flaviviruses, subject vaccination and travel histories, and whether or not the participant lived for >2 years in a country where DENV is endemic. The decision to adjust a cutoff value should be based on an analysis of sensitivities and specificities using a ROC curve, comparing the test with a confirmatory test using an appropriate sample size.