Performance of the Focus HerpeSelect-2 EIA for the detection of herpes simplex virus type 2 antibodies in seven African countries

Abstract

Objective

To compare the performance of the Focus HerpeSelect-2 enzyme immunoassay (EIA) to the gold standard HSV-2 Western blot, among HIV-1 uninfected men and women in East and Southern Africa.

Methods

3399 HIV-1 uninfected women and men from 7 countries in East and Southern Africa were tested for HSV-2 antibody using Focus HerpeSelect-2 EIA. The performance of the HerpesSelect-2 EIA was compared with the gold standard HSV-2 specific Western blot.

Results

Two-thirds (2294/3399) of participants were male and two-thirds (2242/3399) were from East Africa. By Western blot testing, HSV-2 prevalence was 68%, 59% in men and 85% in women. At the manufacturer’s recommended cut-off value of greater than 1.1, the HerpeSelect-2 EIA had a sensitivity of 98.3% and specificity 80.3%. Receiver operating characteristic (ROC) plot analysis indicated that the optimum cut-off was 2.1 or greater with sensitivity 93.9% and specificity 90.5%. Diagnostic accuracy was modestly higher for Southern Africa (AUC=0.979, 95% CI: 0.970-0.988) compared with East Africa (AUC=0.954, 95% CI: 0.942-0.965; p<0.001 for Southern vs. East Africa).

Conclusions

The Focus HerpeSelect-2 EIA has acceptable diagnostic accuracy for determination of HSV-2 serostatus in African HIV-1 uninfected adults. An assay cut-off value of 2.1 or greater results in approximately 90% sensitivity and specificity, against a gold standard HSV-2 Western blot. Diagnostic accuracy differed slightly by geographical region.

Introduction

Herpes simplex virus type 2 (HSV-2) is the most frequent cause of genital ulcer disease worldwide and is an important risk factor for HIV-1 acquisition (1). The HerpeSelect-2 enzyme immunoassay (EIA) (Focus Technologies, Cypress, California, USA) is a commercially available, type-specific serologic test for the detection of antibodies to HSV-2 glycoprotein G (gG) that is frequently used in epidemiologic research studies of HSV-2. However, the HerpeSelect-2 EIA has been reported to have poor specificity for serologic diagnosis of HSV-2 among some African populations, particularly for samples with index values (i.e., the ratio of the optical density of the sample to the optical density of a standard calibrator) in the low positive range (values between 1.1 and 3.4) (2-4). Studies comparing the HerpeSelect-2 EIA to gold standard assays, such as HSV-2 specific Western blot, have proposed various cut-offs (index values 3.1-3.5) to improve specificity (2-3, 5-6), but these were generally done with relatively small populations, only among men or women, or in single geographic areas. We compared the performance of the Focus HerpeSelect-2 EIA to the gold standard for HSV-2 serologic diagnosis, HSV-2 Western blot, among nearly 3400 HIV-1 uninfected men and women from 7 countries in East and Southern Africa.

Methods

Population and procedures

Between November 2004 and April 2007, 3408 HSV-2/HIV-1 co-infected women and men and their HIV-1 uninfected heterosexual partners were enrolled in the Partners in Prevention HSV/HIV Transmission Study, a randomized clinical trial of acyclovir HSV-2 suppressive therapy to reduce HIV-1 transmission (ClinicalTrials.gov number NCT00194519). All participants were ≥18 years of age, and HIV-1 uninfected partners could be either HSV-2 seropositive or seronegative. Couples were from 14 sites in 7 African countries in East (Kenya, Rwanda, Tanzania, and Uganda) and Southern Africa (Botswana, South Africa, and Zambia). As previously reported, HSV-2 suppression provided to the HIV-1 infected partners did not reduce HIV-1 transmission risk to their initially HIV-1 uninfected partners (7). For the present study, we assessed HSV-2 serologic status of the HIV-1 uninfected partners from a blood sample collected at the enrollment visit.

Institutional review boards at the University of Washington and at all collaborating site organizations approved study procedures. All participants provided written informed consent.

Laboratory methods

At study enrollment, HIV-1 uninfected partners provided a serum sample for HSV-2 serologic testing using the HerpeSelect-2 EIA; 12 laboratories performed the testing for the 14 study sites. The manufacturer’s instructions for this assay define a negative result as an index value less than 0.9, an indeterminate result as an index value between 0.9 and 1.1, and a positive result as an index value greater than 1.1. Archived serum aliquots from the enrollment visit were also batch tested at the end of the study at the University of Washington using an HSV type-specific Western blot (8). Western blot readers were blinded to the results of the HerpeSelect-2 EIA. All sites participated in an external quality assurance (EQA) program using a HSV-2 proficiency panel developed at the University of Washington (9).

Statistical analysis

Sensitivity and specificity of Focus HerpeSelect-2 EIA results, compared with Western blot, were calculated, and receiver operating characteristic (ROC) curves were constructed to describe test performance. Optimal EIA index result cutoffs were identified, aiming to achieve both test specificity and sensitivity of approximately 90%. Individuals with indeterminate Western blot results (i.e., HSV-2 gG band indistinct or not apparent after pre-absorption against HSV-1 antigens) (10) were excluded from sensitivity/specificity and ROC analyses. To assess the effect of between-laboratory variation on our estimates of test performance, we calculated adjusted ROC analyses that controlled for clustering by laboratory (11). Statistical calculations were performed using PASW Statistics 17.0 (IBM SPSS Inc, Chicago, IL) and Stata 10.1 (College Station, TX).

Results

Of 3408 HIV-1 uninfected participants enrolled in the study, HerpeSelect-2 EIA and Western blot results were available for 3399 (99.7%). Of these, 2294 (67%) were male and 2242 (66%) were from East Africa. The median age was 34 years (interquartile range [IQR] 28-41). By Western blot testing, HSV-2 seroprevalence was 68% (2304/3399), including 85% (944/1105) in women, 59% in men (1360/2294), 68% (1532/2242) in East Africa, and 67% (772/1157) in Southern Africa. One hundred nine participants (3%) had indeterminate (n=109) Western blot results.

There was generally strong agreement between HerpeSelect-2 EIA and Western blot results (Table 1). Using the manufacturer’s recommended cut-off (index value greater than 1.1) and compared with Western blot, the HerpeSelect-2 EIA had a sensitivity of 98.3% (2264/2304) and specificity of 80.3% (792/986) compared with Western blot for determination of HSV-2 infection (Table 2). At an index value cutoff of 3.5 or greater, sensitivity was 82.9% (1888/2304) and specificity improved to 95.1% (938/986). ROC plot analysis comparing a range of HerpeSelect-2 cut-offs versus gold standard Western blot indicated strong agreement between the two tests: [area under the curve (AUC) =0.962, 95% confidence interval (CI) 0.954-0.971]. An optimum cut-off of 2.1 or greater was identified, giving a sensitivity of 93.9% and a specificity of 90.5%.

Table 1.

Comparison of Focus HerpeSelect-2 EIA values and HSV-2 Western blot results

	Western blot result (row percentage)
EIA Index Value	Negative	Indeterminate	Positive	Total
<0.9	775 (93.5%)	23 (2.8%)	31 (3.7%)	829
0.9-1.1	33 (53.2%)	11 (17.7%)	18 (29.0%)	62
1.2-2.4	98 (32.2%)	38 (12.5%)	168 (55.3%)	304
2.5-3.4	32 (14.4%)	13 (5.9%)	177 (79.7%)	222
≥3.5	48 (2.4%)	24 (1.2%)	1910 (96.4%)	1982

Table 2.

Focus HerpeSelect-2 EIA performance compared to HSV-2 Western blot among HIV-1 uninfected participants from 7 countries in sub-Saharan Africa

	Overall (N=3290)		Females (N=1077)		Males (N=2213)		East Africa (N=2162)		Southern Africa (N=1128)
Index value	Sensitivity (%)	Specificity (%)	Sensitivity (%)	Specificity (%)	Sensitivity (%)	Specificity (%)	Sensitivity (%)	Specificity (%)	Sensitivity (%)	Specificity (%)
0.9	98.9	76.0	99.6	69.2	98.4	77.0	99.1	71.7	98.4	83.4
1.1	98.3	80.3	99.4	72.9	97.5	81.5	98.6	76.8	97.7	86.5
1.3	97.7	83.6	98.9	75.9	96.8	84.8	98.1	79.7	96.8	90.4
1.5	96.6	85.5	98.4	76.7	95.4	86.9	96.9	81.3	96.1	93.0
1.7	95.8	87.1	98.2	79.7	94.1	88.3	96.1	83.3	95.1	93.8
2.0	94.4	89.9	97.9	84.2	91.9	90.7	94.8	87.0	93.4	94.9
2.3	92.0	91.4	95.9	87.2	89.3	92.0	92.6	88.9	90.7	95.8
2.5	90.6	91.9	95.3	87.2	87.3	92.6	91.4	89.4	89.0	96.3
2.7	88.9	92.6	94.7	87.2	84.9	93.4	89.8	90.3	87.2	96.6
3.0	86.9	93.9	93.4	88.7	82.4	94.7	87.7	92.1	85.2	97.2
3.3	84.5	94.6	91.5	89.5	79.6	95.4	85.3	93.0	82.8	97.5
3.5	82.9	95.1	90.5	90.2	77.6	95.9	83.4	93.7	81.9	97.8

We assessed whether demographic factors resulted in differences in the diagnostic accuracy of the HerpeSelect-2 EIA. By ROC testing, overall test performance was modestly better in men (AUC=0.960, 95% CI: 0.951-0.969) than in women (AUC=0.944, 95% CI: 0.914-0.973) [Figure 1A], although this was not statistically significant (p=0.3). Diagnostic accuracy was modestly greater for samples from Southern Africa (AUC=0.979, 95% CI: 0.970-0.988) compared with East Africa (AUC=0.954, 95% CI: 0.942-0.965) [Figure 1B] and this difference was statistically significant (p<0.001). The optimal index value cutoff was 2.7 in East Africa (sensitivity 89.8%, specificity 90.3%) and 1.3 in Southern Africa (sensitivity 96.8%, specificity 90.4%). There were small differences in test performance between countries in the same geographical region (Table 3). Among samples from Uganda, specificity was 81% at an index value of 2.1 or greater, and the optimal index value was 3.4 (sensitivity 90.4%, specificity 90.1%). Test performance did not differ for those with or without a history of genital ulcer disease in the 3 months prior to enrollment (AUC=0.903, 95% CI: 0.732-1.000 versus AUC=0.954, 95% CI: 0.945-0.962) [p=0.50]. In addition, age had no effect on test performance: age 25 years or less (AUC=0.968, 95% CI: 0.951-0.985) versus age 25 years or greater (AUC=0.961, 95% CI: 0.951-0.970) [p=0.38].

Figure 1A.

Diagnostic accuracy of the Focus HerpeSelect-2 EIA by gender

Figure 1B.

Diagnostic Accuracy of the Focus HerpeSelect-2 EIA by geographical region.

Table 3.

Diagnostic accuracy of the Focus HerpeSelect-2 EIA, by country

Region	Country	HSV-2 prevalence	Diagnostic Accuracy (area under the curve, 95% confidence interval)
East Africa	Kenya (n=1375)	71%	0.949 (0.933-0.964)
	Rwanda (n=143)	83%	0.996 (0.989-1.000)
	Tanzania (n=210)	57%	0.981 (0.965-0.998)
	Uganda (n=434)	72%	0.943 (0.915-0.971)
Southern Africa	Botswana (n=322)	71%	0.986 (0.968-1.000)
	South Africa (n=496)	67%	0.987 (0.978-0.996)
	Zambia (n=310)	67%	0.957 (0.931-0.984)

We investigated whether between-laboratory variation accounted for differences in diagnostic accuracy. We recalculated the AUC, 95% confidence intervals, and p-values for the above comparisons with adjustment for clustering by lab, and obtained results that were essentially identical to the unadjusted analyses. In the adjusted models, there were no statistically significant differences in adjusted AUC estimates for comparisons of gender (p=0.43), GUD history (p=0.51), or age group (p=0.47). A statistically significant effect of region remained present in the analysis adjusting for site laboratory (adjusted AUC 0.978, 95% CI: 0.961-0.995 for Southern Africa versus 0.953, 95% CI: 0.939-0.966 for East Africa, p=0.02).

We considered the proportion of individuals with positive and indeterminate Western blot results at various low positive HerpeSelect-2 EIA values. For those with EIA results in the manufacturer’s indeterminate range (i.e., 0.9-1.1, n=62), 29% were positive and 18% indeterminate by Western blot. For those with EIA results between 1.2 and 2.4, 55% were positive and 13% indeterminate by Western blot. Finally, for those with EIA results between 2.5 and 3.4, 80% were positive and 6% indeterminate by Western blot. Overall, of the 109 persons with indeterminate Western blots, 75 (69%) had HerpeSelect-2 EIA values greater than 1.1 and 37 (34%) greater than 2.4 (Table 1). There were no significant differences in the frequency of indeterminate HSV Western blots by gender (p=0.12) or region (p=0.10).

Discussion

This multinational study of nearly 3400 HIV-1 uninfected adults represents the largest investigation of the performance of the Focus HerpeSelect-2 EIA to date in sub-Saharan Africa. In this population with high HSV-2 seroprevalence, a Focus HerpeSelect-2 EIA index value cut-off of 2.1 or greater resulted in approximately 90% sensitivity and specificity. There was some variation in optimal cut-off values by subgroups defined by gender and region. Of the approximately 15% of sera which had index values 1.2-3.4, two-thirds were confirmed as HSV-2 seropositive by Western blot.

Previous studies have demonstrated reduced specificity of the HerpeSelect-2 EIA at an index value cut-off greater than 1.1 for samples from Africa (4, 6), potentially due to cross-reactivity with HSV-1 (12) or HSV-2 sequence diversity (13). In our study population from East and Southern Africa, an index cut-off value 2.1 or greater improved test specificity to approximately 90%, similar to a meta-analysis that suggested index values between 2.2 and 3.5 increased specificity (14); in our population, a cut-off of 2.1 or greater also provided a sensitivity of approximately 94%. Studies from Rakai, Uganda have suggested higher index value cut-offs improve specificity, with an optimal trade-off between sensitivity and specificity identified at index values of 3.2-3.4 (3, 15). Our results for East Africa in general and Uganda in particular, are consistent with those findings. For samples from Southern Africa, we found an index cutoff value of 1.3 provided acceptable trade-off between sensitivity and specificity, lower than the optimal index value of 3.3 from one prior study from that region (5). Regional differences in test specificity could be a result of cross-reacting antibodies due to possible HSV-2 antigenic differences (3). Importantly, between-laboratory variations in test performance did not appear to account for differences in diagnostic accuracy. We also found gender differences in test performance, although this was not statistically significant; at each cut-off value, test sensitivity was better in women while specificity was higher in men. The potential for differential test performance by gender may merit further investigation.

Although improving test specificity is important in epidemiological studies and for clinical diagnosis, shipping sera to a reference laboratory for Western blot confirmatory testing is neither feasible nor cost-effective. Studies have suggested that sequential serologic testing with the Focus HerpeSelect-2 assay plus other commercially available type-specific assays, like Kalon™ HSV-2 gG2 ELISA (Surrey, UK) or Biokit HSV-2 rapid test (Lexington, MA) could be feasible alternatives to Western blot for confirmatory testing of sera that are initially positive on the HerpeSelect-2 EIA (16), although other studies have not demonstrated that serial algorithms improve diagnostic accuracy (6, 17). The approximately 90% sensitivity and specificity values we calculated with modified cut-offs may not be sufficiently precise for use of this assay for serologic diagnosis of HSV-2 in all clinical care settings.

In summary, the Focus HerpeSelect-2 EIA had an acceptable diagnostic accuracy in HIV-1 uninfected women and men from East and Southern Africa, compared to gold standard HSV-2 Western blot, particularly when using an index cut-off of 2.1 or greater.