Comparison of Focus HerpesSelect^® and Kalon^™ HSV-2 gG2 ELISA serological assays to detect herpes simplex virus type 2 (HSV-2) antibodies in a South African population

Abstract

Introduction

Sero-epidemiological studies of herpes simplex virus type-2 (HSV-2) infection in Africa remain difficult to interpret owing to the high rate of false-positive results observed when using the new recombinant gG2 HSV-2 ELISA tests. We compared the performance of two widely used gG2 ELISAs to derive an appropriate testing algorithm for use in South Africa.

Methods

Sera from 210 women attending family planning clinics in Johannesburg were tested using HerpeSelect® and Kalon^™ HSV-2 gG2 assays. Sera from 20 discordant pairs, 44 concordant positive and 33 concordant negative samples were further tested by HSV Western Blot (WB). Sensitivity and specificity of each test and of combination algorithms compared to WB were calculated.

Results

HerpeSelect® had a sensitivity of 98% (95% confidence interval [CI]: 95–100) and specificity of 61% (95%CI: 48–74). Kalon^™ was less sensitive (89%, 95%CI: 83–94) but more specific (85%, 95%CI: 61–100). Seroprevalence may have been overestimated by as much as 14% by HerpeSelect®. Specificity was improved by raising the cut-off index for determination of a positive result for HerpeSelect® (to ≥3.5), but not for Kalon^™. HIV-1 infection reduced the specificity of HerpeSelect® to 30%. Improved sensitivity and specificity were obtained by a two-test algorithm using HerpeSelect® (≥3.5) as the first test and Kalon^™ to resolve equivocal results (sensitivity 92%, 95%CI: 82–98; specificity 91%, 95%CI: 79–98).

Conclusion

Newer HSV-2 serological tests have low specificity in this South African population with high HIV-1 prevalence. Two-step testing strategies could provide rational testing alternatives to WB.

INTRODUCTION

Herpes simplex virus type 2 (HSV-2) is a primary cause of genital ulcers and is one of the most prevalent sexually transmitted infections worldwide¹. Recent serological studies conducted among populations with no specific high-risk sexual behaviour characteristics in sub-Saharan Africa have shown prevalence rates that exceed those of similar populations in the USA and Europe². Up to 70% of high-risk HIV-1 seronegative and up to 85% of HIV-1 seropositive persons are seropositive for HSV-2 in sub-Saharan Africa^{2, 3}. However, sero-epidemiological studies of HSV-2 in Africa have been hampered by concerns that some of the newer HSV-2 ELISAs are associated with high rates of false-positive reactions in African sera. In an evaluation study of thirteen HSV-2 type-specific assays, the specificity ranged from 47 to 99%⁴. In this evaluation, the HerpeSelect® (Focus Technologies) was shown to have a high sensitivity (100%) but a low specificity (71%), while the Kalon^™ HSV-2 gG2 ELISA was one of the best performing tests (sensitivity 93% specificity 98%). Specificity was shown to be lower in HIV-1 seropositive individuals. In another study of sera from populations in South Africa, Zimbabwe, Kenya and Uganda using the HerpeSelect®⁵, 100% concordance with Western blot (WB) was observed in sera from Zimbabwe and South Africa, but was lower for samples from Kenya (96%) and Uganda (88%). More recently, a study comparing HerpeSelect® and Kalon^™ with WB in 120 HIV-1 seronegative men aged 18–24 years in Kenya showed a lower specificity for HerpeSelect® (40%) compared to Kalon^™ (79%)⁶. Another more recent study using 538 Ugandan samples tested with WB, two ELISA assays and a rapid test (Biokit^™) confirmed the lower specificity of HerpeSelect® (51%) which was improved by raising the cut-off value for positive results to 3.2. In the same study, the specificity of the Kalon^™ assay was found to be superior to HerpeSelect®; this was enhanced further by raising the cut-off for positive results to 1.5 which increased specificity from 88% to 92%⁷. This study did not find any significant difference in assay performance by HIV-1 serostatus.

While sensitive tests are more useful for diagnosis, higher levels of specificity are required in epidemiological studies where associations with other infections like HIV-1 are explored. Highly specific testing strategies are required to identify individuals who might benefit from HSV treatment interventions currently being evaluated in trials. Large-scale WB testing is costly, and not feasible in many settings in Africa. For these reasons, a comparative evaluation of the sensitivity and specificity of two HSV-2 specific ELISA-based serological assays was undertaken in a South African population where HIV-1 and HSV-2 prevalence are both high⁸.^{9, 10}

MATERIALS AND METHODS

A total of 210 women aged 18–46 years were recruited from a family planning clinic in Johannesburg, South Africa, during the period from August to November 2003. Serum samples collected from consenting women of unknown HSV-2 serostatus were tested for HSV-2 using the HerpeSelect® ELISA (Focus Technologies Inc., Cypress Hill, Ca) and the Kalon^™ HSV-2 gG2 ELISA (Kalon Biologicals Ltd, Aldershot, UK). Optical density (OD) readings for Kalon^™ and the normalized OD readings for HerpeSelect® were recorded. Samples with normalised OD readings <0.9 were recorded as negative, those with values >1.1 were recorded as positive, and those with intermediate values (0.9–1.1) were recorded as equivocal as per manufacturer’s instructions.

Using a pre-determined sampling strategy, a random selection of specimens with concordant results and all discordant results (with serum remaining) were shipped to the University of Washington, Seattle, USA, for evaluation using a gold standard WB assay which has been previously described¹¹. Samples with remaining serum were tested for HIV-1 using Abbott AxSYM HIV 1/2 gO (Abbott Diagnostics, Wiesbaden-Delkenheim, Germany) in South Africa only when it became apparent that HIV-1 serostatus might influence HSV-2 ELISA results⁴. Indeterminate results were resolved using BioRad Genetic Systems rLAV HIV-1 ELISA (BioRad Laboratories, Redmond, USA).

The sensitivity and specificity of the different tests were calculated, taking into account the sampling strategy, according to methods described by Hawkins et al¹². Only samples with (normalised) OD readings >1.1 were considered positive. Additional analyses were performed to investigate whether sensitivity and/or specificity of the tests could be improved by changing the cut-off values for positive specimens using receiver operator characteristic (ROC) curves and likelihood ratios. We specifically investigated the sensitivity and specificity of a higher cut-off value for HerpeSelect® of ≥3.5 as has been suggested by other authors^{5, 13}. The effect of age and HIV-1 serostatus on sensitivity and specificity were also explored.

All participants were volunteers who gave written informed consent to participate prior to any study-related procedures. This study was approved by the Human Research Ethics Committee of the University of the Witwatersrand and the research ethics committee of the London School of Hygiene & Tropical Medicine, and was conducted in accordance with good clinical and laboratory practice guidelines.

RESULTS

Population characteristics

Participants had a mean age of 25.6 years (range 18–46). The overall HSV-2 seroprevalence for this population varied by as much as 14%, depending on the test used. Of the 210 specimens tested, 168 (80%) of women were HSV-2 seropositive, 40 (19%) were negative, and 2 results (1%) were equivocal using HerpeSelect®. With the Kalon^™ assay, 138 (66%) of the samples were HSV-2 seropositive, 58 (28%) were seronegative, and 14 (7%) results were equivocal. HIV-1 results were available for 145 (69%) participants. The overall HIV-1 prevalence was 52%.

The results of testing using HerpeSelect® and Kalon^™ were compared. Of the 210 samples tested, 178 (85%) had concordant results for both tests: 138 (66%) were concordant positive, 42 (20%) were concordant negative and none were equivocal on both tests. Thirty-two specimens (15%) specimens had discordant results. In the samples with discordant results, the majority (n=30) were positive on HerpeSelect® but either negative (n=16) or equivocal (n=14) on Kalon^™. In two cases, samples were equivocal on HerpeSelect® and negative with Kalon^™. Overall, HerpeSelect® appeared to detect positive specimens more frequently than Kalon^™ (see table 1).

Table 1.

Comparison of HerpeSelect^™ and Kalon^™ test results with resolver Western Blot test for HSV-2 (WB) among 210 South African sera.

HerpeSelect test result	Kalon test result	Number and % of samples		Number tested with resolver test (WB)	Number and % positive with resolver test (WB)
Positive	Positive	138	66%	44	41	93%
Positive	Negative	301	14%	19	9	47%
Negative	Positive	0	0%	0	0	0%
Negative	Negative	422	20%	35	2	6%
Total		210	100.0%	98	52	53%

¹includes samples with equivocal Kalon^™ result

²includes samples with equivocal HerpeSelect ® result

Sensitivity and specificity

A sub-set of 19 samples with discordant ELISA results, 44 samples with concordant positive ELISA results, and 35 samples with concordant negative ELISA results were compared with WB (see table 1). Using the data in this table, sensitivity and specificity were calculated using a method which accounts for this sampling strategy¹² (see table 2). According to the manufacturer’s instructions, the sensitivity of HerpeSelect® was 98% (95% confidence interval [CI]: 95–100) and the specificity was 61% (95%CI: 48–74). The sensitivity of the Kalon^™ assay was 89% (95%CI: 83–94) and its specificity was 85% (95%CI: 61–100).

Table 2.

Sensitivity and Specificity of HerpeSelect ® and Kalon^™ compared with Western blot as a gold standard (see example of detailed calculations in Appendix I)

	Sensitivity % (95% CI)	Specificity % (95% CI)	Correctly classified %
Standard testing
HerpeSelect ® >1.1	98 (95–100)	61 (48–74)	80
Kalon™ >1.1	89 (83–94)	85 (61–100)	80
Modified cut-off value
HerpeSelect ® ≥3.5	94 (89–100)	87 (67–100)	85
HerpeSelect ® ≥3.3	96 (92–100.)	87 (67–100)	86
Kalon™ ≥1.0	92 (87–97)	75 (49–100.)	83

CI, Confidence Interval

Because both HerpeSelect® and Kalon^™ yield continuous results based on OD readings, it was possible to explore the sensitivity and specificity of the test depending on the cut-off value chosen to define a positive test. Initially, we examined the higher cut-off value for HerpeSelect® of ≥ 3.5, which has been proposed by others¹³. While this resulted in a decreased sensitivity (94%), the specificity was substantially improved (87%). Further exploration using ROC curves showed that 3.3 was the cut-off value for optimal sensitivity (96%) and specificity (87%), correctly classifying 86% of samples. For Kalon^™, further interpretation of the ROC curve suggested that there was nothing to be gained in terms of sensitivity by changing the cut-off above or below the recommended index of 1.1. We subsequently analysed whether using two ELISAs in combination improved sensitivity and specificity when compared to WB. The best combination was obtained when using HerpeSelect® at increased cut-off (>3.5) followed by testing of “low positive” and equivocal samples with Kalon^™, yielding sensitivity of 92% (95%CI: 82–98) and specificity of 91% (95%CI: 79–98%). Using this approach 22 (10%) of the original samples tested by HerpeSelect would have required retesting with Kalon.

Effect of age and HIV-1 serostatus on assay performance

In an exploratory analysis, we investigated the effect of age and HIV-1 serostatus on assay performance (see table 3). The sensitivity of both tests was lower in the age group <25 years compared to those in the age group ≥25 years. Conversely, specificity was higher for both tests in the <25 years age group, compared to the older age group. The sensitivity of HerpeSelect® in HIV-1 seropositive specimens was high (100%). By contrast, its specificity was substantially lower in specimens of participants co-infected with HIV-1 (30%), but was improved by raising the cut-off to ≥3.5 (80%). The performance of Kalon^™ was broadly comparable (sensitivity 91%, specificity 72%).

Table 3.

Sensitivity and Specificity of HerpeSelect® and Kalon^™ by age and HIV-1 status.

	Age		HIV-1
	<25 years	≥25 years	HIV seronegative	HIV seropositive
Sensitivity %
HerpeSelect® >1.1	95	100	100	100
HerpeSelect® ≥3.5	92	96	98	97
Kalon™	82	91	80	91
Specificity %
HerpeSelect® >1.1	68	55	100	30
HerpeSelect® ≥3.5	87	93	100	80
Kalon™	87	86	100	72

Table 4 summarises the distribution of index values for samples that gave false positive results by either test when compared with WB. When comparing these values by age group, more false positive samples had index values in the low range (1.1–2.0) in the younger age group compared to the older age group, when tested by HerpeSelect® (4 vs. 2). This was not true for Kalon^™. In the false positive samples for which we had HIV-1 positive results, all four false positive samples (3 HerpeSelect®, 1 Kalon^™) were HIV-1 positive.

Table 4.

Index values giving false positive results for each of the tests.

	No. of samples in each category of index values
	1.1–2.0	2.01–3.0	>3.0	Total
HerpeSelect	6	2	4	12
Kalon	1	2	0	3

DISCUSSION

The sensitivity of HerpeSelect® and Kalon^™ observed in this study is high and similar to previous observations from other African settings where both HSV and HIV-1 prevalence are high^{4, 13, 14}, and compares favourably with the results from industrialised countries¹⁵.

We found a wide variation in specificity between the two tests, with HerpeSelect® demonstrating a high rate of false positive results, using the cut-off value recommended by the manufacturer. This resulted in an overestimation of seroprevalence in this population by as much as 14%. This is in contrast to observations by Hogrefe et al who found a specificity of 100% in sera from South Africa and Zimbabwe⁵, although this was similar to observations from other studies in Uganda, Kenya, Zambia, Benin and Nigeria, where specificity was as low as 40%–70%^{4, 6, 7, 13}.

There are several possible explanations for the higher sensitivity but lower specificity of HerpeSelect® compared with Kalon. One explanation is that HerpeSelect® is more sensitive than Kalon^™, and even WB, in detecting early seroconversion. A study comparing the median time to seroconversion of the three assays found that this was significantly longer for Kalon^™ (120 days) and WB (87 days, p=0.004), than for HerpeSelect® (21 days, p<0.001)¹⁶. A recent study among African patients with genital ulcer disease also found that rates of HSV-2 seroconversion in cases of documented first episodes of genital HSV-2 were significantly higher by HerpeSelect® compared to Kalon^™ (77% vs. 23% at Day 14)¹⁷. The high HSV-2 prevalence in this population suggests that seroconversion is not a rare event. In addition, 50% of the HerpeSelect® false positive tests had readings in the low positive range, which may be suggestive of early infection¹³. However, for this to be true, we would have expected to observe higher false positive rate in the younger age group, compared to the older age group. This was not the case in our study. In fact, we observed a lower sensitivity (of both assays) in the younger population compared to the older population.

An alternative explanation for the differences in specificity could be cross reactivity with other infections, including HSV-1 or HIV-1. While the glycoprotein-G2 tests are generally quite specific for HSV-2, one study found that, in patients with cultured-documented recurrent genital HSV-1 infection, the specificity of Kalon^™ was 100%, whilst the specificity of HerpeSelect® was slightly lower (93%)¹⁶. Golden et al showed the impact of HSV-1 on lowering the specificity of HerpeSelect ® in male STD clients¹⁸. We were unable to test whether there was cross-reactivity with HSV-1 because of the high prevalence of HSV-1 (98% of samples tested by WB) in this population. Perhaps a more plausible explanation relates to the presence of circulating non-specific antibodies, which could either could be the result of hyperglobulinaemia secondary to immune activation caused by HIV-1, or even could be the antibodies to HIV-1 themselves, which might cross react with the G2-specific portion of the test^{19, 20}. Specificity was also shown to be lower in HIV-1 seropositive samples in the analysis of samples from the Four African City Study⁴, for both tests, but substantially lower for HerpeSelect®. This is in contrast to Laeyendecker et al who did not observe any effect of HIV-1 serostatus on test performance, when comparing median index values for HerpeSelect® in HIV-1 infected and uninfected individuals¹⁴. A further study by the same group among Ugandan subjects did not reveal differences in assay performance by HIV-1 serostatus⁷. However, HIV-1 prevalence was lower in this Ugandan population (33%) than in our South African population (52%). Higher rates of co-infection with HIV-1 in the older age groups may also explain the higher specificity observed in the younger age group in our study. While not conclusive, we also noted in our study that all HSV-2 false positive tests with available HIV-1 results were indeed HIV-1 positive.

A third possible explanation is geographical variation in HSV-2 strains. Although data from Europe suggest that the gG2 epitope is fairly well conserved²¹, strains from African populations have not been sequenced. These strains may be more diverse and have different affinities for both the ELISA assays, as well as the WB. Certainly, atypical WB profiles were observed in this study (data not shown), and have been reported by other investigators ^{5, 13,6}

Raising the cut-off value for defining positive results for HerpeSelect® appeared to improve the specificity without compromising sensitivity too much and compares well with the Kalon^™ assay. This approach was eventually used as the strategy for identifying participants with HSV-2 for inclusion in two large multi-centre HSV-2 suppressive treatment trials^{22, 23}. We showed that the same approach did not yield similar improvements in performance for the Kalon^™ assay. This may be because this test is already fairly specific for HSV-2 and further improvements in specificity result in losses in sensitivity.

Finally we showed that using a combination of two tests resulted in high levels of sensitivity and specificity being obtained when compared with WB. Using HerpeSelect® with a higher cut-off and testing all equivocal results with Kalon^™ as the resolver test resulted in a testing algorithm which was suitably sensitive and specific, and only required re-testing of 10% of the original sample. Economic and operational research will be warranted to determine the role of these strategies in other settings. The demand for improved HSV-2 testing strategies is likely to grow with increasing awareness of the high prevalence of HSV-2 in the developing world, and its association with HIV-1 transmission.

In conclusion, high rates of false positivity continue to challenge the performance of the HerpeSelect® assay in African sera. In particular, the poor specificity of the test in HIV-1 seropositive populations warrants its cautionary use and a larger scale investigation. However, adjusting the cut-off and/or using a two-test testing algorithm resulted in significant improvements when compared with using either test alone. The feasibility and cost-benefit of such approaches should be further evaluated.

Key messages

• HSV-2 seroprevalence ranged from 66% with the Kalon^™ ELISA to 80% with the HerpeSelect ® ELISA.

• HerpeSelect® had a high sensitivity but lower specificity while Kalon^™ was less sensitive but more specific.

• HIV-1 infection appeared to reduce the specificity of HerpeSelect®.

• Specificity was improved by raising the cut-off index for determination of a positive result for HerpeSelect® (to ≥3.5), but not for Kalon^™.

Appendix I.

Example of calculation of sensitivity & specificity for HerpeSelect® based on Hawkins et al.

Index test (Focus)	Resolver test (WB)						Total
	Positive			Negative
	Kalon
	Positive	Negative	Total	Positive	Negative	Total
Positive	0.612	0.0681	0.680	0.045	0.075	0.120	0.800
Negative	0.000	0.011	0.011	0.000	0.189	0.189	0.200
Total	0.612	0.079	0.691	0.045	0.264	0.309	1.000

These steps were followed to calculate sensitivity and specificity for Focus ®:

Step 1. Using data from table 1 fill each of the cells. For example1, the value for this cell is calculated as 0.143 (proportion of samples with this result i.e. 30/210 HerpeSelect ® positive, Kalon ^™ negative) × 0.474 (proportion of these samples correctly resolved on WB. i.e. 9/19) = 0.680

Step 2. HerpeSelect ® sensitivity is total positive out of total i.e. 0.680/0.691 = 98.3%. Similarly, specificity is total negative out of total samples i.e.0.189/0.309=61.1%