Use of Epidemiologically Well-Defined Subjects and Existing Immunofluorescence Assays To Calibrate a New Enzyme Immunoassay for Human Herpesvirus 8 Antibodies

Abstract

Agreement between assays for the detection of human herpesvirus 8 (HHV-8) antibodies has been limited. In part, this disagreement has been because assay calibration (i.e., differentiating positive from negative results) has not been done in a standardized fashion with reference to a wide spectrum of HHV-8-infected (true-positive) and HHV-8-uninfected (true-negative) persons. To describe the performance of an assay for HHV-8 antibodies more accurately, we used epidemiologically well-characterized subjects in conjunction with testing on two existing immunofluorescence assays for HHV-8 antibodies to define two groups: a group of 135 HHV-8-infected individuals (true positives), including Kaposi's sarcoma patients and those asymptomatically infected, and a group of 234 individuals with a high likelihood of being HHV-8 uninfected (true negatives). A new enzyme immunoassay (EIA), using lysed HHV-8 virion as the antigen target, was then developed. With the above true positives and true negatives as references, the sensitivity and specificity of the EIA associated with different cutoff values were determined. At the cutoff that maximized both sensitivity and specificity, sensitivity was 94% and specificity was 93%. When the EIA was used to test a separate validation group, a distribution of seropositivity that matched that predicted for the agent of Kaposi's sarcoma was observed: 55% of homosexual men were seropositive, versus 6% seropositivity in a group of children, women, and heterosexual men. It is proposed that the EIA has utility for large-scale use in a number of settings and that the calibration method described can be used for other assays, both to more accurately describe the performance of these assays and to permit more-valid interassay comparison.

There are many demands on serologic assays for the detection of the newly discovered human herpesvirus 8 (HHV-8) also known as Kaposi's sarcoma-associated herpesvirus (3). Highly specific tests with good sensitivity are needed for epidemiologic studies of transmission. Depending upon what transmission routes are substantiated (1, 13, 18), highly sensitive tests may be needed for the screening of semen, organ, and/or blood donors. Finally, a test with both high sensitivity and specificity is needed for individual patient diagnosis.

Although first-generation antibody assays have been useful in confirming the causal role of HHV-8 in Kaposi's sarcoma (KS) (6, 12, 19; T. O'Brien, D. Kedes, D. Ganem, D. Macrae, and J. Goedert, Program Abstr. 6th Conf. Retrovir. Opportun. Infect., abstr. 198, 1999), agreement among assays has been limited (16). In part, this disagreement is because certain assays target different antibodies for which inherent sensitivity and specificity for HHV-8 infection may differ. In other instances, however, assay calibration (i.e., differentiating positive from negative results) has not been done in a standardized fashion with reference to a wide spectrum of HHV-8-infected (true-positive) and HHV-8-uninfected (true-negative) persons. Not only might this lead to interassay disagreement, but it also leaves in question the accuracy of sensitivity and specificity estimates for any one assay.

We have implemented a methodological approach that characterizes the performance of HHV-8 antibody assays more accurately. We first used information from well-characterized subjects in combination with testing on two first-generation immunofluorescence assays (IFAs) to assemble a calibration group that consisted of persons with either a high likelihood of being HHV-8 infected (true positives) or a high likelihood of being HHV-8 uninfected (true negatives). We then developed a new enzyme immunoassay (EIA) and used the calibration group to determine its sensitivity and specificity. Finally, we evaluated the EIA's performance in a separate validation group consisting of persons representing a wide spectrum of risk for HHV-8 infection.

(A portion of this work was presented at the 6th Conference on Retroviruses and Opportunistic Infections, 2 February 1999, in Chicago, Ill. [abstract 485] and at the 3rd National AIDS Malignancy Conference, 26 May 1999, in Bethesda, Md. [abstract C066].)

MATERIALS AND METHODS

Immunofluorescence assays for HHV-8 antibody used in selecting calibration group subjects.

To aid in selecting a calibration group, we used two previously described IFAs. The first, chosen for its high specificity, tests for antibodies to HHV-8 latency-associated nuclear antigen (LANA IFA) (9). The second, a modification of the method of Lennette et al. (10), was chosen for its high sensitivity and tests for both antibodies to replication-associated antigens (REPA) and LANA; we refer to this as the REPA/LANA IFA. We used the LANA IFA to help identify the true-positive component of the calibration group and the REPA/LANA IFA to identify the true-negative component.

LANA IFA.

This assay was performed as originally described (9). With KS patients as the “gold standard,” the assay's sensitivity is 83% (9). Because sensitivity may not be as high in asymptomatic HHV-8-infected persons, we conservatively estimated sensitivity to be 70% when applied to KS patients and asymptomatic infected persons. Previously, only 2 of 404 women, blood donors, and heterosexual men were reactive in the assay (9, 12). If it is conservatively assumed that these two persons were uninfected, the assay's specificity is 402 out of 404 (99.5%).

REPA/LANA IFA.

This assay was performed by modifying the method of Lenette et al. (10). In brief, BCBL-1 cells were induced with tetradecanoyl phorbol ester acetate (Sigma, St. Louis, Mo.) and were spotted onto slides. One modification was to soak slides before testing in phosphate-buffered saline containing 0.1% Triton X-100 (Sigma). A second modification was to prepare a separate mixture of cells (uninduced BCBL-1 cells in a 1:1 ratio with induced BJAB cells) that allowed us both to assess for reactivity against LANA and to control for nonspecific reactivity. Sera were incubated on slides, at dilutions of 1:20, 1:50, and 1:100, followed by successive incubation with anti-human immunoglobulin and conjugate (10). All sera were incubated in parallel on a spot with induced BCBL-1 cells and on a spot with uninduced BCBL-1/induced BJAB cells. Each spot was graded (0, 1+, 2+, 3+, or 4+) for intensity of staining and percentage of cells staining.

We evaluated reactivity against REPA and LANA independently. Our mouse monoclonal antibody to K8.1 protein (B. Forghani, L. Wu, Z. Amad, R. Renne, D. Kedes, and D. Ganem, Program Abstract. 22nd Int. Herpesvir. Workshop, abstr. 487, 1997) defined maximal reactivity against REPA. This antibody reacted with 20 to 30% of induced BCBL-1 cells with homogeneous cytoplasmic staining of 3+ to 4+ intensity, compared with less than 1% of cells in the uninduced BCBL-1/induced BJAB spot; we considered this the criteria for a strongly reactive anti-REPA result. Sera exhibiting specific cytoplasmic staining but not meeting the criteria for strong reactivity (i.e., cytoplasmic staining of 20% or more of induced BCBL-1 cells but with less than 3+ intensity or staining of any intensity with less than 20% of induced BCBL-1 cells) were classified as weakly reactive against REPA. Sera demonstrating staining (of any intensity) against all cells in both the induced BCBL-1 and uninduced BCBL-1/induced BJAB spots were defined as having nonspecific reactivity. Any sera exhibiting punctate nuclear staining in 50% of cells in the uninduced BCBL-1/induced BJAB spot were classified as anti-LANA reactive.

When assigning overall REPA/LANA IFA results, we initially classified subjects as strongly reactive, weakly reactive, nonreactive, or nonspecific based upon their anti-REPA results. Because of the very high specificity of antibodies to LANA (6, 9, 22), we reclassified those subjects initially categorized as nonreactive or weakly reactive on the basis of their anti-REPA result as strongly reactive for their overall REPA/LANA IFA result if they were anti-LANA reactive. The sensitivity of the IFA by Lennette et al. is 97% as defined in KS patients (10). Because our REPA/LANA IFA is slightly modified, we directly estimated its sensitivity by its performance in the true-positive component of our calibration group (defined below). We estimated its lower limit of specificity by determining the percentage of candidates for the true-negative component of our calibration group (defined below) who tested nonreactive in the assay.

Enzyme immunoassay.

After inducing BCBL-1 cells with tetradecanoyl phorbol ester acetate for 7 days, the cells were removed, and the resulting supernatant underwent ultracentrifugation to yield HHV-8 virions. The virions were then lysed with Tris (hydroxymethyl) aminomethane-buffered saline (0.05 M Tris, 0.15 M NaCl, 1% NP-40, 1% deoxycholate, pH 7.4), were sonicated for 4 min., and were then clarified to remove unsolubilized material. The lysed virions were then titrated to determine the optimal dilution for coating microtiter plates. BJAB cells were treated identically to serve as a control.

Sera at a 1:50 dilution were incubated in EIA wells for 60 min at 37°C. Each serum sample was tested in parallel in an HHV-8-lysate-coated well and a well coated by the product of the BJAB preparation. Wells were washed three times with 0.5% Tween 20 and were then incubated with alkaline phosphatase-labeled goat anti-human immunoglobulin followed by an enzyme substrate solution. The reaction was measured spectrometrically with dual beams at 405 and 630 nm. The final optical density (OD) was determined by subtracting the OD of the BJAB well from the OD of the HHV-8-coated well.

Calibration and validation groups. Calibration group.

The purpose of the calibration group was to serve as a reference for the determination of the sensitivity and specificity of the EIA. In the true-positive component of the calibration group, we included 59 patients with biopsy-confirmed AIDS-related KS from the San Francisco Men's Health Study (SFMHS) (25) and from local clinics. To broaden the spectrum of HHV-8 infection beyond those individuals with KS, we also chose 45 human immunodeficiency virus (HIV)-infected and 31 HIV-uninfected homosexual men without KS who had previously tested reactive in the LANA IFA (12). We included these men without KS because persons with a high pretest probability (or prevalence) of infection who test positive on a highly specific assay for infection have a very high posttest probability of being truly infected (21). We estimated their posttest probability of being infected with HHV-8 by using Bayes theorem (Fig. 1A) (21). Estimates of HHV-8 prevalence in homosexual men range from 12 to 93% (2, 5, 6, 7, 9–11, 14, 22); we initially assumed a prevalence of 50%. Although the REPA/LANA IFA also provides information on anti-LANA reactivity, we intentionally used only LANA IFA results when selecting persons for the true-positive group because the LANA IFA format has substantial documentation regarding its very high specificity (9, 12).

FIG. 1.

Calculation of the posttest probability of being HHV-8 infected or uninfected by using the pretest probability of infection, results from testing in an immunofluorescence assay, and a 2 by 2 table. Panel A depicts a generic population and outlines the steps involved in the calculation. A high-risk group of homosexual men without KS is depicted in panel B, and a low-risk group of children, virginal women, and heterosexual men is depicted in panel C. A population of 1,000 individuals is chosen in panels B and C for ease in calculations. Justification for the pretest probabilities and assay sensitivity and specificity values is found in the methods. LANA IFA refers to an immunofluorescence assay, using isolated nuclei from BCBL-1 cells as target antigen, for antibodies directed against latency-associated nuclear antigen. REPA/LANA IFA refers to an immunofluorescence assay for antibodies directed against both replication-associated and latency-associated antigens.

To construct the true-negative component, we utilized the concept that persons with a low pretest probability of infection who test negative on a very sensitive assay have a very high posttest probability of being truly uninfected (21). Accordingly, we selected persons from groups with the relatively lowest prevalence of HHV-8 who also tested overall nonreactive in the REPA/LANA IFA. We sampled 95 18-month-old children, 56 virginal adult women (J. Ruiz, F. Molitor, and W. McFarland, Program Abstr. 12th World AIDS Conf. abstr. 23458, 1998), and the 90 heterosexual men from the SFMHS with the fewest number of female partners; they are referred to as the true-negative group candidates. To determine the posttest probability of being HHV-8 uninfected in those true-negative-group candidates who tested nonreactive in the REPA/LANA IFA, we again used Bayes theorem. We conservatively estimated that HHV-8 prevalence in the true-negative-group candidates was as high as 20%, based upon the highest estimate to date (10). After demonstrating that the true-negative group candidates who tested nonreactive in the REPA/LANA IFA had a very high likelihood of being uninfected (see results), we defined them as our initial true-negative group. We were concerned, however, that including only persons who tested nonreactive in the REPA/LANA IFA and excluding those with nonspecific or weakly reactive results might deplete the true-negative group of truly uninfected persons who nonetheless demonstrate reactivity in the EIA. This would result in overestimating EIA specificity (17). Therefore, we defined a second true-negative group consisting of true-negative-group candidates who tested either nonreactive, weakly reactive, or nonspecific in the REPA/LANA IFA.

Validation group.

To further evaluate the performance of the EIA, 351 individuals different from those in the calibration group were tested. This group consisted of 50 18-month-old children, 86 nonvirginal women (Ruiz et al., Program Abstracts 12th World AIDS Conf., abstr. 23458), 60 heterosexual men from the SFMHS, 78 HIV-uninfected homosexual men without KS from the SFMHS, and 77 HIV-infected homosexual men without KS from the SFMHS. Among SFMHS subjects, we evaluated the association between number of male intercourse partners within the previous 2 years and EIA seropositivity.

Statistical analysis.

Determination of EIA sensitivity and specificity. We evaluated each OD value observed upon testing members of the calibration group in the EIA as a potential cutoff value. At each different OD value, sensitivity was defined as the percentage of individuals in the true-positive group who tested equal to or greater than the cutoff value. Specificity was defined as the percentage of individuals in the true-negative group that had EIA values less than that particular cutoff value. The paired sensitivity and specificity estimates associated with each different cutoff value were graphically depicted in a receiver-operating characteristic (ROC) curve (21).

EIA evaluation using validation group.

After we determined the EIA cutoff value that resulted in the highest estimate of both sensitivity and specificity, we then estimated the seropositivity in each of the five validation subgroups by determining the percentage of subjects who tested equal to or greater than that cutoff value.

RESULTS

Assembling the calibration group.

In the true-positive group, in addition to KS patients, we included homosexual men without KS who were reactive in the LANA IFA after estimating that their posttest probability of being HHV-8 infected was 99% (Fig. 1B). Even if their pretest probability is as low as 20%, those who are reactive in the LANA IFA have an estimated 97% posttest probability of being HHV-8 infected.

To assemble the true-negative group, we first estimated the sensitivity of the REPA/LANA IFA. Considering either a strongly reactive or weakly reactive overall REPA/LANA IFA result as indication of HHV-8 infection and by using our true-positive group as reference, we found that the sensitivity of the REPA/LANA IFA was 123/135 (91%) (Table 1). When the 241 true-negative-group candidates (18-month-old children, virginal women, and heterosexual men) were evaluated in the REPA/LANA IFA, 178 (74%) tested nonreactive (Table 1). We therefore considered the lower limit of specificity for the REPA/LANA IFA to be 74%. With these estimates for the sensitivity (91%) and specificity (74%) of the REPA/LANA IFA and assuming that the pretest probability (prevalence) of HHV-8 infection among true-negative-group candidates was 20%, we then defined the 178 true-negative-group candidates who tested nonreactive in the REPA/LANA IFA as the initial true-negative group after estimating that their posttest probability of being HHV-8 uninfected was 97% (Fig. 1C). If the pretest probability is 10%, the posttest probability of being HHV-8 uninfected given a nonreactive REPA/LANA IFA result is 99%. Even if the pretest probability is as high as 20% and the sensitivity of the REPA/LANA IFA is as low as 80% (with a specificity of 74%), the posttest probability of being HHV-8 uninfected given a nonreactive REPA/LANA IFA result is 94%. To broaden the spectrum of the true-negative group, we defined a second true-negative group (n = 234) by adding those 56 true-negative-group candidates with weakly reactive or nonspecific REPA/LANA IFA results (Table 1) to the 178 with nonreactive results.

TABLE 1.

Detection of antibodies to HHV-8 in persons defined as being HHV-8 infected (true-positive group) and persons with the relatively lowest prevalence of HHV-8 infection (true-negative-group candidates) with an immunofluorescence assay for antibodies to replication-associated and latency-associated antigens (REPA/LANA IFA).

Subjects	No. tested	REPA/LANA IFA result (No. [%])
		Strongly reactive	Weakly reactive	Nonreactive	Nonspecific
True-positive group
Biopsy-confirmed KS	59	48 (81)	8 (14)	3 (5)	0
HIV+,a No KS, LANA IFA+b	45	36 (80)	2 (4)	5 (11)	2 (4)
HIV−,c No KS, LANA IFA+	31	29 (94)	0	2 (6)	0
Total	135	113 (84)	10 (7)	10 (7)	2 (1)
True-negative-group candidates
18-Month-old children	95	1 (1)	25 (26)	60 (63)	9 (9)
Virginal women	56	0	3 (5)	51 (91)	2 (4)
Heterosexual men	90	6 (7)	11 (12)	67 (74)	6 (7)
Total	241	7 (3)	39 (16)	178 (74)	17 (7)

^aHIV+, HIV type 1 seropositive. 

^bReactive in an IFA for antibodies to HHV-8 latency-associated nuclear antigen with isolated nuclei from BCBL-1 cells as target antigen. 

^cHIV−, HIV type 1 seronegative. 

EIA sensitivity and specificity.

At each OD value in the range of observed values for the 135 members of the true-positive group and the 178 members of the initial true-negative group, we estimated EIA sensitivity and specificity (Fig. 2). At an OD cutoff of 0.06, sensitivity was 100% and specificity was 84%. In contrast, at an OD of 0.49, specificity was 100% and sensitivity was 74%. The highest estimates of both sensitivity and specificity were found at an OD of 0.19 (sensitivity 94%, specificity 95%). The area under the ROC curve was 0.99. When we based specificity upon the 234 persons in the second definition of the true-negative group, similar results were obtained. At an OD cutoff of 0.06, sensitivity was 100% and specificity was 78%. At an OD cutoff of 0.53, specificity was 100% and sensitivity was 73%. The highest estimates of both sensitivity and specificity were seen at an OD of 0.19 (sensitivity 94%, specificity 93%). The area under the ROC curve was 0.98.

FIG. 2.

ROC curve depicting the sensitivity (y axis) and specificity (x axis) of an EIA for antibodies to HHV-8. Sensitivity was determined in reference to 135 true-positive individuals and specificity in reference to 178 individuals in the initial definition of the true-negative group. At each OD value observed upon testing the true-positive and true-negative individuals in the EIA, the associated sensitivity and specificity were calculated. The numeric OD values are given alongside the points.

EIA validation.

By using an OD cutoff of 0.19, among validation group subjects we found that HIV-infected homosexual men without KS had the highest HHV-8 seroprevalence (49 subjects seropositive out of 78 tested [63%]) followed by HIV-uninfected homosexual men (36 of 77 subjects [47%]). Seroprevalence was 10% (9 of 86 subjects) in nonvirginal women, 3% (2 of 60 subjects) in heterosexual men, and 0% (0 of 50 subjects) in 18-month-old children. Among SFMHS participants, HHV-8 seropositivity was directly associated with the number of male intercourse partners within the past 2 years. Seropositivity was 3% (2 of 60 subjects) in men with no male intercourse partners within the past 2 years, 28% (9 of 32 subjects) in men with one to five partners within the past 2 years, 60% (34 of 57 subjects) in men with 6 to 25 partners within the past 2 years, and 64% (42 of 66 subjects) in men with more than 25 partners within the past 2 years (P < 0.001, chi-square test for trend).

When tested with the REPA/LANA IFA, validation group subjects had similar results to that obtained in the EIA. Of those testing nonreactive in the IFA, 248 of 261 (95%) were negative in the EIA. Fifty-six out of 58 (97%) who were strongly reactive in the IFA were positive in the EIA, and 27 of 30 (90%) who were weakly reactive in the IFA were positive in the EIA. Two individuals with nonspecific results in the IFA were negative in the EIA.

DISCUSSION

The performance of a diagnostic assay is defined best by its sensitivity and specificity. Valid estimates of sensitivity and specificity depend, respectively, upon challenging an assay with a wide spectrum of individuals who truly have the condition in question (true positives) and a spectrum of persons who do not (true negatives). Estimating the sensitivity of early HHV-8 serologic assays was limited by having only KS patients available as true positives (7, 9, 14, 22). It is now recognized that, among HHV-8-infected persons, antibody titer is highest in KS patients (7). Therefore, determining sensitivity by evaluating only KS patients may result in estimates that are unrealistically inflated (17). Conversely, because clinical mimickers of KS exist (23), sensitivity may be underestimated if not all KS patients have biopsy confirmation. When estimating the sensitivity of our EIA, we took several steps to select a broad spectrum of truly infected persons. First, we selected only KS patients with biopsy confirmation. Second, to broaden the spectrum of infected individuals, we included individuals without KS. Although we estimated that these individuals without KS had a very high probability of being HHV-8 infected, to the extent that some are uninfected and had little or no reactivity in the EIA, our estimate of EIA sensitivity is, in fact, an underestimate of true sensitivity.

While early investigators at a minimum had the presence of KS as a basis for forming a true-positive group, assembling a true-negative group has been vexing due to the lack of any analogous gold standard certifying the absence of infection. For example, the use of blood donors (15, 16, 22, 24) is problematic because donors may include homosexual men. Although the specificity of the IFA by the method of Lennette et al. has been questioned (8), we were able to take advantage of its very high sensitivity to define a group with a very high likelihood of being HHV-8 uninfected. To emphasize the robustness of our inferences, we intentionally chose the highest estimate made by any study to date (20% [10]) for prevalence of HHV-8 infection in our true-negative-group candidates, and we used the lower limit of REPA/LANA IFA specificity (74%) when estimating the posttest probability of being HHV-8 uninfected. To the extent that HHV-8 prevalence in our true-negative-group candidates is lower than 20% and REPA/LANA IFA specificity is higher than 74%, the posttest probability of being HHV-8 uninfected given a nonreactive REPA/LANA IFA result is even higher than 97%. Finally, although the probability of being HHV-8 uninfected among our true negatives is quite high, some persons may indeed be infected. To the extent that some are infected and show substantial EIA reactivity, our estimate of EIA specificity is, in fact, an underestimate of true specificity.

There are limitations to our approach. Just as the LANA IFA does not have 100% sensitivity in KS patients, it may have incomplete sensitivity in asymptomatically infected persons. Therefore, using reactivity in the LANA IFA as the basis for selecting asymptomatically infected persons to broaden our true-positive group may have resulted in sampling only the most easily detectable asymptomatic persons. Although we cannot assess the extent to which this has occurred, we believe that the inclusion of asymptomatically infected persons, even if not entirely representative, is a substantial improvement towards the realistic assessment of assay sensitivity.

By using a wide spectrum of true-positive and true-negative individuals as a reference, we determined that the sensitivity and specificity of our EIA were very high. We attribute the high sensitivity to the inclusion of the entire array of antigens present in the virion; recent work has demonstrated that no single antigen provides 100% sensitivity (26). A similar assay has been described but requires a more extensive purification process (4). Despite a less-rigorous purification procedure, we maintained high specificity, which we attribute to our subtraction of the OD obtained by parallel testing in a BJAB-based control well.

For an EIA that does not completely distinguish true positives from true negatives (i.e., does not have simultaneous 100% sensitivity and specificity) there is no single correct cutoff. The choice of cutoff depends upon desired use. For example, if it is desired to screen organ donors, the cutoff associated with 100% sensitivity should be used. Alternatively, the cutoff associated with 100% specificity should be used in epidemiologic work that aims to identify risk factors for HHV-8 infection (20) or when seeking to identify asymptomatic HHV-8-infected individuals for enrollment in trials to prevent KS. However, if no single cutoff value with near 100% sensitivity and 100% specificity exists, then the assay cannot be used to definitively diagnose individual patients who test higher than the cutoff value associated with 100% sensitivity but lower than the cutoff value associated with 100% specificity.

Our approach to defining EIA cutoffs differs from previous studies (4, 5, 15, 16, 19, 22, 24). Others have set cutoffs by taking the mean of between 5 and 40 individuals and then adding two to five standard deviations. In addition to the possibility that these individuals might actually be infected or might not represent a wide spectrum of uninfected persons, simply using a small sample size upon which to base the calculation may produce substantial variation across studies. Likewise, substantial differences are expected depending upon whether two or five standard deviations are added to the mean. It is unclear how much these methodologic differences in cutoff generation account for the poor interassay agreement (16). A better assessment of agreement would be to have each assay examine sera from a panel of true-negative and true-positive subjects and then compare ROC curves, generated in a standard fashion for each assay.

The high sensitivity and specificity of our EIA combined with its rapid throughput make it useful for large-scale use in a number of settings. As noted above, however, it cannot be used to definitively diagnose the presence or absence of infection in all individual subjects. For this, a confirmatory test, similar to the algorithm used in HIV diagnosis, is needed. Development of a confirmatory test will require the same strategy that we have implemented, using a wide spectrum of true-positive and true-negative individuals as reference. Although 100% sensitivity and 100% specificity may not be attainable, to achieve near-complete discrimination of infected and uninfected persons, a combination of independent quantitative measurements of multiple antibodies directed against both replication- and latent-phase antigens will be needed.