Human Papillomavirus Testing with the Hybrid Capture 2 Assay and PCR as Screening Tools

Abstract

The recognition of high-risk human papillomaviruses (HPVs) as etiological agents of cervical cancer has increased the demands to use testing for HPV for the detection of abnormal cervical smears and for cervical cancer screening. The present study compared the performance of the Hybrid Capture 2 (HC2) assay with that of PCR for the detection of significant cervical lesions in 1,511 women with different risks for HPV infections in three New Independent States of the former Soviet Union. The results showed that the level of agreement between the HC2 assay and PCR was substantial, with a kappa (Cohen) value of 0.669 (95% confidence interval [CI], 0.629 to 0.709). Of the 228 samples with discrepant results, 92 were positive by the HC2 assay but negative by PCR, whereas 136 samples were PCR positive but HC2 assay negative. The positive predictive values (PPVs) of the HC2 assay and PCR in detecting high-grade intraepithelial lesions (HSILs) were 4.5% (95% CI, 3.5 to 5.5%) and 3.6% (95% CI, 2.7 to 4.5%), respectively, and the negative predictive values (NPVs) were 99.6% (95% CI, 99.3 to 99.9%) and 99.3% (95% CI, 98.9 to 99.7%), respectively. The sensitivities of the HC2 assay and PCR for the detection of HSILs were 85.2 and 74.0%, respectively, and the specificities were 67.2 and 64.1%, respectively. In receiver operating characteristic (ROC) analysis, the performance of the HC2 assay for the detection of HSILs was excellent (P = 0.0001); the area under the ROC analysis curve was 0.858 (95% CI, 0.811 to 0.905), and the optimal balance between sensitivity (86.5%) and specificity (80%) was obtained with an HC2 assay cutoff level of 15.6 relative light units/positive control. Use of this cutoff would increase the specificity of the HC2 assay to 80.0% without compromising sensitivity. In conclusion, the results of PCR and the HC2 assay were concordant for 85% of samples, resulting in substantial reproducibility. Both tests had low PPVs, equal specificities, and equal (almost 100%) NPVs for the detection of HSILs; but the sensitivity of the HC2 assay was slightly better.

Cervical carcinoma is the second most common malignancy in women worldwide. Infection with high-risk (HR) types of human papillomaviruses (HPVs) is the single most important risk factor for cervical cancer and its precursors (27, 29). Screening for cervical cancer is traditionally based on Pap smear cytology, which suffers from subjectivity and which depends on the skills of the observer. The Pap test usually shows variable (poor to moderate) sensitivities (30 to 87%), and equivocal cases need to be repeated to improve diagnostics (4, 13). The recognition of HR HPVs as etiological agents of cervical cancer has increased the demands to use testing for HPV for the diagnosis of abnormal cervical smears (11) and even for screening for cervical cancer (9, 18). Testing for HPV has been shown to have higher sensitivities (84 to 100%) than the conventional Pap smear. Also, the negative predictive value (NPV) of testing for HPV DNA is very high; Clavel et al. (1) reported NPVs of 100 and 99.2% for testing for HPV DNA and the Pap test, respectively, whereas Ratnam et al. (15) reported lower figures: 94.3 and 78.1%, respectively. The specificities of testing for HPV have been lower than those of Pap tests in most studies, however (7, 11, 18).

During the past 10 years, PCR has been the “gold standard” technique in HPV diagnostics. The recognized disadvantages of PCR are its extremely high analytical sensitivity and potential for contamination, leading to false-positive results. A second-generation commercial HPV test, the Hybrid Capture 2 (HC2) assay, was introduced as a ready-to-use test for routine diagnostics and includes negative controls and positive controls (PCs). The performance characteristics of PCR and the HC2 assay for testing for HPV have been compared, and the results have varied (2, 10, 14, 19, 24, 26). Recently, several studies have shown that the performance of the HC2 assay is highly comparable to that of PCR for the detection of HPV DNA, with the levels of agreement being greater than 90% (14, 24, 26). Terry and coworkers (24) reported that the HC2 assay and PCR detected 88 and 84% of high-grade lesions, respectively. Contradictory to those findings, however, Cuzick et al. (2) showed that the HC2 assay and PCR have sensitivities of 95 and 74%, respectively, for the detection of histologically confirmed high-grade lesions. Schneede and coworkers (19) found that the two assays showed good agreement for the detection of high-grade lesions but substantial disagreement for the detection of HPV DNA among clinically normal women: a 27% prevalence of HPV DNA by the HC2 assay and only 13% by PCR.

A multicenter trial in three New Independent States (NISs) of the former Soviet Union with a cohort of more than 3,000 women with different risks for HPV infections recently tested the performances of the Pap test, the HC2 assay, and PCR as screening tools (21-23). The present report compares the performance of the HC2 assay with that of the PCR for the detection of clinically significant cervical lesions (high-grade intraepithelial lesions [HSILs]) in these women.

MATERIALS AND METHODS

Study population.

The subjects of this European Union-funded study consisted of 3,187 consecutive women attending six different outpatient clinics in three NISs of the former Soviet Union: Russia, Belarus, and Latvia. The study design was as described earlier (21, 23). The cohort of women consisted of women with different risk factors for HPV infection and cervical intraepithelial neoplasia (CIN): (i) patients who were being screened, (ii) gynecological outpatients, and (iii) those attending sexually transmitted disease (STD) clinics. Three tests were performed with specimens from all women: Pap smear, the HC2 assay, and the PCR test. The material used in the present study came from a cohort of 1,511 women (for whom the PCR test was performed), including 204 STD patients, 382 gynecological outpatients, and 925 women who participated in a local screening program. The mean age of the women was 32.9 ± 11.0 years (age range, 15 to 85 years) (22).

Sample collection.

All women were first subjected to a conventional cervicovaginal Pap smear. Pap smear abnormalities were interpreted and classified by using the 2001 Bethesda System (TBS 2001).

An additional sample for the detection of HPV was taken from the cervix by using the sampling kit for the HC2 assay. All samples tested for HPV were delivered to Turku, Finland, every second week and were immediately analyzed by the HC2 assay method. The rest of the sample was frozen and used for DNA extraction and subsequent testing for HPV DNA by PCR.

HR HPV detection by HC2 assay.

HPV DNA testing by the HC2 assay method was performed with the automated HC2 assay system according to the protocol of the manufacturer. The samples were analyzed only for the presence of HR HPV types 16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, and 68. HPV type 16 (HPV-16) DNA (1 pg/ml) was used as a PC. Samples were classified as HR HPV DNA positive if the relative light unit (RLU) reading obtained from the luminometer was equal to or greater than the mean value for the PC (21-23).

HPV detection by PCR.

DNA was extracted by the high-salt method (12). HPV DNA was detected by PCR with primers GP05+ and GP06+ (25). Confirmation of the specificities of the PCR products was done by hybridization with digoxigenin-labeled oligonucleotide probes specific for HR HPV types (types 16, 18, 31, 33, 35, 39, 45, 51, 52, 54, 56, and 58) (17). After hybridization, the positive spots on the film were graded according to the signal intensities as weak, moderate, or strong. In previous tests of the sensitivity of this PCR method, it was shown that approximately 20 copies of HPV, i.e., 20 SiHa cells mixed with 200 ng of human fibroblast DNA, give a weakly positive signal (17).

Controls for PCR.

To evaluate the PCR assay for possible contamination during DNA extraction, DNA was simultaneously extracted from cultured human fibroblasts. Additionally, every eighth sample tested by PCR contained no DNA. SiHa cell DNA diluted in normal human carrier DNA was used as a PC for HPV DNA detection. DNA extraction, the master mixture for PCR, and the addition of target DNA to the reaction mixture were all done in separate rooms.

Statistical analyses.

Statistical analyses were performed with the SPSS computer program package (version 11.5 for Windows). Frequency tables were analyzed by the chi-square test, with Pearson and likelihood ratio tests for the significance between the categorical variables. Odd ratios and the 95 or 99% confidence interval (CI) were calculated where appropriate by the use of accurate methods. Differences in the means of continuous variables between the groups were analyzed by nonparametric tests or analysis of variance, when applicable. In all analyses, probability values less than 0.05 were regarded as significant.

RESULTS

HPV DNA prevalence.

Among the cohort of 1,511 women included in this study, HR HPV DNA was detectable by the HC2 assay and PCR in 33.7% (509 of 1,511) and 36.6% (553 of 1,511) of the women, respectively. The concordance between the two tests is shown in Table 1. The two tests gave concordant results for 417 positive samples and 866 negative samples, with an overall level of agreement of 84.9% (Cohen's kappa = 0.669). Among the samples with discrepant results, 92 were positive by the HC2 assay but negative by PCR, whereas 136 samples were PCR positive but HC2 assay negative. The women testing HR HPV DNA positive by the HC2 assay but negative by PCR were younger (mean age, 29.8 ± 11.1 years) than those testing positive by PCR but negative by the HC2 assay (mean age, 35.1 ± 10.5 years) (P = 0.0001).

TABLE 1.

Concordance of results of HC2 assay and PCR for HPV detection^a

HC2 assay results	No. (%) of samples with the following PCR results:		Total
	Positive	Negative
HPV DNA positive	417 (81.9)	92 (18.1)	509 (100)
HPV DNA negative	136 (13.6)	866 (86.4)	1,002 (100)
Total	553	958	1,511

^aKappa (λ) = 0.669 (95% CI, 0.629 to 0.709).

Table 2 summarizes the results of HPV DNA detection by the two tests in the three patient categories. Altogether, 53.7% of gynecological outpatients were HPV DNA positive by either of the two tests, followed by 43.1% of the STD patients and 38.1% of the patients being screened. The HPV detection profiles for the three patient categories differed statistically significantly (P = 0.0001 by the chi-square test).

TABLE 2.

HPV detection by HC2 assay and PCR in patients in the three categories

Patient category	No. (%) of samples with the following resultsa:
	PCR+, HC2+	PCR+, HC2−	PCR−, HC2+	PCR−, HC2−	Total
STD	63 (30.9)	10 (4.9)	15 (7.4)	116 (56.9)	204 (100)
Gynecological	135 (35.3)	32 (8.4)	38 (9.9)	177 (46.3)	382 (100)
Screening	219 (23.7)	94 (10.2)	39 (4.2)	573 (61.9)	925 (100)

^a+, positive assay result; −, negative assay result. P = 0.0001 by Pearson chi-square test.

The results of HPV DNA detection in relation to Pap smear abnormalities are shown in Table 3. Of the patients classified as negative for intraepithelial lesions, HPV was detected in 40% of the samples by either or both of the tests. The rate of detection increased in parallel with the increasing severity of the cytological abnormality, with the rate increasing up to 91.7% for patients with HSILs. Among the samples from the latter group of patients, the HPV DNA in one sample (8.3%) was missed by both tests and only the HC2 assay detected HPV DNA in one sample. Importantly, of the 15 women with cytologically confirmed cervical cancer, both tests failed to detect HPV DNA in 3 (20%) women and only the HC2 assay detected HPV DNA in 2 (13.3%) women.

TABLE 3.

HPV detection related to cytological abnormalities in Pap smears

HPV profilea	No. (%) of samples with the following cytological gradeb:
	NIL	ASC	LSIL	HSIL	Cancer	Total
PCR+, HC2+	311 (24.7)	34 (50.0)	24 (64.9)	10 (83.4)	10 (66.7)	389
PCR+, HC2−	120 (9.6)	3 (4.4)	0 (0)	0 (0)	0 (0)	123
PCR−, HC2+	72 (5.7)	6 (8.8)	5 (13.5)	1 (8.3)	2 (13.3)	86
PCR−, HC2−	754 (60.0)	25 (36.8)	8 (21.6)	1 (8.3)	3 (20.0)	791
Total	1,257	68	37	12	15	1,389

^a+, positive assay result; −, negative assay result.

^bNIL, negative for intraepithelial lesions; ASC, atypical squamous cells; LSIL, low-grade intraepithelial lesions.

Table 4 summarizes the performance characteristics of the HC2 assay and PCR amplification for the detection of high-grade cytological abnormalities (HSILs and cancer). To reduce the possibility of error to 1%, we used the 99% CI in Table 4. The HC2 assay seemed to have a slightly better sensitivity, but otherwise, the performance of the HC2 assay was equivalent to that PCR. When the PCR signal was semiquantitatively graded as weak, moderate, or strong, the specificity and the positive predictive value (PPV) increased, with a significant loss of sensitivity. When the cytology cutoff was changed to low-grade intraepithelial lesions, the sensitivity of the HC2 assay dropped (81.2%) and the specificity (68.4%) and the PPV (10.2%) improved slightly, whereas the NPV slightly dropped (98.8%) (data not shown). When the cutoff was further changed to atypical squamous cells, the performance values for the HC2 assay were as follows: sensitivity, 69.7%; specificity, 69.7%; PPV, 18.1%; NPV, 96.0%.

TABLE 4.

Performance characteristics of HC2 assay and PCR in detecting HSILs or higher^a

Diagnostic test	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
HC2 testb	85.2 (82.9-87.5)c	67.2 (64.1-70.3)	4.5 (3.2-5.8)	99.6 (99.2-100)
PCRd	74.0 (71.1-76.9)	64.1 (58.8-69.4)	3.6 (2.4-4.8)	99.3 (98.7-99.9)
PCR
Weak	74.0 (71.1-76.9)	64.1 (58.8-69.4)	3.6 (2.4-4.8)	99.3 (98.7-99.9)
Moderate	55.5 (52.2-58.8)	82.2 (80.5-83.9)	5.3 (3.9-6.7)	99.0 (98.4-99.6)
Strong	33.3 (30.2-36.4)	92.3 (90.5-94.1)	7.3 (5.6-9.0)	98.7 (97.8-99.6)

^aAltogether, 27 patients had intraepithelial lesions graded HSIL or higher.

^bThe cutoff value for the HC2 assay was 1.0 pg/ml (n = 1,511 patients).

^cThe values in parentheses are 99% CIs.

^dPCR positive or negative.

The performance of the HC2 assay for the detection of HSILs is excellent, as shown by the results of receiver operating characteristic (ROC) analysis in Fig. 1. The area under the ROC analysis curve was 0.858 (95% CI, 0.811 to 0.905) (P = 0.0001), and the optimal balance between sensitivity (86.5%) and specificity (80%) was obtained when the HC2 assay cutoff level was approximately 15.6 RLU/PC. When this cutoff level was used to calculate the performance indicators for the HC2 assay, the sensitivity improved slightly (86.5%), whereas there was a significant improvement in specificity (80.0%) and a slight elevation in the NPV (7.4%). The PPV (99.7%) remained unchanged (data not shown).

FIG. 1.

ROC analysis of the HC2 assay for detection of intraepithelial lesions of HSIL or higher.

DISCUSSION

The aim of the present study was to compare the HC2 assay with the PCR for the detection of HPV DNA in a cohort of 1,511 women with different risk factors for HPV infection. The reproducibility measured by using Cohen's kappa value was 67%, which is of the same order of magnitude as that reported in previous studies (14). In our study the overall concordance of the two tests was 85%. In general, PCR detected more HR HPV-positive samples than the HC2 assay did. This is not unexpected, because in our setting the HPV type panel covered by these two tests was different, as described in Materials and Methods. HPV types 59 and 68 were included only in the HC2 assay panel, while PCR detected HPV-54. The rest of the most common HPV types were similarly covered by the two tests, however. Another reason for the different detection rates by PCR and the HC2 assay could be that the DNA used for the PCR was extracted from the transport medium specially designed for the HC2 assay. Although this medium has been shown to work, it is not ideal for this purpose because in this denaturizing liquid, DNA is in the single-stranded form and some DNA is lost in the extraction process. Use of the same samples for both DNA tests was done to avoid bias. By using SiHa and CaSki cells, we attempted to neutralize the transport medium before DNA extraction to improve the yield, but this procedure resulted in no marked differences in the test results (data not shown). To examine the differences in the discrepant results obtained by PCR and the HC2 assay, we sequenced the HPV DNA from four samples that were highly PCR positive but HC2 assay negative and five samples that were highly positive by the HC2 assay but negative by PCR. In the first case (PCR-positive, HC2 assay-negative samples) HPV-16 was found in one sample and had a true false-negative result by the HC2 assay. HPV-55 was found in one sample, and HPV-11 was found in another sample. These HPV types were not included in the panel for the HC2 assay. Nonspecific amplification of human chromosome 11 was found for one sample. Among the HC2 assay-positive, PCR-negative samples, one sample contained HPV-70, one sample contained HPV-58, and the last sample contained HPV-84. Nonspecific amplification of human chromosome 8 was found for one sample, and the last sample contained an unknown DNA sequence. Of these HPV types, HPV-58 was included in both the PCR and the HC2 assay panels and was thus a true false-negative result by PCR.

In the present series, the rates of detection of HR HPV were higher than in some studies reported recently (2, 14) but are in agreement with the rates reported from studies by Rietmuller et al. (16) and Venturoli et al. (26). As has been repeatedly emphasized, however, HPV detection rates are critically dependent on the target populations and the age groups tested, i.e., whether a screening population or patients attending STD or gynecologic clinics are being tested (21-23). Accordingly, Rietmuller et al. (16) detected overall rates of HPV positivity of 32.9% by the HC2 assay and an HR HPV positivity rate of 63.8% for a subgroup of women attending a colposcopy clinic, with an overall positivity rate of 37.8% by PCR. The age range of their patients was very similar to that in the present series (mean age, 32.9 ± 11.0 years; age range, 15 to 85 years), whereas in the other studies cited above, the women tested have generally been older. It is generally agreed that two of the subgroups of women (STD patients and gynecological outpatients) included in our study usually have more risk factors for HPV infections and CIN (20). The prevalence rate of HPV infections in NISs is also known to be very high (20). This study shows marked differences in the presence of HPV infections in women with different risk factors. Interestingly, PCR detected HPV infections better than the HC2 assay in the patients being screened, whereas the HC2 assay performed better in the STD patient subgroup. The reasons for these differences are not straightforward but might be related to (i) different HPV type distributions or (ii) different viral loads.

The HPV detection rates in women with normal cytologies have varied from 4.9 to 8% by the HC2 assay and 3 to 12% by PCR (2, 28). In our series, the prevalence rates of HPV in normal smears were higher, but they were still in agreement with those found in several previous studies (16, 19). Some of these discrepant results may be due to the different systems used to categorize Pap smears. Following the European practice, our Pap test results were originally classified by using a modified Papanicolaou classification and were subsequently adjusted to the TBS 2001 classification for statistical purposes only. Both class 1 and class 2 smears were included in the TBS 2001 category “negative for intraepithelial lesions.” Thus, in strict terms, this category contains smears that are completely normal by all criteria as well as those with reactive atypia for various reasons, making this category considerably heterogeneous. The possibility that some of these smears were in fact from patients with subclinical and latent HPV infections cannot be ruled out (20). Consonant with this view, the HPV detection rate for samples in the category of negative for intraepithelial lesions was somewhat higher by PCR (34.3%) than by the HC2 assay (30.4%) (Table 3). Similar data were recently reported by Rietmuller et al. (16), who found that among cytologically normal smears, 19.5% were positive by the HC2 assay and 25.1% were positive by PCR. This suggests that PCR detects more HPV-positive smears than the HC2 assay in smears in which no histological abnormalities can be detected. By definition, these are latent HPV infections, detectable only by molecular diagnostic tools (20). This is consonant with the known higher analytical sensitivity of PCR compared with that of the HC2 assay, with the former being capable of disclosing HPV infections at a very early phase, as well as infections likely to clear spontaneously without the patients ever developing abnormalities in the Pap smear.

These HPV test-positive, Pap smear-negative women have became major challenges for HPV testing, because transient anogenital HPV infections are very common, particularly in young, sexually active women (3, 6, 20). Because only a minor fraction of these transient HPV infections ever develop into persistent infections, detection of HPV DNA does not inevitably mean the presence of intraepithelial lesions or an increased risk for cervical cancer, which has been documented in an increasing number of studies, including studies with our cohort from NISs (3, 5, 21-23). On the other hand, persistent infections with HR HPV types predispose women to a significantly increased risk for the development of CIN and cervical cancer (8, 20). These data have important practical implications. If HPV DNA detection is used as the only indication for colposcopy referral, the result is a substantial proportion of unnecessary referrals (23). In the present setting, we found sensitivities of 85.2 and 74.1% and specificities of 67.2 and 64.1% for the detection of HSILs by the HC2 assay and PCR, respectively. In screenings for high-grade CIN and carcinomas, the estimated sensitivities for the detection of HR HPV types have been 81% and 83 to 88.4% by PCR and the HC2 assay, respectively, while the specificities have varied from 81.9 to 93% by the HC2 assay and 81.9% by PCR (9, 28). At this point, the different gold standards that are used, i.e., biopsy or cytology, should be emphasized. Both have their advocates, and neither is a 100% perfect gold standard. Biopsy is a good gold standard when it is positive for lesions, but it is less satisfactory when the biopsy result is negative. This is particularly embarrassing in cases in which colposcopy and cytology suggest that there is some pathology in the cervix. The same is true if cytology is used as the gold standard: when the result is positive, it works well, but when the result is negative, one cannot completely exclude the possibility of the presence of high-grade lesions. To provide meaningful data, we used the positive identification of HSILs as the standard to calculate the performance indicators for the HC2 assay and PCR, because the results were available for all women (unlike biopsy results, which were obtained for only a minority of the women).

In quantitative tests like the HC2 assay, test performance depends on the cutoff values used for test positivity (23). When the manufacturer-recommended cutoff was changed from 1 pg of HPV-16 DNA per ml to 15.6 RLU/PC, the sensitivity of the HC2 assay increased to 86.5% and the specificity increased to 80.0%, as shown by our ROC analysis. The same was shown to be the case with PCR, when the signals were semiquantitatively categorized as weak, moderate, and strong.

Apart from the conventional performance characteristics, another important measure of a diagnostic test is the rate of false-negative results. In our study both assays failed to detect HSILs in 1 of 12 (8.3%) samples, and only PCR failed to detect HSILs in another sample. Unfortunately, however, of the 15 samples from patietns with invasive squamous cell carcinomas (SCCs) confirmed by cytology, both the HC2 assay and PCR failed to detect SCCs in 3 (20%) samples and PCR failed to detect SCCs in another 2 samples. In light of the concept that cervical SCC is linked to HR HPV in nearly 100% of cases (26), this failure to detect 20% of the cases of SCC would be unacceptable. To disclose the reasons for this failure, we tested the samples in which SCC was missed by a TaqMan-based quantitative PCR method, and one sample was found to contain HPV-16 at a very low copy number. Two additional samples were negative by real-time PCR analysis; one sample contained a very small amount of human DNA, while the other had a large amount of HPV DNA (data not shown). In this case, the low copy number of the HPV genome in these samples, together with a deficient sample quality, undoubtedly played a role. These false-negative results for high-grade HPV would be a true clinical problem if HPV DNA detection were used as the only screening method. Other studies have reported similar experiences, in which false-negative results for high-grade HPV have not been rare (2, 16, 28).

In conclusion, the results of PCR and the HC2 assay were concordant for 85% of samples, resulting in substantial reproducibility. Both tests had low PPVs and equal (almost 100%) NPVs for the detection of HSILs, but the sensitivity and the specificity were slightly better for the HC2 assay. The HC2 assay is technically well designed and can be easily controlled and performed by the technician, whereas PCR contains several steps that must be carefully optimized, which makes it more difficult to standardize PCR in laboratory settings. In this respect, individual laboratories may safely select either test on the basis of their preferences.