Comparative Evaluation of Three Commercial Systems for Detection of High-Risk Human Papillomavirus in Cervical and Vaginal ThinPrep PreservCyt Samples and Correlation with Biopsy Results

Abstract

Genital human papillomavirus (HPV) is the etiologic agent of more than 99% of all cervical cancers worldwide, with 14 genotypes being considered oncogenic or “high risk” because of their association with severe dysplasia and cervical carcinoma. Among these 14 high-risk types, HPV-16 and -18 account for approximately 70% of cervical cancers. The aim of this study was to evaluate three FDA-approved HPV nucleic acid-based tests for the ability to predict high-grade cervical intraepithelial neoplasias (CIN2 or worse) in corresponding tissue biopsy specimens. Residual specimens (total n = 793, cervical n = 743, vaginal n = 50) collected in ThinPrep PreservCyt medium with a cytologic result of ≥atypical squamous cells of undetermined significance were tested by the Hybrid Capture 2 (HC2) assay (Qiagen, Gaithersburg, MD), the cobas HPV test (Roche Diagnostics, Indianapolis, IN), and the APTIMA HPV assay (Hologic, San Diego, CA). Genotyping for HPV-16 and HPV-18 was simultaneously performed by the cobas HPV test. Results were compared to cervical or vaginal biopsy findings, when they were available (n = 350). Among the 350 patients with corresponding biopsy results, 81 (23.1%) showed ≥CIN2 by histopathology. The ≥CIN2 detection sensitivity was 91.4% by the cobas and APTIMA assays and 97.5% by HC2 assay. The specificities of the cobas, APTIMA, and HC2 assays were 31.2, 42.0, and 27.1%, respectively. When considering only positive HPV-16 and/or HPV-18 genotype results, the cobas test showed a sensitivity and a specificity of 51.9 and 86.6%, respectively. While the HC2, cobas, and APTIMA assays showed similar sensitivities for the detection of ≥CIN2 lesions, the specificities of the three tests varied, with the greatest specificity (86.6%) observed when the HPV-16 and/or HPV-18 genotypes were detected.

INTRODUCTION

Infection with human papillomavirus (HPV) is extremely common, with an estimated 75 to 80% of sexually active adults acquiring the virus before the age of 50 (1). Exposure to HPV may yield a number of clinical outcomes, ranging from asymptomatic infection or benign warts to highly invasive cervical carcinoma. The pathogenesis of HPV is dependent on a number of factors, but it is now understood that certain genotypes of the virus are more closely associated with the development of severe disease, including cancer. HPV genotypes are commonly separated into two broad categories, (i) “low risk” (types 6, 11, 40, 42, 43, 44, 53, 54, 61, 72, 73, and 81) and (ii) “high risk” (types 16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 66, and 68), based on the degree of correlation between infection and the development of cervical cancer. Among the 14 high-risk (HR) genotypes, HPV-16 and HPV-18 have been estimated to account for approximately 70% of cervical cancers (2).

Because of the association of infection with HR HPV and the development of cervical cancer, the detection of HR genotypes with nucleic acid-based tests (HPV tests) is now considered an important component of cervical cancer screening guidelines (3–5). According to the American Society for Colposcopy and Cervical Pathology (ASCCP) and the American Society for Clinical Pathology, women 30 to 65 years of age may be screened by routine cytology (Pap smear) every 3 years or by both cytology and an approved HPV test every 5 years. Importantly, specific genotyping for HPV-16 and -18 is now recommended for patients ≥30 years of age who test positive for HR HPV but negative by cytology because of the increased probability that HPV-16- and/or HPV-18-positive patients will develop cervical dysplasia (6). Recent data have shown that the absolute risk that HPV-16- and/or HPV-18-positive women will develop high-grade (CIN2) or worse (CIN2+) cervical intraepithelial neoplasias is nearly 2-fold greater (11.4% versus 6.1%) than that for women positive for “other” HR HPVs and >10-fold higher (11.4% versus 0.8%) than that for women who are negative for HR HPV (7). These findings support the use of specific genotyping for HPV-16 and HPV-18 in the triage of patients being evaluated for cervical cancer.

In this study, we evaluated and compared three commercially available HR HPV tests (APTIMA HPV [Hologic, San Diego, CA], Digene Hybrid Capture 2 [HC2] [Qiagen, Gaithersburg, MD], and cobas HPV [Roche]) in a U.S.-based referral population by using cervical and vaginal samples collected in ThinPrep PreservCyt medium with a cytologic result of ≥atypical squamous cells of undetermined significance (ASCUS). In addition, we assessed the utility of specific genotyping for HPV-16 and HPV-18 by using tissue biopsy results as the reference standard.

MATERIALS AND METHODS

Study design.

Between February and August 2012, a total of 793 samples (cervical n = 743, vaginal n = 50) that were collected in ThinPrep Pap test vials containing PreservCyt solution (Hologic, Bedford, MA) and had a result of ≥ASCUS by routine cytology were included in this study. Each sample was tested by the Digene HC2 assay (Qiagen), which was the method routinely used in our laboratory at the time of this study. In addition, each sample was tested by the APTIMA HPV (Hologic) and cobas HPV (Roche) assays. The HC2 assay detects 13 HR genotypes, while the cobas and APTIMA assays target 14 HR types. Furthermore, the Roche cobas test simultaneously provides specific genotyping for HPV-16 and HPV-18, if they are present. Among the total of 793 patients, 350 (44.1%) had a colposcopy performed and a corresponding tissue biopsy result that was used as the reference standard. The decision to refer a patient for colposcopy was made by the health care provider on the basis of the 2006 ASCCP guidelines (8). The study protocol was reviewed and approved by the institutional review board at our center.

HR HPV testing.

Samples submitted for routine HPV testing by the HC2 assay were stored at ambient temperature and tested by the APTIMA HPV and cobas HPV assays within 14 days according to the manufacturer's instructions. The HC2 and cobas HPV tests are DNA based, while the APTIMA assay targets mRNA. All of the samples tested by the Roche cobas HPV test were analyzed for 14 HR HPV genotypes, with simultaneous genotyping for HPV-16 and HPV-18, if present. Genotyping for HPV-16 and -18 was not performed by the APTIMA assay, which requires the use of a separate APTIMA 16, 18/45 genotype assay (Hologic). Testing by HC2 was completed with the Rapid Capture System (Qiagen), while the APTIMA and cobas HPV assays were performed on the TIGRIS DTS system (Hologic) or the cobas 4800 (x480/z480) platform (Roche), respectively. The input volume for the HC2 assay is 4 ml, while both the APTIMA and cobas HPV tests use 1 ml of sample (Table 1). Although vaginal samples collected in ThinPrep PreservCyt medium are not an FDA-approved source, they were tested by all three platforms by a self-validated procedure, which was the same protocol outlined by the manufacturer for cervical/endocervical samples.

TABLE 1.

Comparison of the features of three commercially available HR HPV tests

Parameter	cobas HPVtest	APTIMA HPV test	HC2 test
Regulatory status
HR HPV	FDA approved	FDA approved	FDA approved
HPV-16/18 genotyping	FDA approveda	FDA approvedb	NAi
Target	L1 gene (DNA)	E6/E7 mRNA	Full-genome probes
Detection chemistry	TaqMan	TMAc	RNA-DNA HC/chemiluminescence
No. of HR HPV genotypes detected	14	14	13
Vol of sample required (ml)	1	1	4
Internal control	β-Globind	HPV-16 E6/7 RNA transcripte	NAf
Throughputg	∼375	∼1,000h	∼350
Observed inhibition/invalid-result rate (%)	0.6	0.6	NAf

^aHPV-16/18 genotyping is performed simultaneously with the same kit as the 14 HR HPV pool.

^bHPV-16/18 genotyping may be performed with a separate kit from the “other” HR HPV detection assay.

^cTMA, transcription-mediated amplification.

^dDetects cellular β-globin and monitors for specimen quality, extraction, amplification and detection steps of the assay.

^eSpiked internal control that monitors the target capture, amplification and detection steps of the assay.

^fThe HC2 assay does not include an internal control to monitor inhibition. Only samples with visible cell pellets (i.e., sufficient cellular material) were tested by the HC2 assay.

^gApproximate number of samples that can be tested in about 14-hours, assuming one instrument is used.

^hThroughput for the APTIMA HPV test was estimated with the TIGRIS platform.

ⁱNA, not available.

Tissue biopsy results.

The results of all tissue biopsies, including cervical biopsies, vaginal biopsies, and endocervical curettage, were reviewed in a blinded fashion by one of us (M.R.H.) for comparison with the original tissue diagnosis. The review of tissue biopsies followed the 2012 Lower Anogenital Squamous Terminology (LAST) consensus recommendations from the College of American Pathologists and the ASCCP (9). When the review resulted in an increased or decreased grade compared with the original diagnosis, the biopsy was reviewed by a third pathologist (S.E.K.) in a blinded fashion. The diagnosis with two exact agreements was used as the consensus diagnosis. If all three diagnoses disagreed, the cases were reviewed by the two review pathologists to arrive at a consensus diagnosis. In addition, cases with a consensus diagnosis of CIN2 were confirmed by p16^INK4a staining (CINtec p16; Ventana Medical Systems, Tucson, AZ) as recommended in the 2012 LAST guidelines (9). Cases lacking positive p16^INK4a staining were considered to be <CIN2+.

Statistical analysis.

The HPV testing results were compared with the tissue biopsy results, with an interpretation of ≥CIN2 (Table 2) or CIN3+ (Table 3) being considered the reference standard for a positive result. An interpretation of <CIN2 was considered the reference standard for a negative result. Data were analyzed to calculate overall agreement, percent sensitivity, percent specificity, percent positive predictive value (PPV), and percent negative predictive value (NPV) with GraphPad QuickCalcs. The 95% confidence intervals (CIs) were calculated by the modified Wald method.

TABLE 2.

Comparison of three commercially available HR HPV tests to a reference standard of ≥CIN2 on the basis of 350 samples

HPV test and result	No. of samples with tissue diagnosis of ≥CIN2		% Agreement (95% CI)	% Sensitivity (95% CI)	% Specificity (95% CI)	% PPV (95% CI)	% NPV (95% CI)
	Positive	Negative
APTIMA
Positive	74	156	53.4 (48.2–58.6)	91.4 (82.9–96.0)	42.0 (36.3–48.0)	32.1 (26.5–38.5)	94.2 (88.2–97.4)
Negative	7	113
HC2
Positive	79	196	43.4 (38.3–48.7)	97.5 (90.9–99.9)	27.1 (22.2–32.8)	28.7 (23.7–34.4)	97.3 (90.2–99.8)
Negative	2	73
cobas HPV poola
Positive	74	185	45.1 (40.0–50.4)	91.4 (82.9–96.0)	31.2 (26.0–37.0)	28.6 (23.4–34.4)	92.3 (84.7–96.5)
Negative	7	84
cobas HPV-16/18
Positive	42b	36c	78.6 (74.0–82.6)	51.9 (41.1–62.4)	86.6 (82.0–90.2)	53.9 (42.9–64.5)	85.7 (81.0–89.4)
Negative	39d	233

^aIncludes HPV genotypes 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 66, and 68.

^bOf these samples, 41 were also positive for HR HPV mRNA by the Gen-Probe APTIMA test (16/18 genotyping not performed).

^cOf these samples, 31 were also positive for HR HPV mRNA by the Gen-Probe APTIMA test (16/18 genotyping not performed).

^dOf these samples, 32 were positive for other HR HPV (non-genotype 16/18) by the Roche cobas assay. In addition, 33 of these samples were positive for HR HPV mRNA by the Gen-Probe APTIMA test.

TABLE 3.

Comparison of three commercially available HR HPV tests to a reference standard of CIN3+

HPV test and result	No. of samples with tissue diagnosis of CIN3+		% Agreement (95% CI)	% Sensitivity (95% CI)	% Specificity (95% CI)	% PPV (95% CI)	% NPV (95% CI)
	Positive	Negative
APTIMA
Positive	39	191	44.9 (39.7–50.1)	95.1 (83.0–99.5)	38.2 (33.0–43.7)	17.0 (12.6–22.4)	98.3 (93.8–99.9)
Negative	2	118
HC2
Positive	39	236	32.0 (27.3–37.1)	95.1 (83.0–99.5)	23.6 (19.2–28.7)	14.2 (10.5–18.8)	97.3 (90.2–99.8)
Negative	2	73
cobas HPV poola
Positive	39	220	36.6 (31.7–41.7)	95.1 (83.0–99.5)	28.8 (24.0–34.1)	15.1 (11.2–20.0)	97.8 (91.9–99.9)
Negative	2	89
cobas HPV-16/18
Positive	26	52	80.9 (76.4–84.7)	63.4 (48.1–76.5)	83.2 (78.6–87.0)	33.3 (23.8–44.4)	94.5 (91.0–96.7)
Negative	15	257

^aIncludes HPV genotypes 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 66, and 68.

RESULTS

Comparison of three commercial HPV tests.

Among a total of 793 samples tested, the APTIMA and cobas HPV assays both showed an overall inhibition/invalid-result rate of 0.6% (5/793) (Table 1). A positive HR HPV result was obtained by all three assays with 324/793 (40.8%) samples, with the HC2, APTIMA, and cobas HPV tests being uniquely positive with 34 (4.2%), 4 (0.5%), and 23 (2.9%) samples, respectively (Fig. 1A). A negative result was obtained by all three HPV tests with 342/793 (43.1%) samples. Interestingly, the APTIMA HPV test was uniquely negative with 47 (5.9%) samples, compared to 5 (0.6%) samples by HC2 and 14 (1.8%) specimens by the cobas HPV assay (Fig. 1B).

FIG 1.

Venn diagrams showing the direct correlation of all 793 samples that were either positive (A) or negative (B) by one or more commercial HR HPV tests.

Comparison of three commercial HPV tests to a reference standard of ≥CIN2.

Of our 793 patients, 350 (44.1%) had colposcopy with a biopsy performed to which the results of the three HPV tests were compared. The remaining 443 (55.9%) patients were not referred for colposcopy. Among the 350 patients with a corresponding biopsy result, the APTIMA and cobas HPV assays demonstrated a sensitivity identical to that of a reference standard of ≥CIN2 (91.4% [95% CI, 82.9 to 96.0%]) for the detection of HR HPV. In comparison, the Digene HC2 test showed a sensitivity of 97.5% (95% CI, 90.9 to 99.9%), detecting HR HPV in 79 of 81 samples collected from patients with ≥CIN2 lesions (Table 2).

The percent specificities of the three HR HPV tests varied markedly from that of a reference standard of ≥CIN2. The APTIMA HPV test showed a specificity of 42.0% (95% CI, 36.3 to 48.0%), detecting HR HPV mRNA in 156 samples from patients with a biopsy result of <CIN2. In contrast, the HC2 and cobas assays demonstrated specificities of 27.1% (95% CI, 22.2 to 32.8) and 31.2% (95% CI, 26.0 to 37.0%), respectively. The HC2 assay was positive for HR HPV in 196 samples interpreted as <CIN2 by biopsy, while the cobas assay was positive in 185 CIN2-negative samples (Table 2). When only the cobas HPV-16 and HPV-18 assay results were compared to the colposcopy results, the sensitivity was found to be 51.9% (95% CI, 41.1 to 62.4%); however, the percent specificity of the HPV-16 and -18 assay was the highest among all of the tests, with 233/269 (86.6%) samples interpreted as <CIN2 by histopathology also testing negative for HPV-16 and -18 by the cobas assay.

The PPVs of the APTIMA, HC2, and cobas HPV tests were 32.1, 28.7, and 28.6%, respectively, compared to a biopsy result of ≥CIN2. In contrast, the NPVs were determined to be 94.2% for the APTIMA assay, 97.3% for the HC2 assay, and 92.3% for the cobas HPV HR pool test. When the results of only the cobas HPV-16 and HPV-18 genotyping assay were compared to the biopsy results, the PPV and NPV were found to be 53.9 and 85.7%, respectively (Table 2).

Comparison of three commercial HPV tests to a reference standard of CIN3+.

Because of variability that may occur in the grading of tissue biopsy results, especially at the level of CIN2 lesions, we also compared the performance of the three HR HPV tests to an endpoint of CIN3+ (Table 3). These data showed that the APTIMA, HC2, and cobas HPV tests showed the same sensitivity of 95.1% (95% CI, 83.0 to 99.5), testing positive with 39 of 41 specimens with a corresponding biopsy result of CIN3+. In comparison, the percent specificities ranged from 23.6% (95% CI, 19.2 to 28.7) for the HC2 assay to 38.2% (95% CI, 33.0 to 43.7) for the APTIMA test (Table 3). When the performance of the cobas HPV-16 and -18 genotyping assay was specifically compared to a tissue biopsy result of CIN3+, the sensitivity and specificity were determined to be 63.4% (95% CI, 48.1 to 76.5) and 83.2% (95% CI, 78.6 to 87.0), respectively.

The PPVs of the APTIMA, HC2, and cobas HPV tests were 17.0, 14.2, and 15.1%, respectively, compared to a biopsy result of CIN3+, while the NPVs were 98.3, 97.3, and 97.8%, respectively. When the results of only the cobas HPV-16 and -18 genotyping assay were compared to a biopsy endpoint of CIN3+, the PPV and NPV were found to be 33.3 and 94.5%, respectively (Table 3).

DISCUSSION

In this study, we compared three commercial tests for the detection of HR HPV by using biopsy results as the reference standard in a U.S.-based referral population. Several past studies have compared the performance characteristics of HR HPV tests, but most of these have compared only two assays, typically using HC2 as the reference standard (10–16). Furthermore, very few studies have compared the results of multiple HR HPV assays to those of colposcopy with tissue biopsy (17, 18). Szarewski et al. (17) recently compared the results of seven HPV tests with the histopathology results of a cohort of 1,099 women who had been referred to colposcopy clinics in England because of abnormal cytology. Compared to an endpoint of CIN2+, the HC2, cobas, and APTIMA HPV tests demonstrated sensitivities of 96.3, 95.2, and 95.3%, while the specificities were 19.5, 24.0, and 28.8%, respectively. The data from our study are similar, showing the sensitivity of the HC2 to be 97.5%, while both the cobas and APTIMA assays demonstrated a sensitivity of 91.4% compared to a reference standard of ≥CIN2. However, the percent specificities of the HC2 (27.1%), cobas (31.2%), and APTIMA (42.0%) HPV tests were higher in our study (Table 2), which may be due to the different study populations tested. While our results largely corroborate the findings of Szarewski et al., our study extends previously published work by (i) assessing the performance of these three HR HPV tests in a U.S.-based referral cohort, (ii) directly comparing the results of the cobas HPV-16 and HPV-18 genotyping assays to biopsy results with a blinded consensus diagnosis being used to establish the diagnosis, and (iii) applying the new LAST criteria with ancillary p16 staining for interpretation of tissue biopsies (9).

Current cervical cancer screening guidelines recommend the use of specific HPV-16 and -18 genotyping when samples test positive for HR HPV but are negative by cytology (3, 5). Women who are positive for HPV-16 and/or HPV-18 may be referred directly for colposcopy, while patients who are negative for HPV-16 and -18 should have repeat cotesting in 12 months. The data from our study, as well as others (7), demonstrate the high correlation of a positive result for HPV-16 and -18 and the presence of (or absolute risk of developing) ≥CIN2 lesions. Compared to a biopsy result of ≥CIN2, we observed a PPV of 53.9% for the cobas HPV-16 and -18 assay, compared to 28.7, 32.1, and 28.6% for the HC2, APTIMA and cobas HR pool assays, respectively (Table 2). However, the NPV of the cobas HPV-16 and -18 assay was found to be 85.7%, suggesting that a substantial percentage (∼48%, 39/81) of the ≥CIN2 lesions in our study were due to non-16 and -18 HR HPVs (Table 2). This is consistent with previous studies showing that ∼50% of CIN2+ lesions are caused by HPV-16 and/or HPV-18 (19).

This study has several limitations that should be discussed. First, this was not a longitudinal study and therefore did not address the possibility that patients testing positive for HR HPV and negative by biopsy may go on to develop cervical cancer. Second, we did not directly compare the cobas and APTIMA HPV-16 and -18 genotyping assays because of the requirement to test all samples with a separate HPV-16, HPV-18/45 genotyping test by the APTIMA system. Therefore, the results of specific HPV-16 and -18 genotyping in our study should be applied only to the cobas assay. Finally, all of the patients included in our study had a cytology result of ≥ASCUS. This was done to increase the number of positive HR HPV and biopsy results; however, this approach limits the conclusions that can be made to only a referral population. Further studies are needed to better define the comparative performance of HR HPV tests for screening purposes, either in conjunction with routine cytology or as stand-alone tests.

Despite these limitations, this study has a number of strengths, including the direct comparison of three FDA-approved HR HPV test results to biopsy results for a large number of patients (n = 350). In addition, biopsy results were confirmed by a blinded consensus review, and all CIN2 results were confirmed by immunohistochemical staining for p16^INK4a as recommended in the most recent interpretive guidelines (9). Our data show that these assays have similar sensitivities for the detection of patients with ≥CIN2 lesions, but the specificities vary, ranging from 27.1% for HC2 to 42.0% for APTIMA HPV. This observation aligns with those of past reports, showing that the mRNA-based APTIMA HPV test yields good specificity and may decrease the number of referrals for colposcopy compared to DNA-based HR HPV assays (20). Importantly, when we compared the results of the cobas HPV-16 and -18 tests directly to a biopsy endpoint of ≥CIN2, the percent specificity was observed to be highest at 86.6%. The increased specificity and PPV associated with the detection of HPV-16 and -18 in patients with ≥CIN2 lesions support the most recent guidelines that recommend the inclusion of HPV-16 and -18 genotyping in the triage of patients being evaluated for cervical cancer.