Abstract
The cobas human papillomavirus (HPV) test, approved by the FDA in April 2011, is a fully automated assay for the detection of 14 high-risk (hr) HPV genotypes from cervical specimens collected in liquid-based cytology medium using real-time PCR amplification of the L1 gene and TaqMan probes. Results are simultaneously reported as positive or negative for the pooled 12 oncogenic HPV types (HPV31, -33, -35, -39, -45, -51, -52, -56, -58, -59, -66, and -68) from channel 1, with HPV16 and HPV18 genotypes read individually from channels 2 and 3. A fourth channel detects the human β-globin gene as a control for sample adequacy and assay inhibition. To optimize clinical sensitivity and specificity, cutoff values (cycle thresholds [CT]) were established for each channel based on the detection of cervical intraepithelial neoplasia grade 2 (CIN2) or greater (≥CIN2). For women aged ≥21 years with cytology results indicating atypical squamous cells of undetermined significance (ASC-US), CT values provided a sensitivity of 90% (95% confidence interval [CI], 81.5% to 94.8%) for the detection of ≥CIN2 and a specificity of 70.5% (95% CI, 68.1% to 72.7%). The analytic sensitivity (limit of detection) ranged from 150 to 2,400 copies/ml, depending on genotype. The analytic specificity, evaluated by comparing the HPV result with a combined comparator of Sanger sequencing and the Qiagen digene HC2 high-risk HPV DNA test (hc2), demonstrated overall positive agreement of 96.3% for 14 hrHPV types in women with ASC-US cytology results who were aged ≥21 years and 86.1% in women with NLIM (negative for intraepithelial neoplasia or malignancy) cytology who were aged ≥30 years. These and other performance validation studies demonstrate that the cobas HPV test is a fully automated and clinically validated robust test.
INTRODUCTION
Cervical cancer is caused by persistent infection with high-risk human papillomavirus (hrHPV). The incidence of invasive cervical cancer has been significantly reduced over the past 50 years by the widespread implementation of cytology-based screening using the Papanicolaou (Pap) test (1). However, because of the limited sensitivity of a single Pap test, cervical cancer and its precursors (cervical intraepithelial neoplasia grades 2 and 3 [CIN2 and CIN3, respectively]) remain significant public health issues. The recent development of assays that allow the detection of hrHPV DNA in cervical specimens has shown them to be more sensitive than the Pap test for the detection of CIN2 and CIN3 in a single screening round (2–4). To be clinically relevant, HPV assays should be based on clinical rather than on analytic sensitivity, since only a minority of HPV-positive women progress to high-grade cervical disease. Moreover, the clinical utility of an HPV test would be enhanced if it could also identify hrHPV-positive women who are at the highest risk of having or developing high-grade disease, because doing so has the potential to improve both sensitivity and specificity. Given the recent evidence for the significantly greater oncogenic risks of HPV16 and HPV18 relative to the other hrHPV types (5–8), an HPV test capable of specifically identifying these 2 genotypes might be expected to provide additional clinical value. Moreover, all HPV tests must demonstrate analytic sensitivity, accuracy, good reproducibility, and inclusivity and exclusivity.
The cobas HPV test is a fully automated real-time PCR DNA amplification test that received approval by the Food and Drug Administration in April 2011 and was developed to maximize clinical utility. The cutoff for a positive result was determined based on the ability of the assay to detect histologically confirmed high-grade cervical disease, defined as cervical intraepithelial neoplasia (CIN), grade 2 or worse (≥CIN2), rather than on the detection of a minimum number of viral copies. The unique test design also allowed simultaneous reporting of a pooled hrHPV result in addition to individual results for HPV16 and HPV18, the 2 most-oncogenic genotypes (5–8). Here, we report details on the development and validation of the cobas HPV test with emphasis on how its design and performance contribute to the establishment of clinical utility.
MATERIALS AND METHODS
Assay design and conditions.
The cobas HPV test is a highly automated assay for the detection of hrHPV DNA in liquid-based cytology (LBC) specimens using real-time PCR technology with a set of 16 PCR primers (8 forward and 8 reverse) that amplify a ∼200-bp fragment of the L1 gene from all 14 hrHPV genotypes. TaqMan probes labeled with 3 spectrally unique fluorescent dyes allow for the simultaneous detection of 14 hrHPV types from 3 separate channels with real-time PCR technology. Twelve hrHPV types (HPV31, -33, -35, -39, -45, -51, -52, -56, -58, -59, -66, and -68) are detected as a pool in channel 1, and the hrHPV16 and HPV18 genotypes are simultaneously detected individually in channels 2 and 3, respectively. The fourth channel detects a 330-bp amplicon generated from the human β-globin gene as a control for sampling adequacy; a positive β-globin result verifies the presence of human cells in the collection vial.
Specimen preparation is automated using the cobas x 480 instrument. A 1-ml aliquot from cervical specimens collected into 20 ml of PreservCyt LBC medium (Hologic, Bedford, MA) can be loaded onto the cobas x 480 instrument; additionally, residual samples remaining in ThinPrep vials (Hologic, Bedford, MA) may be loaded directly following processing on a ThinPrep processor (9). From each specimen, 400 μl is then digested under denaturing conditions at elevated temperatures and is lysed in the presence of a chaotropic reagent. The extracted DNA is purified through adsorption to magnetic glass particles, is washed and eluted from these particles into 150 μl, and then 25 μl of eluate is added to the PCR master mix for amplification by the cobas z 480 instrument (10).
When the HPV and β-globin targets are present in the sample, oligonucleotide probes labeled with a fluorescent reporter and a quencher bind to genotype-specific polymorphic regions within the sequences amplified by the primers and are subsequently cleaved by the 5′→3′ exonuclease function of the DNA polymerase, generating a fluorescent signal (11). Results, based on fluorescent growth curves (accumulation of fluorescence on the y axis versus cycle number on the x axis), are reported in 4 spectrally distinct channels: channel 1, 12 hrHPV genotypes, reported as a pooled result; channel 2, HPV16; channel 3, HPV18; and channel 4, human β-globin DNA. With one exception, all of the HPV TaqMan probes in the 3 HPV channels are genotype specific. One of the labeled HPV probes detected in the 12-hrHPV-pool channel is designed to detect the presence of a common sequence unique to 3 different hrHPV types: HPV35, HPV52, and HPV58.
Cutoff selection algorithm.
As a TaqMan-based real-time PCR assay, the cobas HPV test generates a growth curve and consequently requires the establishment of cutoff values or a cycle threshold (CT) to distinguish a positive from a negative sample. The objective for establishing a clinically meaningful cutoff value was to maximize the ability to detect histologically confirmed disease of ≥CIN2 (clinical sensitivity) while minimizing the number of HPV-positive samples that did not have cervical precancer (clinical specificity). From a public health perspective, the goal for a screening test is not to detect hrHPV at very low viral loads (analytic sensitivity) but to detect oncogenic HPV associated with precancerous lesions while achieving a crude sensitivity (i.e., without adjusting for verification bias) of ≥90% for the detection of ≥CIN2 (12). The target clinical sensitivity in the population with atypical squamous cells of undetermined significance (ASC-US) was set based on the reported sensitivity of 93% for the Qiagen Hybrid Capture 2 high-risk HPV DNA test (hc2) (Qiagen, Valencia, CA) (13), as prescribed by FDA guidelines. The cutoff was defined as a set of 3 values (c1, c2, and c3) corresponding to the CT values in each of the first 3 channels of the cobas HPV test that optimized test performance in the overall population of women enrolled in the ATHENA (Addressing THE Need for Advanced HPV Diagnostics) trial (14). Testing with the cobas HPV test was performed at 5 test sites: TriCore Reference Laboratories (Albuquerque, NM), Scott & White Memorial Hospital and Scott, Sherwood, and Brindley Foundation (Temple, TX), DCL Medical Laboratories (Indianapolis, IN), LabCorp (Burlington, NC), and Roche Molecular Systems, Inc. (Pleasanton, CA). On the basis of the method described by Kondratovich and Yousef (15), a range of CT values that provided an unadjusted sensitivity of 90% in the overall population was selected, and a cutoff CT was chosen that produced the maximum estimate of specificity among the possible CT values that satisfied the sensitivity requirement.
The clinical cutoff values to achieve the desired clinical sensitivities were selected in an initial study (phase 1) and validated in a subsequent study (phase 2). For phase 1, approximately 29,000 women were screened in the ATHENA trial, as described in detail previously (14). Briefly, a total of 47,208 women aged ≥21 years who presented for routine cervical cancer screening were enrolled in the ATHENA trial and had both liquid-based cytology and HPV testing. All women with abnormal cytology (≥ASC-US) and all women aged ≥25 years with normal cytology who tested positive for HPV were referred for colposcopy and biopsy. In addition, a subgroup of women aged ≥25 years with negative cytology and negative HPV results was randomly selected for colposcopy. Three intended use populations were defined in the trial: (i) women aged ≥21 years with ASC-US cytology results, (ii) women aged ≥30 years with negative for intraepithelial neoplasia (NILM) cytology, and (iii) women aged ≥25 years, independent of cytology result (“overall” population). A total of 4,609 women aged ≥25 years from the overall population and 874 women with ASC-US cytology results aged ≥21 years were selected, all of whom had valid biopsy results. On the basis of these results, cutoff values were determined in the first 3 channels (12 pooled hrHPV types, HPV16, and HPV18, respectively, for channels 1, 2, and 3) of the cobas HPV test to provide the desired clinical sensitivities in the overall population (90%) and in the ASC-US population (93%), while maintaining optimal specificity. The cutoff values selected in phase 1 of the study were validated by estimating unadjusted sensitivity and specificity from approximately 18,000 subjects enrolled in phase 2 of the ATHENA trial; 3,118 subjects (overall population) and 686 women (ASC-US population) had valid biopsy results. Overall, the cobas HPV test CT cutoff was determined using 4,609 biopsy specimens with adjudicated pathology to include 1 cancer, 6 adenocarcinoma in situ, 151 CIN3, 81 CIN2, 342 CIN1, 3,972 normal, and 56 inadequate tissue diagnoses.
HPV PCR Sanger sequencing protocol and combined comparator testing.
The concordance of the cobas HPV test with Sanger sequencing was evaluated using a sequencing protocol optimized and validated by Beckman Coulter Genomics (Morrisville, NC), in accordance with good laboratory practices. Cervical specimens were tested for the presence and absence of both high- and low-risk HPV genotypes by using the L1 PGMY primers (16). HPV-positive samples were defined as those with a detectable 450-bp band following gel electrophoresis. HPV-positive samples were then reextracted, and DNA was amplified with a set of 16 different-HPV-type specific primers (HPV73 and HPV82, in addition to the 14 types detected in the cobas HPV test). Purified amplicons were subjected to bidirectional Sanger sequencing.
Beckman Coulter Genomics received 2,150 clinical cervical samples and extracted DNA using a Qiagen QIAamp media MDx kit (Qiagen, Valencia, CA) and the Qiagen BioRobot universal system (Qiagen, Valencia, CA). Each DNA sample was PCR amplified using the β-globin and PGMY consensus HPV primers (16), and 10 μl of the amplicon was analyzed by agarose gel electrophoresis. Samples that tested positive for HPV in the PGMY PCR were reextracted in triplicate. The extractions were pooled and PCR amplified with each of 16 HPV-specific primers (the 14 hrHPV types described above, as well as HPV73 and HPV82). The amplicon size varied from 398 bp for HPV73 to 692 bp for HPV31. PCR products were purified enzymatically using exonuclease and shrimp alkaline phosphatase and sequenced on each strand using the subtype-specific forward and reverse primers to achieve bidirectional coverage.
For comparison with the cobas HPV test results in the 3 channels (12 pooled hrHPV types, HPV16, and HPV18 for channels 1, 2, and 3, respectively), a Sanger sequencing result showing a mixed infection with HPV16 and any other HPV type was compared only with the cobas HPV Test HPV16 result, and a Sanger sequencing result showing HPV18 and any other HPV type (other than HPV16) was compared only to the cobas HPV test HPV18 result.
To further establish the ability of the cobas HPV test to detect the targeted HPV genotypes, the results obtained from a subset of samples from the ATHENA trial were compared with the results obtained with a combined comparator consisting of Sanger sequencing and the Qiagen hc2 HPV DNA assay. Results of the combined comparator were defined as “positive” only for those samples that were positive by both Sanger sequencing and hc2. The samples that were discordant between Sanger sequencing and the hc2 assay were considered “indeterminate” and were not included in the comparison. This study included the 3 intended-use populations as defined above. Sample size was determined for each analysis population based on the required estimates for the primary objectives of the ATHENA trial (95% confidence interval [CI] width to be within 5% for detection of hrHPV, 7.6% for detection of HPV16, and 13.8% for detection of HPV18, at an estimated 90% agreement between methods). The number of subjects whose samples were selected for sequencing was 2,150 (Table 1). Subjects from the ATHENA clinical trial (7, 14) with ASC-US were selected systematically from the first 1,000 subjects referred for colposcopy. For women with NILM and >ASC-US cytology results, eligible women who were referred for colposcopy and who had valid hc2 test results were selected. The hc2 testing was performed according to the manufacturer's instructions (13).
Table 1.
Percent agreement of the cobas HPV test result versus the combined comparator result by HPV type
| Population (n) | Agreement | Results by HPV typea |
|||||
|---|---|---|---|---|---|---|---|
| HPV16 |
HPV18 |
12 other pooled HPV types |
|||||
| % (no./total no.) | 95% CI | % (no./total no.) | 95% CI | % (no./total no.) | 95% CI | ||
| ASC-USb (aged ≥21 yr) (985) | Positive | 97.2 (69/71) | 90.3–99.2 | 95.0 (38/40) | 83.5–98.6 | 94.6 (226/239) | 90.9–96.8 |
| Negative | 99.1 (918/926) | 98.3–99.6 | 100 (957/957) | 99.6–100 | 95.8 (726/758) | 94.1–97.0 | |
| Overall | 99.0 (987/997) | 98.2–99.5 | 99.8 (995/997) | 99.3–99.9 | 95.5 (952/997) | 94.0–96.6 | |
| NILMc (aged ≥30 yr) (731) | Positive | 100 (39/39) | 91.0–100 | 94.4 (17/18) | 74.2–99.0 | 88.4 (168/190) | 83.1–92.2 |
| Negative | 97.6 (689/706) | 96.2–98.5 | 99.2 (721/727) | 98.2–99.6 | 82.7 (459/555) | 79.3–85.6 | |
| Overall | 97.7 (728/745) | 96.4–98.6 | 99.1 (738/745) | 98.1–99.5 | 84.2 (627/745) | 81.4–86.6 | |
| Overall (aged ≥25 yr) (1,962) | Positive | 98.6 (141/143) | 95.0–99.6 | 91.9 (68/74) | 83.4–96.2 | 92.5 (491/531) | 89.9–94.4 |
| Negative | 98.2 (1,815/1,849) | 97.4–98.7 | 99.4 (1,907/1,918) | 99.0–99.7 | 88.4 (1,293/1,462) | 86.7–90.0 | |
| Overall | 98.2 (1,956/1,992) | 97.5–98.7 | 99.1 (1,975/1,992) | 98.6–99.5 | 89.5 (1,784/1,993) | 88.1–90.8 | |
CI, confidence interval; HPV, human papillomavirus.
ASC-US, atypical squamous cells of undetermined significance.
NILM, negative for intraepithelial lesion or malignancy.
Limit of detection, genotype specificity, and inclusivity.
All analytical studies, such as the limit of detection (LOD) and reproducibility, were carried out using the CT cutoff values derived from optimized clinical sensitivity and specificity as discussed above.
The LOD, defined as the level of HPV DNA in the sample that yields positive test results (above the clinical threshold) in ≥95% of the replicates, was determined for the cobas HPV test at the clinical cutoff for genotypes HPV16, HPV18, and HPV31. The LODs were assessed using (i) plasmids of HPV31, HPV16, and HPV18 in the background of pooled HPV-negative patient specimens collected in PreservCyt solution and (ii) HPV-positive cell lines SiHa (HPV16) and HeLa (HPV18) in PreservCyt solution containing an HPV-negative cell line (HCT-15) background. Plasmids and cell lines were diluted to concentrations lower than, higher than, and at the expected LOD levels. A minimum of 60 replicates was tested for each plasmid or cell line level for each of 3 reagent lots. A total of 30 runs were performed in a period of 5 days using 4 different instrument systems.
To verify that the cobas HPV test was capable of accurately detecting each of the remaining 11 hrHPV genotypes, the LOD at the clinical cutoff was also determined for genotypes HPV33, -35, -39, -45, -51, -52, -56, -58, -59, -66, and -68. Quantified plasmid stocks of each HPV genotype were diluted into a background of pooled HPV-negative patient specimens collected in PreservCyt medium to concentrations lower than, higher than, and at the expected LOD levels. Two lots of reagents were used to produce a minimum of 24 replicates for each positive level with each lot of reagents. For each HPV type, the reported LOD was defined as the lowest testing concentration having a positive hit rate of >95%.
Reproducibility study.
Reproducibility of the cobas HPV test for the qualitative detection of hrHPV was evaluated across lots, sites and instruments, operators, days, and runs at HPV DNA concentrations lower than and higher than the LOD of the test using an 18-member panel composed of pools from clinical samples collected into PreservCyt solution and from samples derived from SiHa and HeLa cell lines. Pooled clinical samples contained HPV16, HPV18, HPV31, or HPV45, and samples derived from SiHa and HeLa cell lines contained HPV16 and HPV18, respectively.
Each panel member was tested for 18 days (6 days per kit lot), in 2 replicates per run, at 3 testing sites. Two operators at each of 3 sites performed 2 runs per day for 3 days each on each of 3 reagent lots. A run was defined as 36 panel-member aliquots and 1 positive and 1 negative control.
Statistical analysis for the reproducibility study was performed as follows. All statistical analyses were performed using SAS/STAT software. The placement of panel members within a run was randomized using PROC PLAN. The number of valid and invalid tests from valid runs was calculated for lot, site and instrument, and operator and also separately by panel member. The data were summarized by the percentage of positive and negative results and the associated exact 95% CIs for agreement estimates for each panel member by lot, site and instrument, operator, day, and run. An analysis of variance was performed on the CT values for each panel member that was expected to be positive for an HPV genotype. Precision was evaluated by using a random-effects model with terms for lot, site or instrument, operator nested within site or instrument, day nested within lot, site or instrument, and operator, run nested within lot, site or instrument, operator, and day, and within-run as random effects by using PROC MIXED.
Results from the model fitted to the CT value are presented as the percentage of variance for each effect (e.g., lot) by panel member. The percentage of the variance for each effect was determined by dividing the variance of an individual effect (e.g., lot) by the total variance.
The total standard deviation (SD) and the total coefficient of variation (CV) (expressed as a percentage) were calculated. The mean and 95% CIs were also calculated using corrected degrees of freedom (Satterthwaite method).
Analytic specificity.
A panel of bacteria, fungi, and viruses, including those commonly found in the female urogenital tract, as well as several HPV types classified as low or of undetermined risk, were tested with the cobas HPV test to assess analytic specificity (17). Each microorganism or virus was spiked at high concentrations (≥1 × 103 units/reaction mixture) into an HPV-negative cervical specimen collected in PreservCyt solution and into an HPV-negative cervical specimen collected in PreservCyt solution spiked with HPV31, HPV16, and HPV18 plasmid DNA at 3 × LOD.
To evaluate potential interfering substances, HPV-positive and HPV-negative cervical specimens, as well as contrived specimens, were used to assess the effects of endogenous and exogenous substances that might be present in cervical specimens (17). The concentrations of endogenous and exogenous substances tested represent conditions that might occur during specimen collection.
RESULTS
Establishing the clinical cutoff values.
The clinical cutoff values required to achieve the desired clinical sensitivities in both the ASC-US and the overall population were selected in an initial training study (phase 1) and then validated in a subsequent study (phase 2). The cutoff values in each channel were chosen using an iterative algorithm to achieve a defined clinical sensitivity for ≥CIN2 in the phase 1 study (see Table S1 in the supplemental material). The cutoff values of CT 40, 40.5, and 40 for the 3 channels achieved sensitivities for detection of ≥CIN2 of 90% (215/239; 95% CI, 85.5 to 93.2%) in the overall population (aged ≥25 years) and of 93% (40/43; 95% CI, 81.4% to 97.6%) and 96.3% (26/27; 95% CI, 81.7% to 99.3%) for ≥CIN2 and ≥CIN3, respectively, in the ASC-US population (aged ≥21 years). CT values for sensitivity were optimized for specificity, resulting in a specificity of 58.4% (95% CI, 56.9% to 59.8%) for the detection of ≥CIN2 in the overall population and of 70.6% (587/831; 95% CI, 67.5% to 73.6%) and 69.5% (95% CI, 66.4% to 72.5%) for ≥CIN2 and ≥CIN3, respectively, in the ASC-US population.
The estimates of sensitivity and specificity in the 2 phases of the study were not statistically different; therefore, the data from phase 1 and phase 2 were combined (Table 2). As expected, both the sensitivity and specificity were higher in the ASC-US population than in the overall population.
Table 2.
Estimated sensitivity and specificity of the cobas HPV test for ≥CIN2 (combined phase 1 and phase 2)a
| Population | Sensitivityb |
Specificityb |
||
|---|---|---|---|---|
| % (no./total no.) | 95% CI | % (no./total no.) | 95% CI | |
| Overall (aged ≥25 years) | 88.2 (380/431) | 84.8–90.9 | 57.8 (4270/7392) | 56.6–58.9 |
| ASC-USc (aged ≥21 years) | 90 (72/80) | 81.5–94.8 | 70.5 (1056/1498) | 68.1–72.7 |
≥CIN2, cervical intraepithelial neoplasia grade 2 or higher; HPV, human papillomavirus.
Estimates of sensitivity and specificity are based on data from subjects with a valid histologic diagnosis from CPR. CI, confidence interval.
ASC-US, atypical squamous cells of undetermined significance.
Concordance of the cobas HPV test with the combined comparator.
The analytic accuracy of the cobas HPV test was determined by comparison with the combined comparator of Sanger sequencing and hc2 in 3 populations, as defined in Materials and Methods.
The results of agreement analysis for these 3 populations for HPV16, HPV18, and the pool of 12 hrHPV types are shown in Table 1. For the overall population, positive agreement for HPV16 was 98.6% (95% CI, 95.0% to 99.6%) and negative agreement was 98.2% (95% CI, 97.4% to 98.7%). For HPV18 in the overall population, positive agreement was 91.9% (95% CI, 83.4% to 96.2%) and negative agreement was 99.4% (95% CI, 99.0% to 99.7%). For the 12 other pooled hrHPV types, the positive agreement in the overall population was 92.5% (95% CI, 89.9% to 94.4%) and the negative agreement was 88.4% (95% CI, 86.7% to 90.0%). For the combined comparator, the discordant results between hc2 and sequencing were deemed indeterminate and were not included in the analysis; the numbers of indeterminates for each population and each cobas HPV test result category are shown in Table S2 in the supplemental material.
Limit of detection (analytic sensitivity).
The LOD for the HPV31 plasmid was 600 copies/ml, and for the HPV16 and HPV18 plasmids, the LOD was 300 copies/ml. The LOD for HPV types HPV33, -35, -39, -45, -51, -52, -56, -58, -59, -66, and -68 ranged from 150 to 2,400 copies/ml (Tables 3 and 4). All results are from the reagent lot that produced the most conservative (highest) LOD in the analysis.
Table 3.
Limit-of-detection levels for some HPVa types and cell lines
| HPV type or cell line and variables | No. positive/no. tested | Mean CTb | Positive (%) | 95% CI (%) |
|---|---|---|---|---|
| HPV31 (copies/ml) | ||||
| 600 | 60/60 | 36.6 | 100 | 94.0–100 |
| 300 | 59/61 | 37.9 | 96.7 | 88.7–99.6 |
| 150 | 49/60 | 38.7 | 81.7 | 69.6–90.5 |
| HPV16 (copies/ml) | ||||
| 1,500 | 60/60 | 36.5 | 100 | 94.0–100 |
| 600 | 60/60 | 37.7 | 100 | 94.0–100 |
| 300 | 55/61 | 39.1 | 90.2 | 79.8–96.3 |
| HPV18 (copies/ml) | ||||
| 1,500 | 60/60 | 36.9 | 100 | 94.0–100 |
| 600 | 60/60 | 38.0 | 100 | 94.0–100 |
| 300 | 42/61 | 39.6 | 68.9 | 55.7–80.1 |
| SiHa (HPV16) (cells/ml) | ||||
| 200 | 60/60 | 36.9 | 100 | 94.6–100 |
| 100 | 60/60 | 38.0 | 100 | 94.6–100 |
| 50 | 53/60 | 39.3 | 88.3 | 77.4–95.2 |
| HeLa (HPV18) (cells/ml) | ||||
| 80 | 60/60 | 35.7 | 100 | 94.0–100 |
| 40 | 60/60 | 36.8 | 100 | 94.0–100 |
| 20 | 56/60 | 38.2 | 93.3 | 83.8–98.1 |
HPV, human papillomavirus.
CT, cycle threshold.
Table 4.
Summary of high-risk genotype limit of detection
| HPV DNA type | LODa (copies/ml) | No. positive/no. tested | Mean CT | Hit rate | 95% CI (%) |
|---|---|---|---|---|---|
| HPV33 | 300 | 24/24 | 38.2 | 100 | 85.7–100 |
| HPV35 | 600 | 23/24 | 38.4 | 95.8 | 78.8–99.8 |
| HPV39 | 300 | 24/24 | 37.9 | 100 | 85.7–100 |
| HPV45 | 150 | 23/24 | 38.0 | 95.8 | 78.8–99.8 |
| HPV51 | 300 | 24/24 | 38.4 | 100 | 85.7–100 |
| HPV52 | 2,400 | 24/24 | 39.1 | 100 | 85.7–100 |
| HPV56 | 1,200 | 23/24 | 38.4 | 95.8 | 78.8–99.8 |
| HPV58 | 600 | 24/24 | 38.6 | 100 | 85.7–100 |
| HPV59 | 300 | 23/24 | 39.0 | 95.8 | 78.8–99.8 |
| HPV66 | 1,200 | 24/24 | 37.7 | 100 | 85.7–100 |
| HPV68 | 1,200 | 24/24 | 38.0 | 100 | 85.7–100 |
LOD, limit of detection.
Reproducibility.
Overall, 111 runs were performed to obtain 108 valid runs. The 2.7% invalid runs (3/111) were due to instrument errors. A total of 3,888 tests were performed on the 18 panel members in the valid runs; 5 of these tests were invalid because of instrument errors.
All valid test results are included in the analyses that reported the percentage of correct results. There were no false-positive results in 216 tests performed on the negative panel members (background negative cell and the pooled negative clinical samples) (see Table S3 in the supplemental material).
The percentages of positive results for the positive panel members are presented in Table S4 in the supplemental material. An analysis of variance of the CT values from valid tests performed on positive panel members yielded overall CVs ranging from 1.1% to 2.5% for the SiHa cell lines, 1.5% to 2.5% for the HeLa cell lines, and 3.5% to 10.3% for the pooled clinical samples.
The overall reproducibility of the cobas HPV test was very high for all positive panel members (i.e., those with HPV DNA concentrations near or higher than the LOD), with 98% to 100% of the test results being positive. For the moderate-positive samples, results were positive for 89% to 96% of tests. For the weak-positive samples, 41% to 56% of the results were positive.
Reproducibility across lots and days was good for both the cell line-derived (including weak-positive panel members) and the pooled clinical samples. The results tended to be less reproducible across sites or instruments for samples with concentrations lower than the LOD, with fewer moderate- and weak-positive samples detected at site 1 than at sites 2 and 3. This trend was largely due to the results from operator 1 at site 1 and was evident for the cell line-derived samples, which contained HPV DNA concentrations lower than the LOD. The differences in hit rates were less apparent among operators for the pooled clinical samples and cell line-derived samples containing HPV DNA concentrations higher than the LOD.
The cobas HPV test was evaluated using 2 sources of HPV DNA: cell lines and pooled clinical samples, including 4 HPV genotypes (HPV16, -18, -31, and -45). The test was highly reproducible across lots, sites and instruments, and days, and the analytic specificity of the test was very high. Reproducibility was high across operators for samples with HPV DNA concentrations higher than the LOD (see Table S4 in the supplemental material).
Potential interfering organisms and substances.
Results for the panel of bacteria, fungi, and viruses (including the low-risk HPV genotypes) indicated that none of these organisms interfered with the detection of HPV31, HPV16, and HPV18 plasmid DNA or produced a false-positive result in the HPV-negative specimens.
Whole blood, peripheral blood mononuclear cells (PBMC), and cervical mucus were also tested as potential endogenous interfering substances found in cervical specimens. No interference was seen for PBMC or cervical mucus at all levels tested. Whole blood showed no interference when present in visually detectable concentrations of up to 2%. At 3%, however, false-negative results were found with some samples, presumably because of PCR inhibition.
DISCUSSION
The sensitivity of PCR assays in detecting very low concentrations of analytes has led to diagnostic advances in the fields of infectious disease, cancer, genetics, and forensics. When PCR technology is used in screening tests, however, there remains some concern that the sensitivity achieved exceeds the clinical relevance of the results. The cobas HPV test is a 4-channel real-time TaqMan PCR test for 14 hrHPV genotypes that was developed to maximize the probability that a positive result is also clinically significant. To achieve this objective, the cutoff for the cobas HPV test was determined and validated in a large cervical cancer screening study using the detection of high-grade cervical disease rather than the number of viral copies as the endpoint. This approach increases the probability that a positive result is likely to indicate the presence of significant cervical disease as opposed to a transient low-level HPV infection. From a public health perspective, the cobas HPV test achieved the clinical sensitivity and specificity desired by setting the appropriate cutoff values (cycle thresholds [CT]) for the detection of HPV16, HPV18, and the remaining 12 hrHPV types. The cutoff values in each channel were chosen using an iterative algorithm to achieve a defined clinical sensitivity for ≥CIN2 in a phase 1 (training) study, i.e., 90% sensitivity in the overall (aged ≥25 years) population study and 93% in the ASC-US (aged ≥21 years) population.
Sequencing the 8-kb HPV genome has revealed 4 distinct clades of HPV (alpha, beta, gamma, and delta), with the viruses that are associated with cervical cancer falling into the alpha clade (18). A distinct HPV genotype is defined as a sequence containing the L1 gene that differs by ≥10% from the L1 sequence of other HPV types. Of the >100 different HPV genotypes, about 40 commonly infect the anogenital tract; 13 of these have been classified as oncogenic based on epidemiologic and in vitro evidence (19), and one (HPV66) is now classified as showing “limited evidence” for oncogenicity (20). Of these 14 hrHPV genotypes, 2 (HPV16 and HPV18) are considered the highest risk genotypes and cause approximately 70% of all invasive cervical cancer cases worldwide (21, 22).
The L1 gene encodes the major capsid protein and is the most conserved sequence among those in the papillomaviruses (18), although there are regions where the sequence varies among different HPV types. Because HPV genotypes have historically been defined by sequence variation in the L1 gene, targeted amplification of the L1 gene allows for differentiation of the genotypes. Moreover, seminal epidemiologic studies using primers placed in the conserved regions of the L1 gene have demonstrated the correlation between HPV status and the development of cervical cancer (23). The fact that the vast majority of molecular epidemiology literature on HPV genotypes and cervical cancer is based on analyses of the L1 gene (23, 24) further validates the importance and uniqueness of the L1 region.
Given the clinical importance of the separate identification of HPV16 and HPV18 (5, 6) and the demonstrated ability of a variety of L1 PCR assays to sensitively detect HPV in cancers and precancers (25–27), the cobas HPV test was developed using primers and type-specific probes that target the L1 region and amplify a ∼200-bp fragment of the L1 gene, thereby allowing for the individual identification of HPV16 and HPV18. In addition, targeting the variable L1 sequences also increases the analytic specificity of the cobas HPV test by avoiding cross-hybridization to the genetically related sequences common to the low-risk genotypes. Moreover, the cobas HPV test is unique in its ability to detect HPV18 separately from the closely related, but lower-risk, HPV45 (8, 28), which is included in the pooled 12 hrHPV channel.
The ability of the cobas HPV test to separately detect the 2 most-oncogenic HPV types allows for the identification of women at the highest risk of high-grade cervical disease and who should be referred for immediate colposcopy. Results from the ATHENA trial demonstrated that the risk of ≥CIN2 in women with normal cytology but who are positive for HPV16 or HPV16/18 (i.e., HPV16 and/or HPV18) is sufficiently high to warrant referral to colposcopy (6), and tests that simply detect hrHPV types as an undifferentiated pool cannot provide this clinically meaningful information.
The critical performance parameters for an HPV test in cervical cancer screening are clinical, rather than analytic, sensitivity and specificity. The assay needs to be robust, accurate, and reproducible in different laboratories. We demonstrated, with data from the technical validation studies presented here, that the cobas HPV test also meets the analytic standards for LOD, accuracy, and reproducibility. The analytic sensitivity (LOD) was determined by using a reproducibility panel of cobas HPV test results obtained with HPV genotype-specific plasmids, as well as with 2 cell lines: HeLa (HPV18) and SiHa (HPV16). The LODs were 40 and 100 cells/ml for the HeLa (HPV18) and SiHa (HPV16) cell lines, respectively, whereas the LOD was 600 copies/ml for the HPV18 and HPV16 plasmids. The LOD for the HPV31 plasmid was 300 copies/ml. For the other 11 hrHPV plasmids, the LOD ranged from 150 copies/ml for HPV45 to 2,400 copies/ml for HPV52. This range of sensitivity might reflect the different abilities of the probes in the PCR TaqMan assay to detect the different hrHPV types in the pool.
The analytic accuracy, determined by comparing the cobas HPV test result with the result of the combined comparator of Sanger sequencing and Qiagen hc2, was evaluated for each of the 3 HPV channels and showed excellent agreement in all 3 channels in 3 different populations: ASC-US (aged ≥21 years), NILM (aged ≥30 years), and overall (aged ≥25 years). Of note, the percent agreement was slightly lower for the channel detecting the 12 remaining (pooled) hrHPV types, and the difference between the percent agreement for the ASC-US and NILM samples presumably reflected a lower viral load of HPV in the NILM samples that might have been near the threshold of detection for both the primers in the cobas HPV test and for the PGMY primers used in the initial phase of the Sanger sequencing protocol. Likewise, the differences seen in the percent agreements among the 3 genotype categories reflect the various CT cutoffs that were chosen to maximize the clinical sensitivity for the detection of high-grade disease and to optimize clinical specificity.
Only those samples that were positive by both Sanger sequencing and hc2 were defined as positive in the composite comparison with the cobas HPV test results. The samples that were discordant between Sanger sequencing and the hc2 assay were considered indeterminate and were not included in the comparison. One important difference to note between the cobas HPV test and the Sanger sequencing protocol was that the cobas HPV test involved a pool of primers and probes, whereas the samples used in the Sanger protocol were initially amplified with the PGMY primers, and those samples that were positive were then reextracted and amplified with a single HPV genotype-specific primer set.
In conclusion, the cobas HPV test was developed with a clinically determined and validated cutoff that maximizes the detection of ≥CIN2. In addition, the ability of the assay to distinguish HPV16 and HPV18 from the other hrHPV types, and thereby identify those women at highest risk for cervical disease, adds clinical value.
Supplementary Material
ACKNOWLEDGMENTS
This work was supported by Roche Molecular Systems, Pleasanton, CA.
Arundhati Rao and Stephen Young participate on the molecular advisory board for Roche. Stephen Young is on the advisory board for Quidel and has received honoraria for his participation. Henry Erlich, Sean Boyle, Mark Krevolin, Rita Sun, Raymond Apple, and Catherine Behrens are employees of Roche Molecular Systems.
Footnotes
Published ahead of print 27 February 2013
Supplemental material for this article may be found at http://dx.doi.org/10.1128/JCM.03386-12.
REFERENCES
- 1. Kitchener HC, Castle PE, Cox JT. 2006. Chapter 7: achievements and limitations of cervical cytology screening. Vaccine 24 (Suppl 3):S3/63–S3/70 doi:10.1016/j.vaccine.2006.05.113 [DOI] [PubMed] [Google Scholar]
- 2. Bulkmans NW, Berkhof J, Rozendaal L, van Kemenade FJ, Boeke AJ, Bulk S, Voorhorst FJ, Verheijen RH, van Groningen K, Boon ME, Ruitinga W, van Ballegooijen M, Snijders PJ, Meijer CJ. 2007. Human papillomavirus DNA testing for the detection of cervical intraepithelial neoplasia grade 3 and cancer: 5-year follow-up of a randomised controlled implementation trial. Lancet 370:1764–1772 [DOI] [PubMed] [Google Scholar]
- 3. Naucler P, Ryd W, Törnberg S, Strand A, Wadell G, Elfgren K, Rádberg T, Strander B, Johansson B, Forslund O, Hansson BG, Rylander E, Dillner J. 2007. Human papillomavirus and Papanicolaou tests to screen for cervical cancer. N. Engl. J. Med. 357:1589–1597 [DOI] [PubMed] [Google Scholar]
- 4. Castle PE, Stoler MH, Wright TC, Jr, Sharma A, Wright TL, Behrens CM. 2011. Performance of carcinogenic human papillomavirus (HPV) testing and HPV16 or HPV18 genotyping for cervical cancer screening of women aged 25 years and older: a subanalysis of the ATHENA study. Lancet Oncol. 12:880–890 [DOI] [PubMed] [Google Scholar]
- 5. Khan MJ, Castle PE, Lorincz AT, Wacholder S, Sherman M, Scott DR, Rush BB, Glass AG, Schiffman M. 2005. The elevated 10-year risk of cervical precancer and cancer in women with human papillomavirus (HPV) type 16 or 18 and the possible utility of type-specific HPV testing in clinical practice. J. Natl. Cancer Inst. 97:1072–1079 [DOI] [PubMed] [Google Scholar]
- 6. Wright TC, Jr, Stoler MH, Sharma A, Zhang G, Behrens C, Wright TL, ATHENA (Addressing THE Need for Advanced HPV Diagnostics) Study Group 2011. Evaluation of HPV-16 and HPV-18 genotyping for the triage of women with high-risk HPV+ cytology-negative results. Am. J. Clin. Pathol. 136:578–586 [DOI] [PubMed] [Google Scholar]
- 7. Stoler MH, Wright TC, Jr, Sharma A, Apple R, Gutekunst K, Wright TL, ATHENA (Addressing THE Need for Advanced HPV Diagnostics) HPV Study Group 2011. High-risk human papillomavirus testing in women with ASC-US cytology: results from the ATHENA HPV study. Am. J. Clin. Pathol. 135:468–475 [DOI] [PubMed] [Google Scholar]
- 8. Kjaer SK, Frederiksen K, Munk C, Iftner T. 2010. Long-term absolute risk of cervical intraepithelial neoplasia grade 3 or worse following human papillomavirus infection: role of persistence. J. Natl. Cancer Inst. 102:1478–1488 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Rao A, Young S, Krevolin M, Sun R, Purner MB, Sharma A, Behrens CM. 2012. Comparison of cobas human papillomavirus test results from primary versus secondary vials of PreservCyt specimens and evaluation of potential cross-contamination. Cancer Cytopathol. 120:380–389 [DOI] [PubMed] [Google Scholar]
- 10. Roche Molecular Diagnostics 2009. cobas 4800 system manual 1.0. Roche Molecular Diagnostics, Pleasanton, CA [Google Scholar]
- 11. Holland PM, Abramson RD, Watson R, Gelfand DH. 1991. Detection of specific polymerase chain reaction product by utilizing the 5′----3′ exonuclease activity of Thermus aquaticus DNA polymerase. Proc. Natl. Acad. Sci. U. S. A. 88:7276–7280 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Meijer CJ, Berkhof J, Castle PE, Hesselink AT, Franco EL, Ronco G, Arbyn M, Bosch FX, Cuzick J, Dillner J, Heideman DA, Snijders PJ. 2009. Guidelines for human papillomavirus DNA test requirements for primary cervical cancer screening in women 30 years and older. Int. J. Cancer 124:516–520 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Digene Corporation 2012. Hybrid Capture 2 High-Risk HPV DNA Test package insert. Digene Corporation, Gaithersburg, MD: http://www.thehpvtest.com/∼/media/5C4BD0982BED4E3788F65B36AF829AAD.ashx [Google Scholar]
- 14. Wright TC, Jr, Stoler MH, Behrens CM, Apple R, Derion T, Wright TL. 2012. The ATHENA human papillomavirus study: design, methods, and baseline results. Am. J. Obstet. Gynecol. 206:46.e1–46.e11 doi:10.1016/j.ajog.2011.07.024 [DOI] [PubMed] [Google Scholar]
- 15. Kondratovich MV, Yousef WA. 2005. Evaluation of accuracy and optimal cutoff of diagnostic devices in the same study, p 2547–2551 In ASA section on statistics in epidemiology. Proceedings of the 2005 Joint Statistical Meeting, Biopharmaceutical Section, Minneapolis, MN [Google Scholar]
- 16. Gravitt PE, Peyton CL, Alessi TQ, Wheeler CM, Coutlee F, Hildesheim A, Schiffman MH, Scott DR, Apple RJ. 2000. Improved amplification of genital human papillomaviruses. J. Clin. Microbiol. 38:357–361 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Roche Molecular Systems, Inc 2011. cobas HPV Test package insert. Roche Molecular Systems, Inc., Pleasanton, CA: http://www.accessdata.fda.gov/cdrh_docs/pdf10/P100020c.pdf [Google Scholar]
- 18. de Villiers EM, Fauquet C, Broker TR, Bernard HU, zur Hausen H. 2004. Classification of papillomaviruses. Virology 324:17–27 [DOI] [PubMed] [Google Scholar]
- 19. Muñoz N, Bosch FX, de Sanjosé S, Herrero R, Castellsagué X, Shah KV, Snijders PJ, Meijer CJ, International Agency for Research on Cancer Multicenter Cervical Cancer Study Group 2003. Epidemiologic classification of human papillomavirus types associated with cervical cancer. N. Engl. J. Med. 348:518–527 [DOI] [PubMed] [Google Scholar]
- 20. Bouvard V, Baan R, Straif K, Grosse Y, Secretan B, El Ghissassi F, Benbrahim-Tallaa L, Guha N, Freeman C, Galichet L, Cogliano V, WHO International Agency for Research on Cancer Monograph Working Group 2009. A review of human carcinogens—part B: biological agents. Lancet Oncol. 10:321–322 [DOI] [PubMed] [Google Scholar]
- 21. de Sanjose S, Quint WG, Alemany L, Geraets DT, Klaustermeier JE, Lloveras B, Tous S, Felix A, Bravo LE, Shin HR, Vallejos CS, de Ruiz PA, Lima MA, Guimera N, Clavero O, Alejo M, Llombart-Bosch A, Cheng-Yang C, Tatti SA, Kasamatsu E, Iljazovic E, Odida M, Prado R, Seoud M, Grce M, Usubutun A, Jain A, Suarez GA, Lombardi LE, Banjo A, Menendez C, Domingo EJ, Velasco J, Nessa A, Chichareon SC, Qiao YL, Lerma E, Garland SM, Sasagawa T, Ferrera A, Hammouda D, Mariani L, Pelayo A, Steiner I, Oliva E, Meijer CJ, Al-Jassar WF, Cruz E, Wright TC, Puras A, et al. 2010. Human papillomavirus genotype attribution in invasive cervical cancer: a retrospective cross-sectional worldwide study. Lancet Oncol. 11:1048–1056 [DOI] [PubMed] [Google Scholar]
- 22. Munñoz N, Bosch FX, Castellsagué X, Díaz M, de Sanjose S, Hammouda D, Shah KV, Meijer CJ. 2004. Against which human papillomavirus types shall we vaccinate and screen? The international perspective. Int. J. Cancer 111:278–285 [DOI] [PubMed] [Google Scholar]
- 23. Clifford GM, Smith JS, Plummer M, Muñoz N, Franceschi S. 2003. Human papillomavirus types in invasive cervical cancer worldwide: a meta-analysis. Br. J. Cancer. 88:63–73 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Smith JS, Lindsay L, Hoots B, Keys J, Franceschi S, Winer R, Clifford GM. 2007. Human papillomavirus type distribution in invasive cervical cancer and high-grade cervical lesions: a meta-analysis update. Int. J. Cancer 121:621–632 [DOI] [PubMed] [Google Scholar]
- 25. Naucler P, Ryd W, Törnberg S, Strand A, Wadell G, Elfgren K, Radberg T, Strander B, Forslund O, Hansson BG, Hagmar B, Johansson B, Rylander E, Dillner J. 2009. Efficacy of HPV DNA testing with cytology triage and/or repeat HPV DNA testing in primary cervical cancer screening. J. Natl. Cancer Inst. 101:88–99 [DOI] [PubMed] [Google Scholar]
- 26. Hesselink AT, van Ham MA, Heideman DA, Groothuismink ZM, Rozendaal L, Berkhof J, van Kemenade FJ, Massuger LA, Melchers WJ, Meijer CJ, Snijders PJ. 2008. Comparison of GP5+/6+-PCR and SPF10-line blot assays for detection of high-risk human papillomavirus in samples from women with normal cytology results who develop grade 3 cervical intraepithelial neoplasia. J. Clin. Microbiol. 46:3215–3221 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Bulkmans NW, Rozendaal L, Snijders PJ, Voorhorst FJ, Boeke AJ, Zandwijken GR, van Kemenade FJ, Verheijen RH, Groningen K, Boon ME, Keuning HJ, van Ballegooijen M, van den Brule AJ, Meijer CJ. 2004. POBASCAM, a population-based randomized controlled trial for implementation of high-risk HPV testing in cervical screening: design, methods and baseline data of 44,102 women. Int. J. Cancer 110:94–101 [DOI] [PubMed] [Google Scholar]
- 28. Kitchener HC, Gilham C, Sargent A, Bailey A, Albrow R, Roberts C, Desai M, Mather J, Turner A, Moss S, Peto J. 2011. A comparison of HPV DNA testing and liquid based cytology over three rounds of primary cervical screening: extended follow up in the ARTISTIC trial. Eur. J. Cancer. 47:864–871 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.