Evaluation and Optimization of the Clinical Accuracy of Hybribio's 14 High-Risk HPV with 16/18 Genotyping Assay within the VALGENT-3 Framework

Hybribio’s 14 High-Risk HPV with 16/18 genotyping real-time PCR (HBRT-H14) is a human papillomavirus (HPV) assay with approval from the China Food and Drug Administration that is widely used in China. VALGENT (VALidation of HPV GENotyping Tests) is an established framework for evaluating HPV tests’ clinical performance relative to validated comparators. The aim of this study was to assess the clinical accuracy of HBRT-H14 following international validation criteria. Within VALGENT-3, clinical performance of HBRT-H14 was compared with Hybrid Capture 2 (HC2), Linear Array HPV genotyping test (Linear Array), and Cobas 4800 HPV test (Cobas).

ABSTRACT

Hybribio’s 14 High-Risk HPV with 16/18 genotyping real-time PCR (HBRT-H14) is a human papillomavirus (HPV) assay with approval from the China Food and Drug Administration that is widely used in China. VALGENT (VALidation of HPV GENotyping Tests) is an established framework for evaluating HPV tests’ clinical performance relative to validated comparators. The aim of this study was to assess the clinical accuracy of HBRT-H14 following international validation criteria. Within VALGENT-3, clinical performance of HBRT-H14 was compared with Hybrid Capture 2 (HC2), Linear Array HPV genotyping test (Linear Array), and Cobas 4800 HPV test (Cobas). VALGENT-3 comprised 1,300 consecutive samples and 300 abnormal cytological samples from the Slovenian cervical cancer screening program. Disease was defined as histologically confirmed cervical intraepithelial neoplasia scoring grade 2 or worse (CIN2+) and CIN3+, and two negative cytology results in a row were a proxy for nondisease. In the total study population, relative sensitivity and specificity of HBRT-H14 versus HC2 for detecting CIN2+ were 0.98 (95% confidence interval [CI], 0.94 to 1.03; P noninferiority[P_ni] < 0.01) and 0.97 (95% CI, 0.96 to 0.99; P_ni = 0.78), respectively. Applying an optimized a posteriori cutoff, defined using Linear Array and Cobas as bridging tests, yielded relative values of 0.98 (95% CI, 0.94 to 1.03; P_ni < 0.01) and 1.01 (95% CI, 1.00 to 1.03; P_ni < 0.01), respectively. In conclusion, HBRT-H14 was as sensitive but less specific than HC2 for detecting cervical precancer at the predefined cutoff. However, HBRT-H14 fulfilled international accuracy criteria for cervical cancer screening when using an optimized cutoff and might be attractive in low-resource settings given its low cost.

INTRODUCTION

Cervical cancer screening with human papillomavirus (HPV) testing offers better protection against invasive cervical carcinoma than cytology and longer intervals between multiple screening rounds (1, 2). Thus, HPV testing is gradually being implemented in organized screening programs for cervical cancer worldwide with the condition that only clinically validated tests for HPV should be used. Commercial HPV assays are abundant, but few have undergone clinical validation and fulfill the international consensus criteria to be used in settings for primary cervical cancer screening (3–6). Thus, the VALidation of HPV GENotyping Tests (VALGENT) framework was launched, seeking comprehensive comparison and clinical validation of HPV assays with genotyping capacity (7). VALGENT has been through several iterative installments using samples from women participating in screening for cervical cancer from Belgium (VALGENT-1) (8–10), Scotland (VALGENT-2) (11–14), Slovenia (VALGENT-3) (15–20), and Denmark (VALGENT-4) (21, 22).

This study first evaluated the clinical accuracy of Hybribio’s 14 High-Risk HPV with 16/18 genotyping real-time PCR assay (HBRT-H14; Hybribio, Chaozhou, China) within the VALGENT-3 panel relative to the standard comparator test, the Hybrid Capture 2 HPV DNA assay (HC2; Qiagen, Gaithersburg, MD, USA) (5–7) using the manufacturer-defined cutoff. Then, in case insufficient accuracy was observed, a cutoff optimization assessment was set up, as performed in previous VALGENT studies (19, 20). Since HBRT H14 separately detects HPV16, HPV18, and 12 other high-risk HPV (hrHPV) types in aggregate, we used two HPV assays as bridging tests providing the same or full genotyping results: the Linear Array HPV genotyping test (Linear Array; Roche Molecular Diagnostics, Branchburg, NJ, USA) (16) and the clinically validated FDA-approved assay with similar genotyping capacity, the Cobas 4800 HPV test (Cobas; Roche Molecular Systems, Alameda, CA, USA) (18).

MATERIALS AND METHODS

VALGENT-3 panel study population.

Altogether, 1,600 samples were collated in Slovenia for the VALGENT-3 panel as previously described (15–19). From the study of HPV prevalence in Slovenia (23), 1,300 consecutive samples were obtained from women age 25 to 64 participating in the national screening program for cervical cancer (screening population). Three hundred samples came from women referred for colposcopy due to abnormal cytology per the VALGENT protocol (7), in particular, atypical squamous cervical cells of undetermined significance (ASC-US) (100 women), low-grade squamous intraepithelial lesions (LSILs) (100 women), and high-grade squamous intraepithelial lesions (HSILs) (100 women, the enrichment population). Two cervical specimens came from each woman: one for conventional cytology and another for HPV DNA testing, placed in ThinPrep medium (ThinPrep PreservCyt solution; Hologic, Marlborough, MA, USA). ThinPrep samples were divided into several aliquots and stored at −70°C for testing with an HPV assay in the VALGENT-3 framework.

The Slovenian Medical Ethics Committee provided ethical approval (consent no. 83/11/09 and 109/08/12).

HPV testing.

All the samples were tested with HBRT-H14, HC2, Linear Array, and Cobas. All the assays were carried out following the manufacturer’s instructions at the Institute of Microbiology and Immunology at the University of Ljubljana’s Faculty of Medicine. DNA for HBRT-H14 and Linear Array testing was extracted from original samples collected in ThinPrep using QIAamp MinElute media kit (Qiagen) following the manufacturer’s instructions.

HBRT-H14 is a TaqMan-based real-time PCR assay targeting the E6 and E7 regions of 14 hrHPV types, allowing concurrent separate genotyping for HPV16 and HPV18 from the 12 other hrHPV types (HPV31, -33, -35, -39, -45, -51, -52, -56, -58, -59, -66, and -68). Signals are detected on four channels with spectrally unique dyes, yielding data for HPV16, HPV18, 12 other hrHPV types, and the human β-globin gene (24). Per the manufacturer’s instructions, samples were deemed HPV positive if the cycle threshold (C_T) was ≤ 40 for all channels. The samples were tested with HBRT-H14 between March 2017 and May 2018.

HC2 is a semiquantitative U.S. FDA-approved test allowing detection of 13 hrHPV types (HPV16, -18, -31, -33, -35, -39, -45, -51, -52, -56, -58, -59, and -68) used as the standard comparator test to assess the clinical performance of HBRT-H14 based on its clinical validity proven in large randomized trials that have a longitudinal follow-up (2, 6). As described previously (15–17), HC2 testing was performed within 2 weeks after obtaining the sample; thus, the samples were tested in 2010 for the screening population and in 2015 for the enrichment population.

Linear Array is a qualitative HPV assay with full genotyping capability. It targets the L1 region of 37 high-risk and low-risk types of HPV (HPV6, -11, -16, -18, -26, -31, -33, -35, -39, -40, -42, -44, -45, -51 to -54, -56, -58, -59, -61, -62, -64, -66 to -73, -81, -82 to -84, -89, and IS39) and the β-globin gene as an internal control (16). Linear Array was performed in 2016. This study defined hrHPV positivity as the presence of at least one of the 14 hrHPV types that was also detected by HBRT-H14.

Cobas is a fully automated multiplex real-time PCR assay targeting the L1 region of 14 hrHPV types allowing partial genotyping of HPV16 and HPV18 as well as pooled detection of 12 other hrHPV types (HPV31, -33, -35, -39, -45, -51, -52, -56, -58, -59, -66, and -68). Internal control of the sample cellularity was carried out on all samples by amplifying the human β-globin gene. Cobas testing was performed in 2015.

Clinical outcomes and statistical analysis.

Following the ZORA (the Slovenian National Cervical Cancer Screening Program) criteria, women with atypical squamous cells for which high-grade lesions (ASC-H) or worse could not be excluded were referred for colposcopy, and women with LSIL or ASC-US were invited for a repeat Pap smear and, with repeated ASC-US or worse, referred for colposcopy. HPV16- and/or HPV18-positive women were directly referred for colposcopy without regard to cytological results. Colposcopy-directed punch biopsy specimens were taken from areas suspected for high-grade lesions, and pathologists blinded to the HPV results carried out histopathological assessment.

Women with cervical intraepithelial neoplasia scoring grade 2 or worse (CIN2+) and CIN grade 3 or worse (CIN3+) confirmed by histology were deemed diseased, and they were included as the denominator for the clinical sensitivity estimation. For clinical specificity estimates, we considered women controls (≤ CIN1) if they had two or more consecutive negative scores for cytology results for intraepithelial lesion or malignancy (NILM) upon enrollment and at the next screening after 12 to 48 months. We calculated clinical sensitivity scores for CIN2+ and CIN3+ as well as clinical specificity for ≤ CIN1 of HBRT-H14, HC2, Linear Array, and Cobas. We assessed clinical performance separately for the entire study population and for women over 30. The McNemar test (McN) was used to compare differences between matched proportions (25). A matched noninferior statistic (ni) with a 90% relative sensitivity threshold and 98% relative specificity threshold was used when comparing clinical performance of HBRT-H14 to HC2 (6, 26). The level of statistical significance for both statistics (P_McN and P_ni) was set at 0.05. All the analyses were carried out using STATA version 14 (StataCorp, College Station, TX, USA).

Moreover, we assessed possible cutoff adaptation by changing the C_T thresholds of the HBRT-H14 and performed cutoff optimization analyses separately for HPV16, HPV18, and other hrHPV channels using two bridging tests: Linear Array and Cobas. Because testing for 14 hrHPV types with Linear Array has demonstrated noninferior clinical accuracy to that of HC2 within VALGENT-3 (16) and its full genotyping capability, Linear Array was employed as a first bridging test and Cobas as a second one (18). The purpose of the HPV16 cutoff optimization was to lower the corresponding C_T to maximize the number of ≤ CIN1 cases with negative HBRT-H14 results while maintaining a maximum of CIN2+ cases that were HPV16+ for both HBRT-H14 and the bridging test. A similar algorithm was applied for HPV18 and other hrHPV types.

RESULTS

Because HC2 has no internal control, specimen validity for HC2 could not be assessed; however, all 1,600 samples had valid Linear Array test results. Six and nine samples had invalid results with HBRT-H14 and Cobas, respectively. Figure 1 presents a flowchart showing the process from the panel collation of samples and the HPV testing to the final endpoint ascertainment for diseased and nondiseased groups. For assessing clinical accuracy, six samples that tested invalid with HBRT-H14 were excluded from further analysis. Of 1,594 samples with valid HPV results in the entire study population, two consecutive normal cytology results were obtained from 1,211 women; they were considered controls and used as the denominator for computing clinical specificity. We used 127 women with CIN2+ and 82 with CIN3+ for the clinical sensitivity assessment.

FIG 1.

Flowchart for panel sample collation, baseline cytology, cytology in the next screening round, and final histology results. Women with histologically confirmed CIN2+ and CIN3+ were used as the denominator for sensitivity, and women with two consecutive negative cytology results (≤ CIN1) were used as the denominator to compute specificity. The next screening round was for the screening population, not the enrichment population.

Table S1 in the supplemental material summarizes HBRT-H14, HC2, Linear Array, and Cobas results of the VALGENT-3 panel. In the screening population, HPV positivity rates detected by HBRT-H14, HC2, Linear Array, and Cobas were 14.4%, 12.1%, 10.7%, and 10.8%, respectively, and 67.0%, 71.3%, 66.7% and 68.7% in the enrichment population, respectively.

Absolute clinical accuracy of HBRT-H14, HC2, Linear Array, and Cobas.

In the entire study population, 120/127 CIN2+ and 78/82 CIN3+ cases were positive with HBRT-H14, corresponding to an absolute CIN2+ and CIN3+ clinical sensitivity of 94.5% (95% confidence interval [CI], 89.0 to 97.8%) and 95.1 (95% CI, 88.0 to 98.7%), respectively (Table 1). Out of 1,211 women with ≤ CIN1, 1,065 tested HPV negative with HBRT-H14, corresponding to an absolute clinical specificity of 87.9% (95% CI, 86.0 to 89.7%). Similar results for women ≥ 30 years of age are presented in Table 1. Absolute sensitivity for detecting CIN2+ and CIN3+, as well as specificity for ≤ CIN1 of HC2, Linear Array and Cobas, are presented in Table 1, where results are reported for the total population and women age 30 and older.

TABLE 1.

Clinical sensitivity for detection of CIN2+ and CIN3+ and clinical specificity for ≤ CIN1 of HBRT-H14, HC2, and Cobas in the total study population and in women ≥30 years old

Study group and test	Clinical sensitivity results						Clinical specificity results
	CIN2+			CIN3+			≤ CIN1
	No. positive/total no.	% of total	95% CI (%)	No. positive/total no.	% of total	95% CI (%)	No. positive/total no.	% of total	95% CI (%)
Total study population
HBRT-H14	120/127	94.5	(89.0–97.8)	78/82	95.1	(88.0–98.7)	1,065/1,211	87.9	(86.0–89.7)
HC2	122/127	96.1	(91.1–98.7)	80/82	97.6	(91.5–99.7)	1,093/1,211	90.3	(89.9–91.9)
Linear Array	125/127	98.4	(94.4–99.8)	82/82	100.0	(95.6–100.0)	1,109/1,211	91.6	(88.4–93.1)
Cobasa	122/127	96.1	(91.1–98.7)	80/82	97.6	(91.5–99.7)	1,102/1,204	91.5	(89.8–93.0)
Women ≥30 yrs
HBRT-H14	93/98	94.9	(88.5–97.8)	63/66	95.5	(87.3–99.1)	917/1,009	90.9	(88.9–92.6)
HC2	94/98	95.9	(89.8–98.9)	64/66	97.0	(89.5–99.6)	937/1,009	92.9	(91.1–94.4)
Linear Array	97/98	99.0	(94.4–100.0)	66/66	100.0	(94.6–100.0)	952/1,009	94.4	(92.7–95.7)
Cobasa	95/98	96.9	(91.3–99.4)	64/66	97.0	(89.5–99.6)	943/1,009	94.1	(92.5–95.5)

^aSeven samples tested invalid by Cobas were excluded from the analysis.

Relative sensitivity and specificity of HBRT-H14 in comparison to HC2.

Table 2 presents the clinical performance of HBRT-H14 in comparison to HC2 with regard to detection of CIN2+, CIN3+, and ≤ CIN1 in the overall study population and in women age 30 and older. Within the entire study population, HBRT-H14 has a relative sensitivity of 0.98 (95% CI, 0.94 to 1.03) for CIN2+ and 0.98 (95% CI, 0.92 to 1.04) for CIN3+. The relative specificity of HBRT-H14 for ≤ CIN1 is 0.97 (95% CI, 0.96 to 0.99). The sensitivity of HBRT-H14 for CIN2+ was noninferior to HC2 (P_ni = 0.003), whereas the specificity of HBRT-H14 for ≤ CIN1 was inferior to HC2 (P_ni = 0.777 and relative specificity <1, with 95% CI excluding unity). Similar results were found when restricting the analysis to women age 30 and older.

TABLE 2.

HBRT-14 versus HC2^a

Cutoff, study group, and CIN grade	Relative sensitivity (95% CI)	Relative specificity (95% CI)	PMcN	Pni
Manufacturer-defined cutoff
Total study population
CIN2+	0.98 (0.94–1.03)		0.480	0.003
CIN3+	0.98 (0.92–1.04)		0.688	0.026
≤ CIN1b		0.97 (0.96–0.99)	0.001	0.777
Women ≥30 yrs
CIN2+	0.99 (0.94–1.05)		1.000	0.006
CIN3+	0.98 (0.92–1.05)		1.000	0.023
≤ CIN1b		0.98 (0.96–0.99)	0.006	0.568
Optimized cutoff
Total study population
CIN2+	0.98 (0.94–1.03)		0.727	0.003
CIN3+	0.98 (0.92–1.04)		0.688	0.026
≤ CIN1b		1.01 (1.00–1.03)	0.016	<0.001
Women ≥30 yrs
CIN2+	0.99 (0.94–1.05)		1.000	0.025
CIN3+	0.98 (0.92–1.05)		1.000	0.023
≤ CIN1b		1.02 (1.00–1.03)	0.005	<0.001

^aShown are the relative sensitivities for CIN2+ and CIN3+ and relative specificities for ≤ CIN1 of the HBRT-14 versus the HC2 in the total study population and in women ≥30 years old using the manufacturer-defined cutoff and using the optimized cutoff.

^bTwo consecutive negative cytology results.

Because the noninferiority criteria were not fulfilled for the clinical specificity of HBRT-H14 when using the manufacturer-defined cutoff, a straightforward cutoff optimization analysis was performed to identify clinically relevant cutoffs, which would lead to noninferior clinical specificity of HBRT-H14 while maintaining noninferior sensitivity for CIN2+ and CIN3+ in comparison to HC2. The boxplots in Fig. 2 show that a reduction of cutoffs of C_T for HPV16, HPV18, and other hrHPV types from the original values of 40 to 37, 35, and 35, respectively, results in satisfactory clinical accuracy. Similar cutoff reduction results were obtained when using Cobas as the bridging test (Fig. S1).

FIG 2.

Box plots showing the distribution of C_T values for HPV16 (A), HPV18 (B), and other hrHPV types (C) using HBRT-H14 (y axis) in women with ≤ CIN1, women with CIN2+ and Linear Array positive for the HPV type considered (e.g., CIN2+, h16_LA+,) and women with CIN2+ but Linear Array negative for the HPV type considered (e.g., CIN2+,h16_LA-). The predefined cutoff for each type was set at 40 (solid red lines). By comparing with LinearArray, we could optimize and set the C_T values to the dashed green lines for HPV16 to 37, for HPV18 to 35, and for other hrHPV types to 35, respectively (dashed green lines).

Table 2 also presents the relative clinical performance of HBRT-H14 in comparison to HC2 after cutoff optimization. HBRT-H14 had a relative sensitivity of 0.98 (95% CI, 0.94 to 1.03; P_ni = 0.003) for CIN2+ and 0.98 (95% CI, 0.92 to 1.04; P_ni = 0.026) for CIN3+. The relative specificity of HBRT-H14 for ≤ CIN1 was 1.01 (95% CI, 1.00 to 1.03; P_ni < 0.001). Restricting the analysis to women older than 30 yielded comparable results.

DISCUSSION

HBRT-H14 is a real-time PCR-based assay designed for HPV-based screening for primary cervical cancer. It is capable of simultaneous differentiation of HPV16 and HPV18 from the pool of 12 other hrHPV types. The China Food and Drug Administration approved HBRT-H14 in 2012, and it has been widely used in Chinese hospitals and laboratories (27). However, only a few data on clinical performance have been published in peer-reviewed literature (24, 27). In 2013, Bian et al. (24) evaluated the clinical accuracy of HBRT-H14 and HC2 on 424 SurePath samples (Becton, Dickinson and Company, Sparks, MD, USA) derived from a Chinese screening population. However, due to the nonmatched design of Bian et al.’s study, which randomly divided samples into two groups and tested them with either HBRT-H14 or HC2, noninferior statistics could not be calculated. Nevertheless, HBRT-H14 demonstrated good absolute sensitivity and specificity for detecting CIN2+ (96.3% and 78.2%, respectively), similar to that of HC2 (78.0% and 79.4%, respectively).

As far as we know, no study has assessed the clinical performance of HBRT-H14 according to the international validation criteria for HPV tests used for cervical cancer screening (6). This study evaluated the clinical performance of HBRT-H14 for detecting high-grade CIN within the VALGENT-3 framework for the first time. Compared to the standard HC2 comparator test, HBRT-H14 demonstrated noninferior sensitivity for detecting CIN2+ and CIN3+, but regardless of the population studied (the entire study population or women age 30 and older), it demonstrated inferior specificity for ≤ CIN1. Therefore, an optimization analysis was performed post hoc, and a new cutoff was identified, allowing a gain in specificity without loss of sensitivity. By using the post hoc-defined cutoff, HBRT-H14 met the noninferiority screening criteria that Meijer et al. defined in 2009 (6).

Our study used both Linear Array and Cobas as bridging tests in the cutoff optimization analysis and generated similar new cutoffs. This suggests that other validated assays with limited genotyping such as Cobas can also be used as bridging tests for clinical performance optimization of alternative HPV tests using genotyping capacity in the future when Linear Array is no longer commercially available. Namely, the manufacturer of Linear Array (Roche) announced the discontinuation of the production of Linear Array on 18 March 2019, and Linear Array has been no longer commercially available since 1 January 2020.

In a recent study, Xue et al. compared the performance of HBRT-H14 to that of Cobas for detecting CIN2+, using a total of 214 samples from a colposcopy referral population in China (27). HBRT-H14 had a clinical sensitivity of 94.6% (95% CI, 86.9 to 97.9%) and specificity of 66.0% (95% CI, 56.6 to 74.4%) and was considered noninferior to Cobas. Although the noninferior statistic was not reported in the original paper, we were able to calculate it a posteriori based on reported data. Interestingly, Xue et al. considered samples HPV positive if C_T values were ≤36 instead of ≤40 as predefined by the manufacturer. Although the rationale of using an alternative cutoff was not explained in Xue’s study, it is relatively close to the optimized cutoff generated and suggested in our study, and it reaffirms the necessity of cutoff optimization for HBRT-H14 clinical use in cervical cancer screening. For HPV tests that generate quantitative signal strengths and allow flexibility for use in various clinical settings (primary screening, triage of women with borderline cytology, or treatment follow-up), comprehensive analysis and further studies are needed to achieve the best clinical value.

Previously published VALGENT reports and cervical cytology biobank studies have shown that archived aliquoted ThinPrep liquid-based specimens stored at −70°C can be safely used several years after initial collection to evaluate the clinical performance of HPV DNA tests (16, 18, 28). Since the influence of prolonged specimen storage on the clinical performance of HBRT-H14 was not assessed in our study, this can be considered a study limitation; however, based on our experience with the evaluation of other HPV tests in the VALGENT-3 framework, we strongly believe that evaluation on fresh ThinPrep samples would have yielded accuracy values equal to those observed here. Another potential limitation of our study is that HBRT-H14 was evaluated in a single domesticated population of cervical samples. However, as presented in the VALGENT protocol paper (7), the absolute specificity of any HPV assay varies substantially between specimens taken from populations with different indications for HPV testing (e.g., primary screening, triage of women with borderline cytology, or treatment follow-up), but the relative specificity of HPV assays compared to the standard comparator assay was similar in different settings (e.g., the population of patients from different countries), indicating that the relative accuracy is a robust validation parameter for the assessment of diagnostic test accuracy.

HBRT-H14 has several advantages: it has high throughput and is a technically undemanding assay providing relatively quick results (2 to 3 h), and it can be operated on the great majority of real-time PCR platforms at low cost (27). Moreover, the analytical performance of HBRT-H14 was evaluated in the 2013 and 2014 WHO HPV LabNet international proficiency studies, showing high proficiency performance (100% proficient on all four data sets) (29).

To conclude, at the predefined cutoff of test positivity, HBRT-H14 was as sensitive as but less specific than HC2 for detecting CIN2+. However, with an a posteriori optimized cutoff, HBRT-H14 fulfilled the international criteria for use in screening for cervical cancer.