Effectiveness of a simple rapid HPV DNA test in rural Nigeria

Abstract

Success of the new HPV DNA test for low-resource settings (careHPV^™ test, QIAGEN Gaithersburg, Inc.; Gaithersburg, MD) requires good test performance when operated by personnel with limited laboratory experience. We evaluated the transferability, reliability, and accuracy of the careHPV test nested within a cervical screening project in a large Nigerian village. CareHPV testing was performed on screen-positive (n=345) and screen-negative (n=42) women attending colposcopy (68.3% of referred). Biopsies of abnormal-appearing areas were processed and read in the U.S. CareHPV specimens taken immediately before colposcopy were processed up to four times [in the field] by two secondary school graduates without laboratory experience, trained for this study. Specifically, QIAGEN Gaithersburg trained a laboratory-inexperienced U.S. researcher, who trained the first local technician who, in turn, trained the second. Residual specimens were sent to the U.S. for MY09/MY11 PCR testing for 13 carcinogenic genotypes (HPV16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 68) plus HPV66 (included in careHPV). Intra-rater agreement was 98.8% (Kappa=.97) and 98.9% (Kappa=.97) for technicians 1 and 2, respectively while inter-rater agreement was 96.3% (Kappa=.90). Agreement with MY09/MY11 PCR (virologic reference standard) was 89.3% (Kappa=.73) with 74.2% sensitivity and 95.7% specificity. The careHPV test detected 12 (80%) of 15 histologically-confirmed cervical intraepithelial neoplasia grade 2 (CIN2) or worse lesions, with an estimated 83.0% specificity to detect < CIN2. In a challenging low-resource setting with minimal intervention, the careHPV test performed adequately with high specificity but possibly lower sensitivity than HPV DNA tests currently used in controlled situations.

Introduction

In 2008, over 85% of an estimated 530,000 new cases of cervical cancer were diagnosed in less developed countries¹. Virtually all cases of cervical cancer worldwide are caused by persistent cervical infection with one or more of approximately a dozen carcinogenic genotypes of human papillomavirus (HPV)^2–4. Although effective prophylactic HPV vaccines have been developed, they have relatively high cost and require a three-dose regimen⁵. Moreover, they do not treat pre-existing infections and related disease. Fortunately, with adequate screening and treatment, the vast majority of cervical cancer can be prevented during the typical precancerous period lasting 10–15 years or more before invasion^6–7. Yet, the conventional screening model (Pap smear followed by colposcopically-directed biopsy) is neither sufficiently developed nor sustainable in most countries with limited resources⁸.

Screening for the presence of one or more carcinogenic HPV genotypes is appealing due to high sensitivity and negative predictive value as well as superior reliability compared to other screening methods^9–10. HPV testing is amenable to self-sampling outside the clinic which allows expanded population coverage¹¹. Also, whereas cytology has well-documented and logistical limitations, HPV DNA tests continue to improve and are now being developed for low-resource settings.

Successful HPV screening programs in less developed countries require affordable and simple HPV DNA tests that will perform well in remote settings. One such test is the careHPV^™ test (QIAGEN Gaithersburg Inc.; Gaithersburg, MD)^12–14. Widespread implementation to emerging global markets in less developed countries will likely require personnel without extensive laboratory background (i.e., training of a trainer) themselves to setup and train operators of the devices. We sought to assess the practicality of establishing and running the careHPV test in a rural setting using local personnel without laboratory experience and evaluate the reproducibility and accuracy of the careHPV test compared to an established HPV DNA PCR test and a histopathologic diagnosis.

Materials and Methods

Project Itoju is a population-based HPV prevalence study conducted in Irun, a large rural Nigerian village. The careHPV test was performed in the second phase for women referred to a diagnostic colposcopy examination. Details of the enrollment screening phase are described elsewhere¹⁵. Briefly, we consented and enrolled ~1500 women aged 15+ who attended an initial screening visit. At the screening visit locally-trained nurses conducted a cervical exam on non-virgins, which included visual inspection with acetic acid (VIA) and the collection of a cervical specimen used for liquid-based cytology (LBC) (prepared and read in the United States) and PCR-based HPV testing also done in the United States. Women with any positive screening result (VIA positive, non-normal LBC or PCR positive for carcinogenic or possibly carcinogenic HPV types) and a random selection of screen-negative controls (VIA negative, normal LBC, and PCR negative for carcinogenic or possibly carcinogenic HPV) were referred to colposcopy at a local clinic in Irun established for this study.

At the colposcopy visit, a specimen for careHPV was taken from all women using a digene^® cervical brush (QIAGEN Gaithersburg, Inc.; Gaithersburg, MD) and placed into 1 ml of careHPV Collection Medium (CCM). Subsequently, up to 4 acetowhite areas were biopsied. An ECC specimen was taken if the cervical transformation zone was not visualized by colposcopy and/or the woman was age 40 or older. Biopsy and ECC specimens were preserved in 10% formalin and later processed and read in the United States. CareHPV specimens were stored at ambient temperature (25–31°C) and tested four times within two weeks. Residual specimens were sent frozen to the United States and tested for the presence of any carcinogenic HPV using the same research PCR assay with which enrollment specimens were tested.

CareHPV training began at the QIAGEN Gaithersburg laboratory where a National Cancer Institute (NCI) researcher without laboratory background received training in the careHPV test according to a training protocol developed by Research and Development at QIAGEN Gaithersburg. The training protocol involved the following steps; 1) explanation and overview of careHPV technology, 2) demonstration of theoperation and maintainence of careHPV equipment, 3) one run by the trainer to demonsrate steps of the assay, 4) side by side runs by the trainee with trainer observation and assistance until the trainee felt comfortable, and 5) an evaluation run with a training panel of targets of known HPV DNA concentrations to demonstrate reproducibility and accuracy. The following month, the NCI scientist spent 3 days training a local technician in a simple office at the Obafemi Awolowo University in Ile-Ife, Nigeria. This first technician successfully passed the training and demonstrated competency by correctly running a training panel unaided 4 times. The first technician then spent 3 days independently training a second technician who successfully passed by correctly running the same training panel with the test unaided 5 times. Both technicians had secondary school training and neither had laboratory experience.

The first and second technicians each tested the same specimens in two separate batches. The simple careHPV machine will fail the batch rather than provide results if the internal control results are outside valid range. If a batch failed, all specimens in that batch were tested again and the results of the subsequent batch were considered in the analysis.

We note some exceptions to the standard, simple careHPV test protocol. Before the technicians moved the careHPV system to Irun, 14 specimens were processed at the office in Ile-Ife. CareHPV kits were stored in Ile-Ife, as opposed to the on-site Irun clinic. To avoid early expiration due to the long duration of the project, and lacking the capability for repeated importations, kits were sometimes stored at approximately 25°C. This is lower than the ambient temperature that approached 30°C. In several circumstances when the technicians were not available, careHPV could not be performed within 2 weeks after collection, requiring that the specimens be frozen at −20°C before processing (n=56). The technicians were not completely unsupervised; NCI researchers and QIAGEN Gaithersburg scientists conducted quality control remotely as electronic log files were periodically extracted and sent via Internet for review. Midway through the study, the careHPV test controller showed failing results for the first technician. A review of log files indicated poor performance (low positive calibrator mean to negative calibrator mean [S/N] ratios and high coefficients of variation). The technician admitted losing interest in the study and the second technician alone tested the remaining specimens. Finally, we note that careHPV test kits and CCM were not commercially available but were manufactured by Research and Development at QIAGEN Gaithersburg. In addition, the careHPV equipment system was a pilot model and not the final commercial system.

To measure test reliability, we calculated agreement and Kappa statistics within technicians and between technicians. To calculate agreement between the HPV PCR and the careHPV results, we considered the first technician’s first result or, if not available, the second technicians’ first result. For the comparisons of the careHPV test to PCR, we considered specimens PCR positive if one or more of the 13 carcinogenic types (16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 68) plus HPV66 (included in careHPV) were detected. Virologic sensitivity and specificity of the careHPV test were calculated using the HPV PCR results as a reference standard.

The careHPV test results were compared with a clinical reference standard of the worst histology from colposcopically-guided biopsy and excisional treatment. Sensitivity was calculated among women with a histology result from concurrent biopsy or excisional treatment of cervical intraepithelial neoplasia grade 2 (CIN2) or worse. Specificity to detect women without confirmed high grade disease was also calculated. Because the careHPV test was performed at the referral colposcopy visit among a subset of the screening population, our estimate of test specificity was adjusted to apply to all women attending the screening visit in the first phase of the study. Specifically, we extrapolated the age-stratified test performance of careHPV among randomly-selected screen-negative controls (women with negative screening results who were randomly selected to invite to colposcopy) to all remaining women with negative screening results in the cohort. Compliance with colposcopy visit among women with negative screening results was assumed to be the same as that observed among randomly-selected screen-negative women, after stratifying by age.

In sensitivity analyses, we considered the age-stratified performance of the careHPV test among women within the age range (30–45 years) considered for HPV-based screening programs. Older women are excluded because the treatment available in resource-poor settings, cryotherapy, is not effective in the older population. Analyses were performed using Stata 11.0 software (Stata Corp LP, College Station, TX).

The protocol was reviewed and approved by both the Nigerian and the NCI institutional review boards (NCT 00804466). Appropriate treatment among positive women with diagnosed cervical precancer and cancer is now underway, using a combination of cryotherapy, LEEP, and conization for HPV and CIN and more advanced treatments for invasive cancer.

Results

A total of 387 women attended the colposcopy visit (345 of 500 [69%] referred for any positive screening result and 42 of 67 [63%] randomly-selected screen-negative controls, p=0.3) (Figure 1). Mean and median times between enrollment and colposcopy visit were 10.7 and 11.2 months (range: 5.4–17.4). The first and second technicians processed 47.6% and 98.3% of specimens at least once, respectively. After initial difficulties, most runs of the careHPV test were successful. Specifically, during the first two months of running the careHPV test, we experienced complications with the reagents and a fire occurred in the storage room. Therefore, 10 of the 20 runs during this time were invalid. Subsequent to a new shipment of reagents, the first technician successfully ran 10 of 15 tests with the last four tests failing as an indicator of poor technician performance described earlier. The second technician successfully ran 28 of 29 runs with one incomplete test due to a power failure.

Figure 1.

Considering only the results that passed the machine quality control, agreement within technicians was high (98.8% of 81 and 98.9% of 348 for the first and second technicians, respectively with corresponding Kappa values of .97 for both technicians). Agreement between technicians was also high at 96.3% (Kappa=.90) (Table 1). Despite high agreement, the first technician had significantly more positive results than the second, when the two disagreed (6 versus 1, p<.01).

Table 1.

Intra- and inter-rater agreement of the careHPV^™ tests in women age 18–85*

First technician intra-rater agreement
	Second test
First test	Positive	Negative	Total
Positive	20	0	20
Negative	1	60	61
Total	21	60	81
98.8% agreement, Kappa=0.97, McNemar’s p=.32

Second technician intra-rater agreement
	Second test
First test	Positive	Negative	Total
Positive	76	2	78
Negative	2	268	270
Total	78	270	348
98.9% agreement, Kappa=0.97, McNemar’s p=.66

Inter-rater agreement of the first vs. second test of each technician
	Second technician
First technician	Positive	Negative	Total
Positive	40	6	46
Negative	1	141	142
Total	41	147	188
96.3% agreement, Kappa=0.90, McNemar’s p<.01

PCR and careHPV results (result of first test of first technician, else first test of second technician) were available for 299 women. Agreement varied by age (Table 2a). Among all women, agreement was 89.3% with a Kappa of .73. The careHPV test was less likely to test positive than PCR for carcinogenic HPV (25.1% vs. 29.8%, respectively, p<.01). Among 23 discordant cases where PCR was positive and the careHPV test was negative, one woman had a histologically-confirmed CIN3 lesion (HPV18 positive). Among 9 discordant cases that were PCR-negative and careHPV-positive, one histologically-confirmed CIN3 was detected. Considering PCR results as a virologic reference standard, careHPV correctly identified 74.2% of 89 women infected with carcinogenic HPV and 95.7% of 210 women without carcinogenic HPV infections. When considering results for 111 women within the age range typically considered for screen and treat strategies using HPV testing and immediate cryotherapy (30–45 years), agreement was 92.8% (Kappa=.74, p=.15) (Table 2b).

Table 2.

The CareHPV test compared to the MY09/MY11 PCR test from the same specimen

CareHPV resulta	PCR resultb
	Positive		Negative		Total
All women	N	%	N	%	N	%
careHPV positive	66	74.2	9	4.3	75	25.1
careHPV negative	23	25.8	201	95.7	224	74.9
Total (row %)	89	29.8	210	70.2	299	100.0
89.3% agreement, Kappa=0.73, McNemar’s p <.01

Age 18–29 (mean=23.8)
careHPV positive	16	88.9	1	3.3	17	35.4
careHPV negative	2	11.1	29	96.7	31	64.6
Total (row %)	18	37.5	30	62.5	48	100.0
93.8% agreement, Kappa=0.87, McNemar’s p =.56

Age 30–45 (mean=37.4)
careHPV positive	14	70.0	2	2.2	16	14.4
careHPV negative	6	30.0	89	97.8	95	85.6
Total (row %)	20	18.0	91	82.0	111	100.0
92.8% agreement, Kappa=0.74, McNemar’s p =.15

Age 46–85 (mean=58.9)
careHPV positive	36	70.6	6	6.7	42	30.0
careHPV negative	15	29.4	83	93.3	98	70.0
Total (row %)	51	36.4	89	63.6	140	100.0
85.0% agreement, Kappa=0.66, McNemar’s p =.05

^aThe careHPV test results from the first technician (n=186), else the second technician (n=113) if the first technician did not test specimen. CareHPV positivity is at 1.0 RLU threshold.

^bWomen are considered PCR positive if at least one of 13 carcinogenic HPV genotypes (16, 18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, 68) or HPV66 tested positive using a signal strength of one or greater as the threshold.

Table 3 presents the performance of the careHPV test compared to the histopathology results. Of 15 women with a confirmed CIN2 or worse (6 CIN2, 8 CIN3 and 1 cancer), 12 (80%) tested careHPV positive. The three women testing careHPV negative had RLU values of .647 (HPV18, 35, and 53 by PCR), .513 (PCR missing) and .303 (was also PCR-negative). Among 367 women without histologically-confirmed CIN2 or worse lesions, careHPV specificity was 78.5%; and highest for women age 30–45 at 85.3% (p=.04). When imputing to the entire screening population, the estimated specificity if women without a positive screening result and not randomly selected for colposcopy had been invited (n=815) was 83.0% and the estimated negative predictive value was 97.9%. For comparison, the estimated HPV PCR test sensitivity and specificity were 84.6% (of 13 cases of CIN2 or worse) and 72.7% (of 286 women with less than CIN2), respectively (data not shown). The imputed specificity of the HPV PCR test was 83.5%.

Table 3.

Women’s risk of cervical intraepithelial neoplasia grade 2 (CIN2) or worse by careHPV result

	Total		Histologically-confirmed CIN2 or worse		Less than CIN2		Estimated women with less than CIN2e
All women			N	% (sensitivity)	N	% (specificity)	Imputed N	% (estimated specificity)
careHPV positive	91	23.8	12a	80.0	79	21.5	145	17.0
careHPV negative	291	76.2	3b	20.0	288	78.5e	707	83.0
Total (row %)	382	100.0	15	3.9	367	96.1	852	100.0
Age 18–29 (mean=23.8)
careHPV positive	21	30.0	1	100.0	20	29.0	53	35.0
careHPV negative	49	70.0	0	0.0	49	71.0	98	65.0
Total (row %)	70	100.0	1	1.4	69	98.6	151	100.0
Age 30–45 (mean=37.4)
careHPV positive	20	14.6	0	0.0	20	14.7	20	6.4
careHPV negative	117	85.4	1c	100.0	116	85.3	291	93.6
Total (row %)	137	100.0	1	0.7	136	99.3	311	100.0
Age 46–85 (mean=58.9)
careHPV positive	50	28.6	11a	84.6	39	24.1	72	18.4
careHPV negative	125	71.4	2d	15.4	123	75.9	318	81.6
Total (row %)	175	100.0	13	7.3	162	92.6	390	100.0

CareHPV result is of first, else second, technician. CareHPV positivity is at 1.0 RLU threshold.

^aIncludes 1 cancer and 5 CIN3 lesions

^bConsists of 3 CIN3 lesions (1 PCR negative, 1 PCR positive for HPV18, 35 and 53, and 1 missing PCR result)

^cIncludes 1 CIN3 lesions

^dIncludes 2 CIN3 lesions

^eAdjusted to the entire population screened in first phase of the study. We extrapolated the test performance of careHPV among randomly-selected screen-negative controls to all remaining women with negative screening results in the cohort.

Discussion

In a validation study with real-world training and testing, the careHPV test had high intra- and inter-rater agreement. When comparing the careHPV test with HPV DNA testing by PCR, we found that the assays had good agreement, particularly among women within the age range typically considered for screening (30–45 years). However, using the PCR test as a reference standard for careHPV, sensitivity to detect women with carcinogenic HPV was 74.2% (95% CI: 63.8%-82.9%) while specificity to detect women without carcinogenic HPV infection was 95.7% (95% CI: 92.0%-98.0%).

In spite of examining 387 women in the context of colposcopic referral (most after an abnormal screening result), relatively few histologically confirmed cases of high grade lesions were detected (n=15) in this rural Nigerian screening population, providing inadequate power to accurately measure careHPV sensitivity. An initial study of the careHPV test among 2,530 women in China reported a sensitivity of 84.3% (95% CI: 75.8%-92.8%) to detect 70 women with histologically-confirmed CIN2 or worse, similar to our estimation of sensitivity at 80.0% (95% CI: 51.9%-95.7%)¹².

In this study, we estimate that careHPV is 83.0% (95% CI: 82.2%-87.2%) specific for identifying women without high-grade disease, similar to that reported by Qiao et al. of 87.5% (95% CI: 86.1%-88.8%)¹². The estimated negative predictive value of careHPV was 97.9% (95% CI: 96.6%-98.8%), and would provide assurance to women testing negative of their minimal risk of cervical precancer. Indeed, the high negative predictive value even over periods of 10 or more years means most (80% or more) older women testing HPV negative are provided reassurance of minimal risk. Such reassurance means women can be screened only 1 or 2 times in their lifetime while clinical management (e.g., a second test, colposcopy, or immediate treatment) can be focused on those who test HPV positive¹⁶.

We evaluated the performance of the careHPV test in a low-resource rural setting anticipated to represent the actual implementation of the test. In the first careHPV studies^12–14 highly-skilled QIAGEN Gaithersburg scientists travelled to the study locale to set up the careHPV system and train local technicians using a 3–5 day training protocol followed by an additional 5–7 days of observation of clinical testing. In this study, QIAGEN Gaithersburg staff did not train the technicians or set-up or closely monitor the laboratory as in other Research and Development trials^12–14. Kits (including reagents) were stored according to QIAGEN Research and Development recommendation, but not consistently refrigerated prior to testing.

Overall, we experienced little difficulty training technicians. In fact, the second technician, who was trained by the first one, eventually outperformed the first. In this study we were able to monitor performance remotely through remote review of log files stored within the careHPV test controller, a quality control mechanism available only in Research and Development trials where the controllers were modified to allow access to the log files via USB port. It is noteworthy that in separate occasions when the first technician’s performance waned and when some careHPV kits were exposed to high temperatures in an accidental fire, the test controller correctly indicated failure, showing problems with reagents or technician performance.

Outside of the technical functioning of the careHPV test, we did encounter errors with laboratory technicians. These included mistakes transcribing the specimen ID from the tube to the form as well as transcribing the careHPV result from the controller to the form. These problems were resolved through minor protocol modifications and the subsequent errors were minimal.

Although the colposcopy clinic was in the same location as enrollment screening and held at various days and times, we achieved only 69.0% compliance with follow-up colposcopy despite repeated direct intervention and a concerted community effort. In particular, we focused our final efforts to bring back 33 women with a high grade or cancer cytology result, and 14 (42.4%) still did not attend colposcopy. Such poor compliance and limited access to health services may warrant forgoing colposcopic confirmation of lesions in exchange for immediate treatment with a safe, effective and low cost technique such as cryotherapy.

Our results suggest that the careHPV test performs well in challenging remote settings with minimal interventions and has high specificity and adequate sensitivity. Incorporating careHPV into screening must now face the larger programmatic challenges of implementing large-scale programs using a batched test such as: access and affordability, transportation, lack of constant power supply in low resource-settings, participant organization and follow-up, and the likely need for self-sampling to make the screening program practical in some settings¹⁷.

Novelty and Impact.

Results from one of the first field trials of a novel HPV DNA assay created for low resource settings, the careHPV test, are presented. The careHPV test performed well in a challenging and remote setting with minimal intervention, suggesting the assay has high specificity and adequate sensitivity.