Initial evaluation of a new cervical screening strategy combining human papillomavirus genotyping and automated visual evaluation: the Human Papillomavirus–Automated Visual Evaluation Consortium

Abstract

The HPV-Automated Visual Evaluation Consortium is validating a cervical screening strategy enabling accurate cervical screening in resource-limited settings. A rapid, low-cost human papillomavirus (HPV) assay permits sensitive HPV testing of self-collected vaginal specimens; HPV-negative women are reassured. Triage of positive participants combines HPV genotyping (4 groups in order of cancer risk) and visual inspection assisted by automated cervical visual evaluation that classifies cervical appearance as severe, indeterminate, or normal. Together, the combination predicts which women have precancer, permitting targeted management to those most needing treatment. We analyzed CIN3+ yield for each HPV-Automated Visual Evaluation risk level (HPV genotype crossed by automated cervical visual evaluation classification) from 9 clinical sites (Brazil, Cambodia, Dominican Republic, El Salvador, Eswatini, Honduras, Malawi, Nigeria, and Tanzania). Data from 1832 HPV-positive participants confirmed that HPV genotype and automated cervical visual evaluation classification strongly and independently predict risk of histologic CIN3+. The combination of these low-cost tests provided excellent risk stratification, warranting pre-implementation demonstration projects.

Introduction

Almost all cases of cervical cancer are caused by persistent infections with one of a dozen genotypes of human papillomaviruses (HPV).^1,2 One dose of prophylactic HPV vaccine in adolescents can prevent cervical cancer that might otherwise manifest decades later.³ However, HPV vaccine coverage is low and unequal globally, not all important types are covered, and women past the age of vaccination remain at risk. Because of the long latency between infection and cancer, full impact of implemented multivalent vaccine programs will take several subsequent decades, with many avoidable deaths unless screening and treatment programs are also implemented.⁴ Consequently, improving the coverage of accurate cervical screening remains an international health priority, especially in underserved, resource-limited settings.⁵

Although HPV testing is recommended for screening, there are very few affordable, accurate HPV-based screening tests and strategies. The HPV-Automated Visual Evaluation (PAVE) Consortium is a cervical screening research effort specifically targeting resource-limited settings.⁶ Cooperative PAVE research, organized as a consortium of independent clinical centers, is scheduled to conclude in mid-2025. This midstudy evaluation was conducted roughly halfway through enrollment. The results are very promising, warranting a brief communication. The analysis was performed on a dataset frozen as of July 16, 2024, which includes 27 654 women (ie, 20 788 HPV-negative women and the first 1832 of the 6866 HPV-positive women to have completed the management protocol). The analytic dataset includes Brazil (n = 6119 enrolled), Cambodia (n = 5071), the Dominican Republic (n = 599), El Salvador (n = 6198), Eswatini (n = 375), Honduras (n = 50), Malawi (n = 2764), Nigeria (n = 5828), and Tanzania (n = 650).

Women at PAVE Consortium sites were recruited for cervical cancer screening using a variety of community awareness campaigns, workplace screenings, door-to-door household contact, and child and reproductive health clinics. In some countries, HIV clinic attendees were also invited to participate. Visits and treatments were free of charge, and at some sites, participants were supported by transportation stipends according to local institutional review board approvals.

Methods

Figure 1 shows the shared screening methods and strategy in PAVE, which was conducted under written informed consent. After signed informed consent, women collect a self-sampling of vaginal cells for HPV testing. Women with a negative result are reassured. Women with a positive test are called for a triage visit with a speculum evaluation of the cervix with acetic acid followed by digital image taking; at least 1 biopsy of the ectocervix and/or endocervix is/are required in all HPV-positive women. Images at 1 minute after acetic acid are sent for an AI-based evaluation (AVE) for the classification of potential abnormalities into 3 categories: normal, indeterminate, and severe. The yield of histologically confirmed CIN3+ cases is calculated for the crossing of the 2 tests: HPV and AVE. At the clinic for this phase of the PAVE study, standard of care (visual inspection with acetic acid, colposcopy or cytology) sets the threshold for treatment of HPV-positive women with thermal ablation, large-loop excision of the transformation zone, or referral for advanced care.

Figure 1.

Human Papillomavirus–Automated Visual Evaluation strategy outline from recruitment to treatment. ^aColor code illustrates a possible management recommendation based on risk of CIN3+, for each cell in the risk table derived by crossing human papillomavirus type group and AVE classification: Low <2%, Medium 2-9.9%, High 10-19.9%, Highest >20% (Table 1). The stability of risk estimates and resultant recommendations is low because of small numbers of participants with high-grade disease, which are expected to increase by more than fourfold by end of study. Abbreviations: AI = artificial intelligence; Neg = negative.

PAVE is assessing a novel cervical “screen-triage-treat” approach designed for women aged 25 years (given HIV) or 30-49 years. The primary objective is to target and treat precancers (stringently defined as cervical intraepithial neoplace grade 3, adenocarcinoma in situ, or cancer among women infected with at least 1 of the carcinogenic HPV types), especially in known high-risk areas with limited resources. The screening step is based on sensitive testing of self-sampled vaginal specimens. Even in HPV high-prevalence settings using a sensitive assay, most women in midadulthood will screen HPV negative and can be reassured of low cancer risk for some years ahead.⁷ However, HPV infections are common enough in most high-risk regions to seek added triage testing (ie, prioritization of management with treatment intensity matching the severity of the findings). We combined 2 relatively affordable triage methods that complement each other, namely, a new “extended” HPV genotyping assay and automated cervical visual evaluation.^8-10 This report tabulates the research performance across the PAVE Consortium of these novel tests in combination in predicting CIN3+, which was treated (as was CIN2 for safety) by thermal ablation, large-loop excision of the transformation zone, or advanced care as needed.⁶

The devices and reagents chosen for PAVE were adapted with company collaborators sharing the goal of expanding accurate cervical screening to resource-limited regions. However, National Cancer Institute and PAVE research collaborators have no financial interest in these particular choices, and the strategy anticipates further improvements. Vaginal self-samples were collected using FLOQSwabs (Copan, Brescia, Italy), transported dry in the original tubes and tested locally using a novel isothermal DNA amplification assay (ScreenFire HPV RS Test, Atila Biosystems, Sunnyvale, CA, USA). ScreenFire sensitivity is high and similar to reference polymerase chain reasion; thus, if negative, participants are reassured of low cancer risk.¹¹ If positive, the assay yields HPV results in 4 channels that reflect natural evolutionary groupings with respect to cervical precancer and cancer risk: HPV16 is most carcinogenic, followed by HPV 18 and 45; HPV 31, 33, 35, 52, and 58; and lastly, HPV 39, 51, 56, 59, and 68.^8,12 In case of multiple infections, the highest risk channel is reported because the risk of multiple concurrent infections is determined mainly by the most carcinogenic type present.^1,13 HPV-positive women are referred for triage to a speculum examination where images are collected using an Iris digital camera (Liger Medical, Lehi, Utah, USA) 60 seconds after application of acetic acid (60 seconds postapplication images used in automated cervical visual evaluation training). Visual inspection with acetic acid or colposcopic impressions are assessed, and pathology samples are collected. Up to 4 punch biopsies or a soft-brush biopsy (SoftBiopsy/SoftECC, Histologics, Anaheim, California, USA) are collected from acetowhite areas, and an endocervical sample or cytology is collected when no lesion is visible. The Iris device converts through an attachment to conduct immediate thermal ablation. If indicated, women are referred instead for large-loop excision of the transformation zone or advanced care. Large-loop excision of the transformation zone tissue, if applicable, is also used for histologic evaluation. Treatment, if provided at the same visit, is done after all sample and image collection.

The automated cervical visual evaluation algorithm used for this analysis is based on previously published PAVE research designed to optimize diagnostic accuracy and classification repeatability. The objective in developing the algorithm was to find a trustworthy visual aid that further stratifies risk of cervical cancer when combined with HPV genotype groups.^10,14 Briefly, the algorithm chosen for this analysis uses a DenseNet 121 neural network architecture with a quadratic weighted kappa loss function, balanced ground truth classification during training, Monte Carlo dropout, and a 3-class ground truth design.

A critical part of artificial intelligence (AI)–based visual evaluation algorithms is the correct labeling of training images. The present automated cervical visual evaluation model was trained on a variety of digital images from more than 10 000 women from non-PAVE studies and enriched with Liger Iris images from only 120 PAVE patients, kept to the needed minimum to prevent overfitting.¹⁰ AI algorithms trained on too high of a proportion of the data can lead to overfitting and overly optimistic prediction statistics.¹⁴ Portability between devices is not yet possible without retraining (ie, additional training that includes images of CIN2+, indeterminate, and normal taken with the new device).^15,16 Automated cervical visual evaluation model training is based on a 3-level classification model designed to attain high confidence in the most definite patients with disease and those without. For training, severe cases are defined as having high-risk HPV positivity and adjudicated CIN2 or worse histology. (The testing is even more stringent, requiring CIN3+ diagnosis with CIN2+ considered secondarily.) Normal controls are high-risk HPV-negative without visual acetowhitening. All other participants are combined into the indeterminate training class.

The algorithm (see below) is intended to run as point of care on the Iris device, but during this first phase, digital images were transferred, and the algorithm training and testing were computed on specialized graphics processing unit servers.

Results

The HPV groups and automated cervical visual evaluation results when crossed form 12 risk-based strata, which rank risk from highest to lowest. This allows different management recommendations to be adapted by health authorities depending on risk tolerance and treatment resources.¹⁶

The results stress reproducibility, accuracy, and risk stratification. Almost all (98.0%) study participants had 2 or more images taken in rapid succession at 1-minute postacetic acid application. The automated cervical visual evaluation classification showed substantial agreement between the 2 images (weighted kappa of 0.78). In total, 85.2% of paired images had the same automated cervical visual evaluation classification, 14.4% were off by one class (ie, normal and indeterminate or indeterminate and severe), and only 0.7% disagreed by 2 classes (ie, normal and severe).

The main results regarding accuracy are shown in Table 1. Among HPV-positive women, the overall yield of CIN3+ was 7.5%. The 2 biomarkers alone had similar CIN3+ yield, with automated cervical visual evaluation ranging from 1.1% to 13.8% (P_trend < .0001) and HPV ranging from 3.6% to 20.8% (P_trend <.0001). When combined, the PAVE approach generated a larger range of CIN3+ yield of 0.0% up to 29.0%. Specifically, when automated cervical visual evaluation was evaluated within each HPV category, the yield of CIN3+ increased as the automated cervical visual evaluation classification increased from normal to indeterminate to severe. Within each automated cervical visual evaluation classification, risk increased by HPV type stratum. Risk stratification also held for women living with HIV (data not shown). Of note, the HPV 18 and 45 and HPV 31, 33, 35, 52, and 58 channels showed very similar risks of CIN3+. ScreenFire is designed to distinguish the HPV 18 and 45 channel from the HPV 31, 33, 35, 52, and 58 because the former yields increased risk of invasive cancer, particularly adenocarcinoma. By design (age restriction meant to capture the peak of treatable precancers prior to invasion), there were few cancers in this study sample (approximately 10% of participants with high-grade disease).

Table 1.

Detection of CIN3+ participants in 4 HPV channels ranked by carcinogenicity and automatic visual evaluation strata ranked by level of abnormality^a

	Automatic visual evaluation
	CIN3+/numberb
HPV genotyping	Normal, No. (%)	Indeterminate, No. (%)	Severe, No. (%)	Total,c No. (%)
HPV 39, 51, 56, 59, 68	0/128 (0.0)	6/332 (1.8)	13/199 (6.5)	19/659 (3.6)
HPV 31, 33, 35, 52, 58	1/82 (1.2)	18/331 (5.4)	28/217 (12.9)	47/630 (7.5)
HPV 18 and 45	1/48 (2.1)	4/143 (2.8)	15/102 (14.7)	20/293 (6.8)
HPV 16	1/27 (3.7)e	22/123 (17.9)e	29/100 (29.0)e	52/250 (20.8)
Totald	3/285 (1.1)	50/929 (5.4)	85/618 (13.8)	138/1832 (7.5)

Abbreviations: HPV = human papillomavirus.

^aEach numeric table cell represents the yield of CIN3+ with specific HPV channel positivity and automatic visual evaluation label. Percentages are derived from computing the number of CIN3+ detected in the cell divided by the total observations in the table cell multiplied by 100.

^bThe table shows that within each automatic visual evaluation category, there is an increasing yield of CIN3+ with increasing carcinogenicity of the HPV channels, and similarly, within each HPV channel, there is an increasing yield of CIN3+ with increasing image severity.

^cTest for linear trend for the percentage of CIN3+ yield within HPV strata; P value less than .0001.

^cTest for linear trend for the percentage of CIN3+ yield within AVE strata; P value less than .0001.

^dTests for linear trends for automatic visual evaluation categories within each stratum of HPV and HPV categories within each stratum of automatic visual evaluation were all less than .01 except for the automatic visual evaluation normal stratum in which the HPV trend based on smaller numbers of cases had a P value of .06.

The comparable risks of CIN2+ are shown in Table S1; comparison of the 2 endpoints shows as predicted that risk stratification is better for CIN3+, the more certain surrogate endpoint closer to invasive cancer risk. Histologic CIN2 is an equivocal classification that inevitably contains error, weakening causal associations.¹⁷

To illustrate the translation of the risk estimates into clinical management, the 9-stratum genotype (combining HPV 18 and 45 and HPV 31, 33, 35, 52, and 58) and automated cervical visual evaluation classifications (normal, indeterminate, severe) were ordered from highest to lowest yield of CIN3+. The ordering can be used to calculate the cumulative proportion of participants with CIN3+ that could be found by referring or treating increasing proportions of HPV positives.¹⁶ The calculation indicates that approximately 90% of cases could be identified within approximately 60% of HPV positives. At a population level (accepting from prior research the strong reassurance given by a negative ScreenFire test), and assuming HPV positivity of approximately 25%, our approach could identify approximately 95% of all cases within approximately 15% of the overall screened population. With somewhat lowered sensitivity, exclusion of the lowest-risk strata could substantially reduce referrals.

Discussion

The preliminary PAVE efficacy data suggest that HPV genotyping and automated cervical visual evaluation are complementary screening biomarkers, which in combination could provide accurate cervical cancer screening and triage. The results are in line with established understanding of HPV phylogeny, natural history, and multistage cervical carcinogenesis.¹⁸ Of note, visual screening is optimal in women in their thirties; performance degrades with age.¹⁹ The combination could fit with 1-dose HPV prophylactic vaccination as part of cervical reduction programs linking mothers and daughters.

The PAVE strategy could be provided relatively inexpensively and was designed for this purpose. However, it remains relatively costly and complex for the lowest-resource settings. We continue to seek simpler and faster (true point-of-care) strategies. For example, a simpler version of ScreenFire with prefilled wells that require very little pipetting has recently been developed and is being implemented. Development work to increase speed, accuracy, and portability of the AI approaches is ongoing. Finally, for very low-resource settings, automated cervical visual evaluation alone is being considered as a tool to assist with visual inspection accuracy when HPV testing is simply unaffordable.

Full analysis of PAVE will be based on approximately 50 000 screened women. Case numbers will increase by roughly fivefold, permitting more thorough examination of important subgroups (especially women living with Human immunodeficiency virus) and identification of look-alike visual imposters (eg, the combination of inflammation and ectopy that can appear to be a case). A major limitation of PAVE is that the automated cervical visual evaluation classification was not generated locally on the Iris processor, and the full PAVE combination was not used clinically in real-time. A full effectiveness study is needed to examine the impact on performance in real use, particularly once the automated cervical visual evaluation is performed point of care (currently taking 20 seconds) on the processor in the image capture device. Such a trial could permit comparison of different triage combinations of HPV-type groups, provider visual assessment, and automated cervical visual evaluation and could assess affordability and sustainability more directly. It would permit study of communication strategies to patients and providers concerning the use of HPV typing and AI algorithms for screening and the use of risk to guide management decisions. Such data, if supportive, would provide international public health decision makers with sufficient evidence to move toward implementation.

Author contributions

Brian Befano (Conceptualization, Data curation, Formal analysis, Methodology, Writing—original draft, Writing—review & editing), Jayashree Kalpathy-Cramer (Conceptualization, Methodology, Resources, Supervision, Writing—review & editing), Didem Egemen (Conceptualization, Formal analysis, Writing—review & editing), Federica Inturrisi (Conceptualization, Data curation, Writing—review & editing), Jose Jeronimo (Conceptualization, Writing—review & editing), Ana Cecilia Rodriguez (Conceptualization, Data curation, Writing—review & editing), Nicole Campos (Conceptualization, Writing—review & editing), Miriam Cremer (Project administration, Writing—review & editing), Ana Ribeiro (Investigation, Writing—review & editing), Kayode Olusegun Ajenifuja (Investigation, Writing—review & editing), Andrew Goldstein (Investigation, Writing—review & editing), Amna Haider (Investigation, Writing—review & editing), Karen Yeates (Investigation, Writing—review & editing), Margaret Madeleine (Investigation, Writing—review & editing), Teresa Norris (Investigation, Writing—review & editing), Jacqueline Figueroa (Investigation, Writing—review & editing), Karla Alfaro (Investigation, Writing—review & editing), Tainá Raiol (Investigation, Writing—review & editing), Clement Adepiti (Investigation, Writing—review & editing), Judith Norman (Investigation, Writing—review & editing), George Kassim Chilinda (Investigation, Writing—review & editing), Bariki Mchome (Investigation, Writing—review & editing), Yeycy Donastorg (Investigation, Writing—review & editing), Xolisile Dlamini (Investigation, Writing—review & editing), Gabriel Conzuelo (Investigation, Writing—review & editing), Adekunbiola Banjo (Investigation, Writing—review & editing), Pauline Chone (Investigation, Writing—review & editing), Alex Mremi (Investigation, Writing—review & editing), Arismendy Benitez (Investigation, Writing—review & editing), Zeev Rosberger (Investigation, Writing—review & editing), Te Vantha (Investigation, Writing—review & editing), Ignacio Prieto-Egido (Data curation, Software, Validation, Writing—review & editing), Jen Boyd-Morin (Data curation, Project administration, Software, Writing—review & editing), Christopher Clark (Conceptualization, Data curation, Formal analysis, Writing—review & editing), Scott Kinder (Conceptualization, Data curation, Formal analysis, Writing—review & editing), Nicolas Wentzensen (Conceptualization, Writing—review & editing), Kanan Desai (Conceptualization, Writing—review & editing), Rebecca Perkins (Conceptualization, Writing—review & editing), Silvia de Sanjose (Conceptualization, Funding acquisition, Investigation, Writing—review & editing), and Mark Schiffman (Conceptualization, Formal analysis, Funding acquisition, Project administration, Writing—review & editing).

Supplementary material

Supplementary material is available at JNCI: Journal of the National Cancer Institute online.

Funding

The consortium sites are responsible for their own clinic and personnel costs. Research equipment and supplies and the NCI-affiliated core staff are funded by the National Cancer Institute Intramural Research Program including supplemental funding from the NCI Cancer Cures “Moonshot” Initiative. Equipment and supplies were purchased; no commercial contributions were obtained. Brian Befano was supported by NCI/NIH under Grant T32CA09168.

Conflicts of interest

Brian Befano –NCI/NIH Grant T32CA09168.

Jayashree Kalpathy-Cramer—Nothing to disclose.

Didem Egemen—Nothing to disclose.

Federica Inturrisi—Nothing to disclose.

José Jeronimo—Nothing to disclose.

Ana Cecilia Rodríguez—Nothing to disclose.

Nicole Campos—NCI Intramural Research Program, including supplemental funding from the Cancer Cures Moonshot Initiative. NCI grants U01CA199334/U01CA253912.

Miriam Cremer—Nothing to disclose.

Ana Ribeiro—Nothing to disclose.

Kayode Olusegun Ajenifuja—Nothing to disclose.

Andrew Goldstein—Nothing to disclose.

Amna Haider—Nothing to disclose.

Karen Yeates—Nothing to disclose.

Margaret Madeleine—U54 CA242977. ScreenFire Assay, Attila—Swabs/Copan. IRIS, Liger—Scanner, Motic.

Teresa Norris—HPV Global Action has received a contract from the Public Health Agency of Canada for the 2024-2026 period. Merck has provided unrestricted grants for projects aimed at raising awareness and educating various populations. These projects include producing white papers, reports, materials, organizing think tanks, webinars, expert symposiums, social media campaigns, school presentations, and a pride campaign. Roche has provided an unrestricted grant for a project focused on self-sampling for cervical screening in underserved populations and for hosting a symposium. Hologic has also given an unrestricted grant for a webinar. Pfizer has provided an unrestricted grant for vaccine education initiatives. The Azrieli Foundation has made a donation to support the general mission of HPV Global Action, as has PayPal Giving Fund Canada. Various high schools across Canada have made donations to support sexual health presentations. Merck has sponsored an HPV vaccine-related event, in exchange for expert consultation. Hologic has funded attendance at the Eurogin conference in Stockholm and supported a Patient Advocacy Group Day event in Brussels. Additionally, the President of HPV Global Action is involved with the organization as a registered charity in both Canada and the European Union, and serves as the Treasurer for Coalition Priorité Cancer au Québec, a registered charity in Quebec. There was also a donation of McDonald's gift cards from McDonald Incentives, which were distributed during sexual health presentations.

Jacqueline Figueroa—Nothing to disclose.

Karla Alfaro—Nothing to disclose.

Tainá Raiol—Nothing to disclose.

Clement Adepiti—Nothing to disclose.

Judith Norman—Nothing to disclose.

George Kassim Chilinda—Nothing to disclose.

Bariki Mchome—Nothing to disclose.

Yeycy Donastorg—Nothing to disclose.

Xolisile Dlamini—Nothing to disclose.

Gabriel Conzuelo—NCI—Single Visit Clinical validation of ScreenFire, a Low-Cost HPV Test: Efficacy and Cost Effectiveness (SCALE) (R01CA266059). NCI—PROGRESS: PRevention of cervical cancer using the Genotyping scREening and Same-day Self-sampling. Basic Health International—President and founder. Organon—Trainer. Merck—Advisory board.

Adekunbiola A Banjo—Nothing to disclose.

Pauline Chone—Nothing to disclose.

Alex Mremi—Nothing to disclose.

Arismendy Benitez—Nothing to disclose.

Zeev Rosberger—HPV Global Action (co-applicant) Public Health Agency of Canada (2024-2026) contract to HPV Global Action; small stipend given to me for services rendered. Canadian Immunization Research Network (2023-2024) PI: Samara Perez (co-investigator) – partial funds transferred from the PI at the McGill University Health Centre to the Lady Davis Institute for Medical Research. Quebec Cancer Coalition Payment made to my institution: Lady Davis Institute for Medical Research. Vice-President, Scientific Affairs, HPV Global Action – a registered charity in Canada and the European Union): unpaid position.

Te Vantha—Nothing to disclose.

Ignacio Prieto-Egido—NCI—Contract between the NCI and the EHAS Foundation where EHAS provides data collection tools for this work. Spanish Agency for International Development Cooperation—AECID, Spain/Comunidad de Madrid—Grants for development projects related with primary healthcare in Guatemala and Peru. Fundación EHAS, Madrid, Spain—Unpaid role as director of the foundation.

Jen Boyd-Morin—Nothing to disclose.

Christopher Clark—Nothing to disclose.

Scott Kinder—Nothing to disclose.

Nicolas Wentzensen—Nothing to disclose.

Kanan Desai—Nothing to disclose.

Rebecca Perkins—NCI—contract, American Cancer Society—Travel for academic meetings, American Cancer Society—Steering committee, National Roundtable on Cervical Cancer, National HPV vaccination Roundtable, NCI—Co-Chair, Enduring Guidelines.

Silvia de Sanjose—Nothing to disclose.

Mark Schiffman—Nothing to disclose.

Data availability

The data underlying this article can be made available and shared for research purposes in accordance with institutional review boards on reasonable request to the local study authorities.