Artificial Intelligence for Anal Cancer Screening: Multicenter Multicontinental Validation of a Trinary Model in High-Resolution Anoscopy

Abstract

Background and Aims

Anal cancer incidence is rising, primarily due to increasing rates of human papillomavirus infection. High-resolution anoscopy (HRA) is the gold standard diagnostic modality for evaluating human papillomavirus–related anal lesions; however, its broader adoption is limited by subjectivity and interobserver variability. This study aims to develop and validate a deep learning–based trinary classification model that detects and classifies high-grade squamous intraepithelial lesion (HSIL), low-grade squamous intraepithelial lesion (LSIL), and non-dysplastic lesions (eg, inflammation) on HRA frames.

Methods

A multicentric study was conducted to develop a convolutional neural network for automatic detection and classification of HSIL, LSIL, and non-dysplastic lesions. A You Only Look Once (YOLO)-v11 model was trained and validated on 191,951 frames (from 107 HRA procedures) containing histologically confirmed lesions, collected from 5 different imaging devices across 5 independent centers. Performance metrics (recall, precision, accuracy, and F1-score) were calculated at the object classification level as weighted averages across 5 confidence thresholds (0.45, 0.47, 0.50, 0.52, and 0.55).

Results

For HSIL and LSIL, recall was 97.9% and 98.9%, and precision was 93.9% and 98.5%, respectively. For non-dysplastic lesions, recall was 98.7%, and precision was 96.0%. The overall classification accuracy on the test set was 88.1% (95% CI: 78.2–98.1).

Conclusion

This is the first interoperable artificial intelligence model capable of simultaneous detection and trinary classification of anal canal lesions (HSIL, LSIL, and non-dysplastic mimickers). By addressing key limitations of HRA, this model supports broader clinical applicability and represents a significant step toward real-world deployment and personalized artificial intelligence–assisted care.

Introduction

Anal cancer (AC) is a rare malignancy, although its incidence is increasing.¹ This rise is primarily attributed to the growing prevalence of human papillomavirus (HPV), a well-established carcinogenic factor responsible for approximately 90% of squamous cell carcinoma cases. Squamous cell carcinoma is often preceded by high-grade squamous intraepithelial lesions (HSIL). The HPV-related carcinogenic pathway contributes to variability in AC prevalence across different population groups, with higher rates observed in individuals with human immunodeficiency virus (HIV), men who have sex with men, women with a history of lower genital tract neoplasia, and organ transplant recipients.²

Similar to cervical cancer screening, the goal of AC screening is the accurate detection of HSIL. Treatment has been shown to reduce the incidence of AC by nearly 60% in people living with HIV.³ Also, early detection limits the incidence of late-stage diagnoses, which is associated with poor survival and reduces the need for chemoradiation therapy. Although associated with significant acute and chronic adverse effects, it is the standard of care across all cancer stages.⁴

Current recommendations are fully aligned, recognizing high-resolution anoscopy (HRA) as the gold standard diagnostic method for assessing HPV-related anal lesions after any abnormality in screening. HRA enables direct visualization of lesions using magnification and contrast agents, which improves lesion detection, facilitates targeted biopsies, and guides clinical management.⁵

Despite its clinical value, HRA remains a complex and resource-intensive procedure, posing significant challenges to current AC screening practices. HRA is a highly operator-dependent technique with a steep learning curve. Clinicians with limited experience frequently fail HSIL detection.⁶ Furthermore, distinguishing true dysplastic lesions (low-grade squamous intraepithelial lesion [LSIL]/HSIL) from non-dysplastic changes, which are relatively frequent and can visually mimic HPV-related lesions (eg, inflammation, reactive changes), presents a significant diagnostic challenge, even for experienced practitioners, resulting in substantial diagnostic variability limiting appropriate triage, surveillance, and treatment planning.⁷

In line with current clinical practice, high referral rates for HRA are largely driven by the limited specificity of initial screening methods, including cytology and HPV cotesting, which tend to overidentify potential lesions.⁸ They are also driven by rising HPV prevalence, high recurrence rates, and broader eligibility resulting from the inclusion of newly recognized high-risk groups in screening protocols.⁵

The mismatch between demand and availability of HRA services is a well-recognized barrier to effective screening. The International Anal Neoplasia Society reinforces this reliance on specialized resources by recommending risk-stratified screening only in settings where HRA is accessible.⁵ However, this dependence limits equitable implementation, particularly in resource-constrained environments. These limitations make HRA a critical bottleneck in the AC screening pipeline, underscoring an urgent need for innovation.

Artificial intelligence (AI) holds substantial potential to address long-standing challenges in HRA by enhancing lesion detection, reducing operator dependency, and supporting clinical decision-making. While previous AI efforts have shown promise in certain aspects of HRA lesion differentiation, they often face limitations such as reliance on smaller, homogeneous datasets, or focus primarily on binary classification of HPV-related lesions,9, 10, 11 and have yet to fully address the complexities of real-world clinical deployment and the full spectrum of lesion identification. In fact, to date, no published studies have specifically focused on the capacity of automatic differentiation between dysplastic from common non-dysplastic mimickers in HRA.

The aim of this study was to develop and validate the first interoperable, deep learning model for HRA capable of trinary detection and classification of HSIL, LSIL, and non-dysplastic lesions.

Methods

Study Design, Data Collection, and Dataset Preparation

This multicenter study was conducted between 2020 and 2023 and involved 5 international centers located in the United States (n = 18 HRA), Brazil (n = 37 HRA), Argentina (n = 37 HRA), France (n = 11 HRA), and the United Kingdom (n = 4 HRA). A total of 107 HRA procedures were performed between 2020 and 2023.

The study utilized data captured from 5 distinct HRA devices: 4 colposcopes and 1 videoproctoscope. The colposcopes were the Kolplast colposcope (Kolplast CIA, São Paulo, Brazil), the Zeiss FC 150 colposcope (Carl Zeiss Meditec AG, Jena, Germany), the Seiler 955 5-Step colposcope (Seiler Instrument, St. Louis, USA), and the Newton CPLP Config 3 LED colposcope (Newton Microscópicos, São Paulo, Brazil). The videoproctoscope used was the THD Proctostation HRA Module (THD SpA, Correggio, Italy).

Each participating center obtained ethical approval in accordance with the Declaration of Helsinki. All procedures were performed by physicians specialized in HRA and included data acquisition of the entire procedure, including nonstained and stained phases (could be with acetic acid, lugol, or both). Tissue collection and histopathological assessment adhered to all established standards.

Anonymized video recordings were generated for every HRA procedure, in compliance with General Data Protection Regulation requirements. These recordings were subsequently processed for annotation purposes. The study exclusively included iconographic data from lesions that were biopsied, using histology as the gold standard for training/validation of the model.

Lesion-containing frames were initially annotated by a consensus of 2 physicians, resulting in 198,715 object-level annotations across 191,951 frames. At the procedure level, HSIL was present in 47 procedures (43.9%), LSIL in 48 procedures (44.9%), and non-dysplastic lesions in 12 procedures.

Dataset Preparation and Artificial Intelligence Model Development

The dataset was divided into training (70%), validation (20%), and test (10%) sets with a frame-level split (Figure 1). The validation set was used to monitor model performance and to explore different confidence score thresholds.

Figure 1.

Study design scheme.

We developed a convolutional neural network for automatic detection and classification of lesions into “HSIL,” “LSIL,” and “non-dysplastic lesion” (Figure 2). For this task, we employed the You Only Look Once (YOLO)-v11 object detection model, which identifies relevant lesions by drawing bounding boxes around them and assigning classification labels accordingly. To achieve this, the input image is first processed by a deep neural network, which generates a high-level feature map that represents the image. This feature map is then partitioned into an N × N grid, where each grid cell is tasked with identifying objects whose centers lie within its area.

Figure 2.

Examples of AI-corrected predictions for high-grade squamous intraepithelial lesion (HSIL), low-grade squamous intraepithelial lesion (LSIL), and non-dysplastic lesions. AI model predictions are shown in blue boxes, while physician consensus annotations are shown in green boxes.

To enhance bounding box prediction, non-maximum suppression was applied as a post-processing step. This procedure discards redundant detecting by comparing overlapping bounding boxes through their intersection over union scores in conjunction with confidence levels. The non-maximum suppression confidence and intersection over union thresholds were carefully adjusted to optimize the performance to maximize the model’s performance, achieving a balance between precision and recall in lesion detection.

Performance Evaluation

The model’s performance was evaluated at the object level to facilitate clinical interpretability and ensure consistency with the diagnostic reference standard. Each frame was assigned a single diagnostic label (“HSIL,” “LSIL,” or “non-dysplastic lesion”) based on the correspondent histopathological analysis. A frame was considered a true positive if at least one lesion detection of the correct category was detected in it.

Evaluation metrics included precision, recall, F1-score, and accuracy. The validation set was used to examine model behavior across a wide range of decision thresholds (0.30–0.80). From this analysis, a near-optimal threshold range (0.45–0.55) was identified, within which model performance was most stable. The test set was then evaluated across this range, and metrics were reported as weighted averages with 95% confidence intervals.

Statistical analysis was conducted using Scikit-learn, version 0.22.2.

Ethical Considerations

This investigation was conducted retrospectively and did not involve any interventional procedures. Clinical management and therapeutic decisions remained unaffected by the study. Ethical approval was obtained prior to initiation from the respective institutional review boards of Wake Forest Hospital, Instituto de Infecciologia Emílio Ribas, Centro Privado de Cirurgia y Coloproctologia, NHS Royal Liverpool University Hospital, and Group Hospitalier Paris Saint-Joseph. All procedures complied with the ethical standards of the Helsinki Declaration.

Results

The final dataset comprised 191,951 HRA frames obtained from 107 HRA procedures across 5 international centers. Table 1 summarizes the distribution of frames across centers and dataset splits.

Table 1.

Distribution of High-Resolution Anoscopy (HRA) Frames by Center/Device and Dataset Split (70% Training, 20% Validation, 10% Testing)

	Brazil (Kolplast)	US (Zeiss FC 150)	Argentina (Newton)	UK (Seiler 955)	France (THD HRA module)
Training	109,481	11,890	7701	1553	3752
Validation	31,278	3393	2205	444	1071
Test	15,634	1690	1099	220	540
Total	156,393	16,973	11,005	2217	5363

Model behavior was assessed during validation set across a range of confidence score thresholds between 0.30 and 0.80, as shown in Table 2. Based on this analysis, the test set was subsequently evaluated across a narrower range of 5 confidence thresholds (0.45, 0.47, 0.50, 0.52, and 0.55), and performance metrics were reported as weighted averages with 95% confidence intervals.

Table 2.

Model’s Performance Across Decision Thresholds During Validation (0.30–0.80)

Threshold	Lesion category	Recall (%)	Precision (%)	F1-score (%)	Overall accuracy (%)
0.30	HSIL	95.3	97.1	96.2	95.5
	LSIL	98.2	99.0	98.6
	Non-dysplastic	97.1	98.4	97.8
0.40	HSIL	96.9	96.2	96.6	95.9
	LSIL	98.6	98.8	98.7
	Non-dysplastic	98.1	98.0	98.1
0.50	HSIL	97.9	94.0	95.9	95.3
	LSIL	98.9	98.5	98.7
	Non-dysplastic	98.7	96.1	97.4
0.60	HSIL	98.8	89.1	93.7	92.5
	LSIL	99.2	97.5	98.4
	Non-dysplastic	99.2	89.7	94.2
0.70	HSIL	99.5	79.9	88.6	85.7
	LSIL	99.5	94.3	96.8
	Non-dysplastic	99.7	75.9	86.1
0.80	HSIL	99.8	57.9	73.3	64.0
	LSIL	99.8	74.9	85.6
	Non-dysplastic	99.8	47.8	64.6

Recall, precision, F1-score, and overall accuracy are reported per lesion category: high-grade squamous intraepithelial lesion (HSIL), low-grade squamous intraepithelial lesion (LSIL), and non-dysplastic lesion.

During the test set, the performance metrics were consistent across lesion categories. For HSIL, the model presented a recall (the same as sensitivity) of 97.9% (95% CI: 97.6–98.2) and a precision (the same as positive predictive value) of 93.9% (95% CI: 92.8–95.1). For LSIL, recall was 98.9% (95% CI: 98.8–99.0), while precision was 98.5% (95% CI: 98.3–98.7). For non-dysplastic lesions, the model attained a sensitivity of 98.7% (95% CI: 98.5–98.8) and a precision of 96.0% (95% CI: 94.9–97.2). Table 3 describes performance metrics, during the test set, across the 3 lesion classifications. Table 4 provides the confusion matrix of model classification performance.

Table 3.

Test-Set Performance of the Trinary Classification Model at the Frame Level

Performance metric	HSIL (95% CI)	LSIL (95% CI)	Non-dysplastic (95% CI)	Overall accuracy (95% CI)
Precision (%)	93.9 (92.8–95.1)	98.5 (98.3–98.7)	96.0 (94.9–97.2)	—
Recall (%)	97.9 (97.6–98.2)	98.9 (98.8–99.0)	98.7 (98.5–98.8)	—
F1-score (%)	90.7 (83.5–97.9)	96.1 (92.0–100.3)	89.7 (79.2–100.2)	—
Accuracy (%)	—	—	—	88.1 (78.2–98.1)

Recall, precision, and F1-score are reported per lesion category: high-grade squamous intraepithelial lesion (HSIL), low-grade squamous intraepithelial lesion (LSIL), and non-dysplastic lesion. These metrics are reported as weighted averages with 95% confidence intervals, computed across five confidence thresholds (0.45, 0.47, 0.50, 0.52, and 0.55). Overall accuracy is shown with 95% confidence intervals.

Table 4.

Confusion Matrix of Model Classification Performance Across the Test Set

	True HSIL	True LSIL	True non-dysplastic
Predicted HSIL	12,285	2	2
Predicted LSIL	6	19,003	0
Predicted non-dysplastic	0	0	7836

The overall classification accuracy on the test set was 88.1% (95% CI: 78.2–98.1). Figure 3 illustrates the discriminatory capacity of the model through precision-recall curves.

Figure 3.

Precision-recall curves illustrating the discriminatory capacity for HSIL, LSIL, and non-dysplastic lesions. Precision (positive predictive value) on the y-axis indicates the accuracy of positive detections, while recall (sensitivity) on the x-axis represents the detection rate of actual lesions. Higher precision indicates a lower false-positive rate, and higher recall means a lower false-negative rate.

Discussion

In this multicentric multidevice study, we developed and validated the first trinary AI model for HRA, designed to simultaneously detect and classify distinct types of anal lesions while providing object-level localization through bounding boxes. This capability extended beyond differentiating HPV-related lesions (HSIL vs LSIL) to also distinguishing them from non-dysplastic ones, with comparable recall (sensitivities) and precisions (positive predictive values), all exceeding 90%.

These findings need to be interpreted in the context of current clinical challenges of HRA. Although this technique is the gold standard procedure for detecting and treating HPV-related dysplastic lesions, it is associated with a steep learning process. Even among experience physicians, diagnostic accuracy remains suboptimal. For those beginning to practice this technique, the greatest challenge lies in detecting all relevant lesions. For experienced physicians, however, the difficulty is not recognizing that something is abnormal but rather in accurately distinguishing the lesion itself. While the use of stainings (acetic acid and/or lugol) can theoretically enhance differentiation, their diagnostic reliability is limited and they may increase the risk of false positives (eg, an acetowhite is caused by inflammation rather than dysplasia). Consequently, an AI-based technology capable of assisting physician during the procedure, not only in avoiding missed lesions but also in differentiating them in real time, would represent a major advance for this specialized (niche) technique. Such an approach could promote broader accessibility of HRA and enable more efficient diagnosis and/or treatment of clinically relevant lesions.

Within this context, previous studies have started to explore the development of AI to HRA, although development remained at an exploratory stage. The first study, published in 2022 (sensitivity of 91% for HSIL vs LSIL differentiation), was a single-center pilot study that included a limited number of procedure of only one type of HRA device.⁹ A second study, published in 2024 (sensitivity of 97% for HSIL vs LSIL differentiation), also single-center and device-specific, introduced a subanalysis of the model’s performance across different stages of the procedure, suggesting that performance metrics were minimal affected by presence of stainings, bleeding, or burnt tissue.¹⁰ A third study, also published in 2024 (sensitivity of 94% for HSIL vs LSIL differentiation), expanded the dataset by including 2 high-volume centers and initiated interoperability by incorporating data from more than one HRA device.¹¹

Building on this work, the present study represents another significant step toward enhancing the robustness and trustworthiness of a technological solution that may be relevant role in AC screening. On the one hand, it preserves the main advantages of previous works by adhering to FAIR criteria, particularly interoperability, by including data from 5 different HRA devices. This sine quan on criteria for AI development is especially relevant for a technique routinely performed with a wide variety of colposcopes and proctoscopes, as interoperability is key step toward making an AI model a viable scalable (worldwide) solution. On the other hand, it contributes to increasing the model’s robustness and mitigates selection bias by integrating transatlantic data from multiple centers and heterogeneous populations across Europe and the Americas, representing the largest dataset for AI development in HRA to date.

Importantly, the studies published to date have relied on training and validation datasets composed solely of frames containing histologically confirmed HPV-related lesions. The incorporation of other frame types has not yet been explored, which suggests that models developed so far would face limitation when applied in real-world settings, as they were unlikely to be fully adapted to segments of images not containing lesions. The introduction of a trinary model, therefore, represents a meaningful step toward addressing this limitation and enhancing its technological readiness level, bring it closer to everyday clinical practice.

Another clinically relevant feature of this model concerns its explainability. Until now, explainability in published studies has primarily relied on heat maps. Although these can indicate regions with a higher probability of representing lesions (eg, guiding biopsies), their interpretation is often challenging for clinicians without expertise in data science, especially when compared with more intuitive object-level representation provided by bounding boxes. Building on this, the present AI model (based on a YOLOv11 model) may facilitate real-time decision-making in clinical practice by not only detecting relevant lesions and highlight them with a bounding box but also supporting the physician in estimating the most probable histological diagnosis. While biopsies will remain for now indispensable as the gold standard for establishing a definite diagnosis, the integration of such an AI system could still represent a substantial enhancement of the technique if validated in future multicentric external cohorts.

Despite its strengths, the present study is not without limitations. The most relevant one is associated with the insufficient dataset size, which precluded a split based at the procedural level. To maximize data use for model development, this restriction was accepted. However, it introduces the possibility that similar frames from the same procedure may be present in both training and testing sets, increasing the risk of data leakage. Performance metrics in a real-life clinical setting may not be as high as those presented in this study. Moreover, class imbalance must be acknowledged, especially the limited number of non-dysplastic cases, as this may affect the robustness of the model in this class. Additionally, the exclusion of instrument-containing frames from the training dataset. While this decision was made to standardize image quality and focus the model on epithelial characteristics, it may reduce applicability in real-world scenarios where the presence of biopsy forceps, brushes, or other instruments within the frame is common. A further limitation is the reliance on still frames, which restricts direct extrapolation to video-based HRA, although such transfer is likely feasible. Another consideration is that the model was evaluated at the frame level, rather than at the object level, a strategy intentionally chosen to provide outputs that are more intuitive and clinically interpretable.

Conclusion

Although challenges remain, the potential of AI models in HRA is evident. Will development of such AI algorithms be able to overcome the challenges of HRA? Our opinion is clear. By enabling real-time, automated detection and classification of lesions (including differentiation between HSIL, LSIL, and non-dysplastic), such tools could reduce interobserver variability, support both junior and experienced clinicians in decision-making, and enhance diagnostic accuracy. Procedural-level external validation of these models in prospective, multicentric studies is necessary to prove they can work in a clinical environment. Future pathways may incorporate video-based data, enabling a dynamic assessment. Additionally, combining AI with clinical metadata (eg, HIV status, cytology, and HPV genotype) may further refine risk stratification. Prospective studies should also assess whether AI-enhanced HRA improves clinician performance during HRA. Ultimately, widespread implementation of interoperable, explainable, and validated AI systems could enhance HRA and make it a more accessible, standardized, and available tool, aimed at reducing the incidence of AC.