International Image Concordance Study to Compare a Point of Care Tampon Colposcope to a Standard-of-Care Colposcope

Abstract

Objective

Barriers to cervical cancer screening in low resource settings include lack of accessible high quality services, high cost, and the need for multiple visits. To address these challenges, we developed a low cost intra-vaginal optical cervical imaging device, the Point of Care Tampon (POCkeT) colposcope, and evaluated whether its performance is comparable to a standard-of-care colposcope.

Methods

There were two protocols, which included 44 and 18 patients respectively. For the first protocol, white light cervical images were collected in vivo, blinded by device, and sent electronically to 8 physicians from high, middle and low income countries. For the second protocol, green light images were also collected and sent electronically to the highest performing physician from the first protocol who has experience in both a high and low income country. For each image, physicians completed a survey assessing cervix characteristics and severity of precancerous lesions. Corresponding pathology was obtained for all image pairs.

Results

For the first protocol, average percent agreement between devices was 70% across all physicians. POCkeT and standard-of-care colposcope images had 37% and 51% percent agreement respectively with pathology for high-grade squamous intraepithelial lesions (HSILs). Investigation of HSIL POCkeT images revealed decreased visibility of vascularization and lack of contrast in lesion margins. After changes were made for the second protocol, the two devices achieved similar agreement to pathology for HSIL lesions (55%).

Conclusions

Based on the exploratory study, physician interpretation of cervix images acquired using a portable, low cost, POCkeT colposcope was comparable to a standard-of-care colposcope.

INTRODUCTION

Since cytology’s introduction 70 years ago¹, the Papanicolaou (Pap) test (if abnormal) followed by colposcopy has been the standard-of-care for cervical cancer prevention in the United States². While this strategy has been very successful in high income countries, the implementation requires trained cytopathologists and expensive infrastructure^3,4, which is not sustainable in most low-resource settings⁵. Colposcopes are expensive and require specialized training for proficient use, meaning they may not be practical in low-resource settings due to cost of equipment, maintenance and training⁶. Barriers to cervical cancer screening include lack of accessible high quality services, high cost, and the need for multiple visits⁷.

The World Health Organization recommends visual inspection with acetic acid (VIA) as the most efficient strategy to detect cervical cancer precursors in low-resource settings if human papillomavirus (HPV) screening is unavailable^8,9. In VIA, 3–5% acetic acid is applied to the cervix during a speculum-based pelvic exam, which the health provider evaluates for presence of acetowhitening with the naked eye. Studies have demonstrated screening with VIA is simple, affordable, sensitive and can be conducted by nurses to detect precancerous changes of the cervix⁵. However, VIA has lower specificity than HPV screening, leading to higher referral rates which may often overwhelm the referral centres, and when paired with a “see and treat” paradigm such as cryotherapy, leads to overtreatment¹⁰. In addition, interpretation is subjective and the success of VIA is highly dependent on training. Without recording images, it is difficult to ensure quality and improve training of health providers, particularly because images are not recorded and re-evaluated for skill enhancement¹¹.

Previously we described a low cost point of care tampon (POCkeT) colposcope for cervical cancer screening in low resource settings¹². The POCkeT colposcope combines benefits of colposcopy (magnification, green filter, digital image capture) with the cost, maintenance and accessibility criteria for low resource environments. The design was inspired by the tampon, which enables the camera to be at a much closer distance to the cervix (30 mm) than a traditional colposcope (300 mm). This provides comparable image quality while substantially decreasing cost through the use of a consumer grade light source and camera. The POCkeT colposcope is no more expensive than a SLR digital camera, but through bench testing we have demonstrated that it has comparable field of view, resolution and contrast compared to a standard-of-care colposcope¹². Additionally, the POCkeT colposcope can be plugged into a cell phone, tablet or computer which provides power to the device; therefore, it does not directly require electricity from a wall plug or power adapter, which enables it to be used in low resource environments where a majority of the population may not have access to electricity¹³. The goal of this study was to compare the images acquired with a POCkeT colposcope to a standard-of-care colposcope routinely used at Duke University Medical Center (DUMC).

METHODS

Device Description

A detailed comparison of both the POCkeT colposcope and standard-of-care colposcope systems has been previously described¹². Briefly, the POCkeT colposcope is an investigational research device that is intra-vaginal, portable, and low cost (Figure 1). Specifically, the POCkeT colposcope weighs 1 pound, primarily contains a consumer grade 5 megapixel camera and white and green light emitting diodes (LEDs) on the tip of the probe, and due to the lost low cost of these components has an anticipated price of $500 USD. The POCkeT colposcope interfaces with either a cell phone, tablet, or computer, which both provides power to the POCkeT colposcope and enables image capture. Thus, the POCkeT colposcope does not directly require electricity from a wall plug or power adapter. Images from the cervix are collected with the tip of the probe approximately 35mm from the cervical os. The standard-of-care colposcope (Leisegang Optik2, CooperSurgical, Inc. 2012. Trumbull, CT, USA.) at DUMC is stationary and must be plugged into an electrical outlet. The standard-of-care colposcope weighs 175 pounds, uses a 18 megapixel DSLR camera for digital image capture through a single chamber, and costs approximately $20,000 USD. Images of the cervix are collected approximately 300mm from the cervical os.

Figure 1.

Procedures

The study design is a diagnostic accuracy cohort study. Concordant cervical image pairs collected under Duke University Medical IRB approved protocol (Pro00008173) and registered ClinicalTrials.gov (NCT02477124) included adult females undergoing diagnostic colposcopy and/or Loop Electrosurgical Excision Procedure (LEEP) for previously diagnosed cytological and/or histological abnormalities of the cervix at DUMC. All potential participants were introduced to the study by a healthcare professional familiar with their medical history. All patients who participated signed an informed consent form. There were two protocols – the first included 44 patients, and the second included 18 patients. For both protocols, a speculum was placed in the vagina, and 5% acetic acid was applied to the cervix of each patient. The speculum could be used to manipulate the orientation of the cervix for better visualization and was required for both standard-of-care and POCkeT colposcope image capture. In the first protocol (N=44 patients), the standard-of-care colposcope was used first to capture images, and then the POCkeT colposcope was used to captured images of the same patient’s cervix using only white light. Approximately 2 minutes elapsed between acetic acid application and image capture with the standard-of-care colposcope, and an additional 3 minutes elapsed before image capture with the POCkeT colposcope. Preliminary analysis of images acquired with the first procotol indicated that acetic acid needed to be reapplied prior to image capture with each colposcope in order to optimize image contrast and that including green light could improve visualization of vasculature. Thus, in the second protocol (N=18 patients) the POCkeT colposcope was used to first capture images using both white and green light, and then acetic acid was reapplied before using the standard-of-care colposcope to capture images of the same patient’s cervix. All clinical decisions were completed using the standard-of-care colposcope, including directing biopsy or LEEP, which were sent for pathological interpretation at DUMC. All images had corresponding pathology, and pathologists were blinded to clinical interpretation from colposcopy. The cervix images were blinded and the corner was marked with a randomized image identifier with a secured key, which did not indicate which device captured the image. No other identifying marks were placed on the image, and all images evaluated in both protocols were raw and unmodified. Physicians viewed each image individually and were blinded to the patient’s prior history and to pathological interpretation.

Participating colposcopists

Eight physicians were surveyed: one from Indian Council of Medical Research (New Delhi, India), one from Cancer Institute WIA (Chennai, India), two from Kilimanjaro Christian Medical Center (Moshi, Tanzania), two from Duke University Medical Center (Durham, North Carolina, USA), one from La Liga Peruana de Lucha Contra el Cancer (Lima, Peru), and one from Kenyatta University School of Medicine, (Nairobi, Kenya). The eight participants are currently practicing obstetrics and gynecology, benign gynecology, or gynecologic oncology at their respective institutions with experience ranging from 1–40 years.

Image compilation

Image pairs were split, and individual images were randomized and placed into PDF documents to allow for physicians to review one image per page and zoom into regions of interest. Images from the first protocol (N=44 patients) were evaluated between November 2014 and October 2015. Six separate documents were created due to email size restrictions. Images from the second protocol (N=18 patients) were reviewed in April 2016 by one physician who was chosen based on his extensive experience working in both high and low-income settings (United States and Tanzania) and also because he performed the best when evaluating the first cohort of images.

Clinical assessment survey and image concordance study

Physicians were emailed a fillable PDF survey (see fillable PDF survey, Supplemental Digital Content 1, which physicians were emailed to score each cervix image) to complete for each cervix image. A web-based version of the survey was also available and could be completed by using a HIPAA compliant secured online survey created with the REDCap software platform¹⁴. Both the PDF and web-based version of the survey contained identical questions and formatting, and the method of survey completion was selected by each physician based on their preference. The survey included the randomized image identifier code, basic demographic information about the participating clinician, technical questions regarding image quality, and clinical questions, which were based off of features assessed in the modified Reid Index¹⁵ (see Table, Supplemental Digital Content 2, which includes definitions of features asked in clinical evaluation survey). The survey ended with the question “What is your grading of the cervix?” This was based on the physicians’ overall interpretation of each image and not solely on the Reid Index. The dataset was exported for statistical analysis using Stata version 14.0 (College Station, Texas) and Microsoft Excel (Microsoft Office Professional Plus 2013, Redmond, WA)¹⁷.

Analysis of the image concordance data

Both the original 44 image pairs and the additional 18 image pairs were assessed based on the response to “What is your grading of this cervix?”. 3X3 contingency tables broken down into normal, low-grade squamous intraepithelial lesion (LSIL) and high-grade squamous intraepithelial lesion (HSIL) were created to compare both systems to pathology. Histopathology was classified as normal, low-grade squamous intrapithelial (CIN 1) or high-grade squamous intraepithelial neoplasia (CIN 2+).

Role of funding source

We acknowledge financial support from the NIH Quick Trials Grant 1R21Ca162747. The funding source had no involvement in this study.

RESULTS

Pathology classification along with relevant demographic information for both protocols of image pairs is included in Table 1. A rendering of the POCkeT colposcope and representative image pairs captured with the standard-of-care colposcope and the POCkeT colposcope are shown in Figure 1.

Table 1.

Patient demographics and biopsy confirmed pathology for both protocols. Protocol 1 includes the original image pairs (44 patients) evaluated by 8 physicians. Protocol 2 includes the additional image pairs (18 patients) with procedural corrections to improve HSIL diagnosis. Unknown or Not Available categories are often high because many patients are referred to Duke from elsewhere, and therefore their medical record is incomplete.

		Protocol 1 (44 pairs)	Protocol 2 (18 pairs)
Age (years)	Mean	31 ± 8	31 ± 6
	Range	18 – 56	22 – 41
HIV test result	Negative (-)	21 (48%)	1 (6%)
	Not available	23 (52%)	17 (94%)
HPV test result	Positive (+)	27 (61%)	1 (6%)
	Negative (-)	1 (2%)	1 (6%)
	Not available	16 (36%)	16 (88%)
Cytology	Normal	4 (9%)	0 (0%)
	Abnormal	34 (77%)	15 (83%)
	ASC-US	11 (25%)	1 (6%)
	ASC-H	2 (5%)	2 (11%)
	LSIL	11 (25%)	3 (17%)
	HSIL	8 (18%)	9 (50%)
	AGC	2 (5%)	0 (0%)
	Unknown	6 (14%)	3 (17%)
Race	White	13 (30%)	5 (28%)
	Asian	1 (2%)	0 (0%)
	African American	9 (20%)	12 (67%)
	Unknown	21 (48%)	1 (6%)
Pathology	Normal	21 (48%)	4 (22%)
	LSIL	10 (23%)	3 (17%)
	HSIL	13 (30%)	11 (61%)

Image Concordance Analysis for 44 image pairs from the first protocol

For the first protocol of 44 patients, the overall percent agreement between the two devices and the percent agreement for each device compared to histopathology achieved by all 8 physicians are shown in Table 2. Physician interpretation between the two devices agreed 71% of the time for normal images and 51% of the time for HSIL images. For normal images, physician interpretation of the standard-of-care and POCkeT colposcopy images achieved 71% and 73% agreement respectively with pathology. For HSIL images, physician interpretation of the standard-of-care and POCkeT colposcopy images achieved 51% and 37% agreement respectively with pathology. Thus, both the standard-of-care and POCkeT colposcopes achieved comparable percent agreement when compared to pathology for normal cervices, but the POCkeT colposcope underperformed when detecting HSIL lesions. Sources of discrepancy were further investigated in Figure 2 and addressed through a second protocol.

Table 2.

Overall percent agreement for both devices compared to pathology as well as the POCkeT compared to a standard-of-care colposcope for original 44 image pairs. Overall statistics were calculated by compiling all physician responses (352 total image pairs). Pathological breakdown of the additional image pairs was 21 normal (48%), 10 LSIL (23%), and 13 HSIL (30%).

	Protocol 1
	Percent Agreement (%)
	Normal	LSIL	HSIL	Total
Standard-of-care Colposcope vs Pathology	71	24	51	54
POCkeT Colposcope vs Pathology	73	18	37	50
POCkeT Colposcope vs Standard-of-care Colposcope	89	33	51	69

Figure 2.

The percent agreement achieved between the interpretation of images captured with the standard-of-care and POCkeT colposcopes by individual physicians with different levels of experience was on average 70%, but there was inter-observer variability (Figure 2A). To determine the source of variability and if level of performance was correlated with experience, we examined the relationship between physician years of experience and percent agreement between the POCkeT colposcope and pathology. Physician experience was self-reported as number of years each physician has been performing colposcopy. As seen in Figure 2B, while the R² value is low, there is an increasing trend observed between the ability to correctly diagnose HSILs and years of experience in performing colposcopy. Thus, a histogram was created to understand what features led to the incorrect diagnosis of HSIL images acquired with each colposcope in the first protocol.. This evaluation was performed for the following cases in which: (1) both the standard-of-care and POCkeT colposcopes led to a correct diagnosis (both correct) (2) the standard-of-care colposcope led to a correct diagnosis, while POCkeT colposcope led to an incorrect diagnosis (standard-of-care correct, POCkeT incorrect), (3) the standard-of-care colposcope led to an incorrect diagnosis while POCkeT colposcope led to a correct diagnosis (standard-of-care incorrect, POCkeT correct), and (4) both standard-of-care and POCkeT led to an incorrect diagnosis (both incorrect). As shown in the histogram in Figure 2C (and in the Figure, Supplemental Digital Content 3, which contains representative HSIL image pairs from the first cohort that were correctly and incorrectly diagnosed), the lack of delineation of lesion margins and vascularization were identified as the most frequent reasons for incorrect diagnosis of HSlLs with the POCkeT colposcope.

Image Concordance Analysis for 18 image pairs from the second protocol

Eighteen additional images were collected after the first protocol, with the addition of two procedural changes to address lesion margin and vascularization. Sample images are shown in Figure 3. Additional image evaluations were completed by the highest performing physician from the first protocol who has experience in both a high and low income country. As shown in Table 3, the standard-of-care and POCkeT colposcopes both achieved 50% agreement with pathology for (n=4) normal cervices. For HSIL images, the standard-of-care and POCkeT colposcopes both achieved 55% agreement with pathology. Thus in the second protocol, both the standard-of-care and POCkeT colposcopes achieved comparable percent agreement when compared to pathology for both normal and HSIL cervices, indicating that the correction of frequently missed features (lesion margin and vascularization) increased the accuracy of POCkeT colposcope to pathology among HSIL images.

Figure 3.

Table 3.

Percent agreement for both devices compared to pathology as well as the POCkeT compared to a standard-of-care colposcope for the second protocol of image pairs, which were procedurally corrected for lack of visualization of lesion margin and vascularization. Only the highest performing physician completed these additional blinded and randomized evaluations for a total of 18 image pairs. Pathological breakdown of the additional image pairs was 4 normal (22%), 3 LSIL (17%), and 11 HSIL (61%). The two procedural adjustments in these additional 18 image pairs greatly improved the ability of the POCkeT colposcope to correctly identify pathological HSILs.

	Protocol 2
	Percent Agreement (%)
	Normal	LSIL	HSIL	Total
Standard-of-care Colposcope vs Pathology	50	67	55	56
POCkeT Colposcope vs Pathology	50	33	55	50
POCkeT Colposcope vs Standard-of-care Colposcope	80	33	86	67

DISCUSSION

The purpose of this study was to explore whether image quality is comparable between a standard-of-care colposcope (175 pounds, $20,000) and POCkeT colposcope (1 pound, anticipated price $500) for the interpretation of cervical images from normal tissues as well as cervices with low and high grade pre-cancers. Average percent agreement between the two sets of images was approximately 70%. Physician interpretation of colposcopy images (with both devices) agreed less frequently with histopathologic assessment than with each other. Previous studies have also demonstrated poor agreement between colposcopy and histopathology^18,19,20.

In the first protocol, physician interpretation of images collected with both devices achieved similar agreement to histopathology for normal cervices (>70%); however, physician interpretation of POCkeT colposcope images agreed less frequently with histopathology (37%) compared to standard-of-care colposcope images (51%) for HSIL cervices. Further investigation of HSIL images revealed that differences were due to decreased visibility of lesion margins and vascularization in POCkeT images. This is likely attributed to the lack of acetic acid enhanced contrast when imaging with the POCkeT colposcope (5–10 minutes) compared to that with the standard-of-care colposcope (1–5 minutes). It has been shown that acetic acid-induced whitening of HSILs maximizes around 3–5 minutes after application of acetic acid to the cervix, but that after five minutes the effects of whitening begins to drop off rapidly ²¹. The official guide for colposcopy examination from the IARC states acetic acid should be reapplied every 2–3 minutes throughout the examination because the effects of acetowhitening may begin to fade after one minute¹⁵. Frequently, vascularization was not identified in misdiagnosed POCkeT colposcope images while vascularization was identified in their matching standard-of-care images. Analysis of 18 additional image pairs confirmed that imaging immediately following acetic acid application and the addition of a green filter to enhance vascular contrast helped improve the ability of the of POCkeT colposcopy to identify HSIL lesions (55%) such that it was on par with the standard-of-care colposcope (55%). However, the revised study design in the second protocol of patients is not able to delineate the contribution of acetic acid enhancement versus the enhanced vascular contrast to the improved percent agreement between the POCkeT colposcope and pathology.

A study strength was the blinded randomized evaluation of images obtained with both the standard-of-care and POCkeT colposcopes. No indicators were visible to identify which colposcope device captured each image. A semi-quantitative analysis based on physician responses to the clinical assessment survey was used to understand which factors led to inaccurate and accurate diagnoses. Having histopathologic confirmation for all cervices provided an independent gold standard to which both POCkeT and standard-of-care colposcopy could be compared. In the second protocol of patients there was improved percent agreement between the POCkeT colposcope and the standard-of-care colposcope for HSIL images. This was achieved by analyzing the sources of error from the first study, which led to an amendment of the IRB protocol to re-apply acetic acid before imaging with each colpscope. Another strength of the study was having a single physician evaluate the images after the first inter-physician comparison was performed. There is often disagreement between physicians with regard to image evaluation and diagnosis. A study of 939 cervical images, found that pairs of colposcopists only agreed on diagnosis for 56.8% of images¹⁹. Hence, the first protocol focused on inter-observer variations with the goal of identifying the best performing physician, while the second protocol enabled comparison between protocols to examine the effects of acetic acid application and the addition of a green filter by one physician expert. Documenting physician history allowed for assessment of diagnostic performance as a function of experience in colposcopy to identify the single reader for the second set of images.

There were also several limitations associated with the study design that are briefly described below. First, this is a feasibility study of a new device in a small sample with a limited number of physicians interpreting the results. Future studies could incorporate a larger number of patients and physician readers to further validate the POCkeT colposcope and develop a detailed training methodology. This study confirms that all colposcopy, whether completed with the POCkeT or a standard-of-care colposcope, has several limitations that only histological sampling during or following the visualization can overcome. Anotherlimitation of the study was the number and size of biopsies acquired during diagnostic colposcopy exams. Liberal biopsy practice often leads to higher grade pathology²². Since the patient population was mixed between colposcopy and LEEP cases, if the biopsy practice was conservative during colposcopy, it is possible that both devices could be under-reporting the detection of HSIL precancerous lesions. Also, the clinical history of patients was not provided alongside the image, although colposcopists often factor in this information when making a clinical decisions. Providing clinical history alongside the images may have improved correlation of both devices with histopathology. Another study design concern is that static colposcope images were evaluated by physicians when colposcopy is normally a dynamic evaluation. However, past studies have explored the difference and found lack of significant differences between static and dynamic image evaluation when evaluating and interpreting colposcopy²³. Additionally, physicians did not consistently report on lesion size and location, which will be more relevant in the context of a larger sample size. All images evaluated were captured in a high-income setting (United States), so many lesions were of lower grade than what would typically be found in low-resource settings where frequent screening is uncommon.

In summary, the POCkeT colposcope is portable, low-cost, and may have the potential to be used in locations where standard colposcopy is inaccessible in both high and low income countries. The POCkeT colposcope provides adequate magnification, green light to visualize vasculature, and can capture and store images, which can be used to ensure comparable image quality as is achieved when compared with a standard-of-care colposcope. Additionally, clinicians can move the POCkeT colposcope back from the cervix and rest it on the speculum while taking a biopsy, which could enable physicians to leverage the light source from the POCkeT colposcope but still have the ability to manipulate the biopsy device. This approach mimics what is typically done with a standard-of-care colposcope – specifically, physicians often use a standard-of-care colposcope to first identify the area for biopsy, but do not take the biopsy under colposcopic guidance. Additionally, once real-time imaging and evaluation is implemented, POCkeT colposcope concordance results are likely to improve over the static evaluation presented in this study. The POCkeT colposcope is concordant with the standard-of-care colposcope when the same clinical protocol is used, which paves the way for assessing the benefits over VIA in a low-resource setting in future studies.

CONCLUSION

In the first protocol, the POCkeT colposcope performed comparably to the standard-of-care colposcope at identifying normal cervices. After changes were made to improve the protocol, the POCkeT colposcope performed comparably to the standard-of-care colposcope at identifying both normal and HSIL lesions. Thus, this exploratory study demonstrates that physician interpretation of cervix images acquired using a portable, low-cost, POCkeT colposcope was comparable to a standard-of-care colposcope.

Supplementary Material

Supplemental Data File _doc_ pdf_ etc.__1. Supplemental Digital Content 1.

Fillable PDF survey that physicians were emailed to score each cervix image. pdf

Supplemental Data File _doc_ pdf_ etc.__2. Supplemental Digital Content 2.

Table with definitions of features asked in clinical evaluation survey. pdf

Supplemental Data File _doc_ pdf_ etc.__3. Supplemental Digital Content 3.

Figure with representative HSIL image pairs from the first cohort that were correctly and incorrectly diagnosed. pdf

List of all abbreviations and acronyms

POCkeT

point of care tampon

LEEP

loop electrosurgical excision procedure

HSIL

high-grade squamous intraepithelial lesion

LSIL

low-grade squamous intraepithelial lesion

Pap

Papanicolaou

HPV

human papillomavirus

VIA

visual inspection with acetic acid

DUMC

Duke University Medical Center