Impact of Patient Reminders on Papanicolaou Test Completion for High-Risk Patients Identified by a Clinical Decision Support System

Abstract

Background: A clinical decision support system (CDSS) for cervical cancer screening identifies patients due for routine cervical cancer screening. Yet, high-risk patients who require more frequent screening or earlier follow-up to address past abnormal results are not identified. We aimed to assess the effect of a complex CDSS, incorporating national guidelines for high-risk patient screening and abnormal result management, its implementation to identify patients overdue for testing, and the outcome of sending a targeted recommendation for follow-up.

Materials and Methods: At three primary care clinics affiliated with an academic medical center, a reminder recommending an appointment for Papanicolaou (Pap) testing or Pap and human papillomavirus cotesting was sent to high-risk women aged 18 through 65 years (intervention group) identified by CDSS as overdue for testing. Historical control patients, who did not receive a reminder, were identified by CDSS 1 year before the date when reminders were sent to the intervention group. Test completion rates were compared between the intervention and control groups through a generalized estimating equation extension.

Results: Across the three sites, the average completion rate of recommended follow-up testing was significantly higher in the intervention group at 23.7% (61/257) than the completion rate at 3.3% (17/516) in the control group (p < 0.001).

Conclusions: A CDSS with enhanced capabilities to identify high-risk women due for cervical cancer testing beyond routine screening intervals, with subsequent patient notification, has the potential to decrease cervical precancer and cancer by improving adherence to guideline-compliant follow-up and needed treatment.

Introduction

Although the introduction and implementation of cervical cancer screening have reduced cervical cancer incidence and death in the United States by >60%,¹ 12,000 new cases and 4,000 deaths occur annually.² Screening programs aim to identify an abnormality at a point when intervention will make a meaningful difference in patient outcome. The progression from precancerous cervical cytology abnormalities to cervical cancer occurs over many years, which allows for successful screening as long as appropriate intervention and follow-up occur in response to abnormal screening results.³ Highest-risk groups for the development of cervical cancer are women who have never received screening, have been under-screened, and have delayed or no follow-up of abnormal results.^4–7

One possible factor contributing to inadequate care of women with a history of abnormal results of cervical cytology Papanicolaou (Pap) or human papillomavirus (HPV) testing or other risk factors is the complexity of cervical cancer screening and management guidelines. The American Society for Colposcopy and Cervical Pathology (ASCCP) updated the guidelines in 2012.^8,9 Studies of clinician application of the ASCCP screening guidelines reflect low levels of understanding and compliance. Teoh et al.¹⁰ reported that 12.1% of gynecology and primary care clinicians surveyed were not aware of the updated guidelines 1 year after release and just 5.7% answered all knowledge questions correctly. More than one-half of surveyed U.S. Pacific Northwest gynecologists reported performing screening tests more often than recommended by the updated guidelines.¹¹ Even among the 824 clinicians (65%) of 1,268 surveyed gynecology and primary care clinicians who endorsed support for the ASCCP screening guidelines, only 15% recommended the correct test type and screening intervals across all age groups.¹²

Fewer studies have focused on adherence to the more complicated guidelines for management of abnormal cervical cytology (Pap) or HPV tests. A Canadian study of compliance with guidelines developed in Ontario for managing low-grade abnormalities found overutilization of colposcopy referral and, more concerning, a lack of recommended follow-up in 13.4%–14.0% of women.¹³ In a recent publication specifically assessing compliance with ASCCP 2009 guidelines for management of abnormal Pap test results, more than one-half of patients in 1 of 3 university-based practices did not receive guideline-adherent intervention or were lost to follow-up.¹⁴ The lead author of the ASCCP guidelines acknowledged that (1) the management algorithms are complex and will likely only increase in complexity with new test modalities, and (2) information technology must be applied to assist clinicians.¹⁵

Clinical decision support systems (CDSSs) offer the potential to improve appropriate follow-up of high-risk patients by analyzing electronic health records (EHRs) to accurately identify patient populations that are not compliant with the guidelines-based recommendations. However, Pap test reports are in text format, and current CDSSs available in EHRs use only discrete data for decision making and therefore do not provide decision support for patients with abnormal cervical cytologic or HPV results.

We previously developed a CDSS to automate real-time recommendations to clinicians on cervical cancer screening intervals.^16,17 Subsequently, we demonstrated the potential of an enhanced CDSS that included management of abnormal results with natural language processing.¹⁸ Our CDSS involves a complex work flow, including 54 clinical pathways—13 dedicated to routine screening and 41 for patients with risk factors or abnormal test results. In this study, we evaluated the application of the enhanced CDSS to identify high-risk women overdue for follow-up testing at three primary care sites within Mayo Clinic in Rochester, Minnesota. We also evaluated the response rate of the women to electronic or mailed requests to schedule an appointment for Pap testing or Pap and HPV (Pap/HPV) cotesting.

Materials and Methods

The overall design was to compare response rates (Pap or Pap/HPV cotest completion) among women sent electronic or letter reminders with historical control subjects who were not sent reminders. The Mayo Clinic Institutional Review Board approved this study.

Setting, CDSS, participants, and controls

The study was conducted at three primary care sites affiliated with Mayo Clinic in Rochester, Minnesota, an academic medical center. A total of 25,500 women aged 18 through 65 years receive care at these clinics annually, with 63% at the Baldwin Clinic, 22% at the Northeast Clinic, and 15% at the Northwest Clinic. All clinical sites use the same EHR tool, GE Centricity. The CDSS extracted information from the EHRs for all women aged 18 through 65 years empaneled to primary care family medicine or internal medicine clinicians (i.e., physicians, nurse practitioners, and physician assistants) at the participating clinical sites.

The CDSS was built external to the EHR and interfaces through Web services to retrieve discrete data elements. However, much of the information needed for the CDSS to process complex care recommendations is in text format, such as pathology, cytology, or virology reports. Translating the unstructured reports into recognizable data elements for the decision support tool required use of natural-language processing to parse out the text from the EHR into a useable format. After its generation, care recommendations for each patient were stored in an accessible structured table.

Study participants were women identified by the CDSS as being at increased risk for cervical precancer or cancer and overdue for guidelines-based follow-up. Specifically, high-risk women were defined through two descriptions. First, they were women with a history of any abnormal results of Pap test, HPV test, or colposcopic biopsy who were overdue for follow-up per ASCCP management guidelines, which included women with past cervical intraepithelial neoplasia (CIN) grades 2–3 who did not have documentation of 1-, 2-, and 5-year cotesting following treatment or subsequent routine follow-up for 20 years. Second, the women were those who had a history of in utero diethylstilbestrol exposure, cervical cancer, solid organ transplant, or human immunodeficiency virus infection or were receiving long-term immunosuppressant medication and were overdue for screening, defined as >1 year since the last Pap test.

Patients were excluded if they already had a primary care or gynecology clinic appointment scheduled for a Pap test in the following 3 months. We also excluded patients who had their Pap test follow-up completed at an outside facility or had declined screening because of medical comorbidities and limited life expectancy. These exclusion criteria could not be identified by the CDSS; instead, they were assessed through EHR review for the intervention and control groups by the study team (K.L.M., M.E.K., and M.R.S.) or by the patient's primary care clinician. Control subjects were identified by the CDSS as appropriate candidates for the intervention at 1 year before the intervention due date at each clinical site.

Intervention

Women identified as appropriate candidates for the intervention received one electronic reminder for follow-up if they were registered on the clinic patient portal or one mailed letter reminder if they were not. The reminders explained that the patient had been identified as being at higher risk based on past results and was due for follow-up, emphasizing the importance of detecting treatable precancerous cervical lesions. The patient was given an electronic link or a phone number to schedule a clinic appointment. First, reminders for the Northeast Clinic were sent, staggered over 6 weeks because of concern about appointment access. Later, as appointment availability was not observed to be a problem at this first site, reminders were sent on a single date for the Northwest Clinic and Baldwin Clinic.

Data analyses

We described the data by mean (standard deviation) for quantitative variables and by count and percentage for qualitative variables. We used the general estimating equation extension of logistic regression model to describe the relationship between test completion and intervention, adjusting for study site and patient demographic characteristics and accounting for possible dependencies in response rates for individual patients between the two periods.¹⁹

Results

Baseline demographic characteristics for the intervention and control groups are presented in Table 1. Among all patients included for analysis, the mean age was 43.0 years; 2.1% had limited English language proficiency; 5.5% were black or African American; and 4.7% were uninsured. The intervention and control groups did not significantly differ in age, race/ethnicity, insurance type, education level, or English language proficiency.

Table 1.

Patient Demographics

	Patientsa
Characteristic	Intervention group (n = 257)	Control group (n = 516)	p
Age, mean (SD), years	43.9 (13.1)	42.6 (13.5)	0.22
Race/ethnicity
Asian, not Hispanic or Latino	2.5 (6/242)	2.0 (10/500)
Black or African American, not Hispanic or Latino	7.0 (17/242)	4.8 (24/500)
Hispanic or Latino	1.2 (3/242)	1.8 (9/500)
White, not Hispanic or Latino	89.3 (216/242)	91.4 (457/500)	0.56
Insurer
Private	73.2 (188/257)	72.1 (372/516)	0.83
Government	21.8 (56/257)	23.5 (121/516)
None	5.1 (13/257)	4.5 (23/516)
Education level
High school degree or less	19.5 (48/246)	20.9 (103/494)
Some college	40.1 (100/246)	41.3 (204/494)
4-Year college degree	21.5 (53/246)	21.1 (104/494)
Postgraduate study	18.3 (45/246)	16.8 (83/494)	0.53
Limited English language proficiency	3.2 (8/248)	1.6 (8/503)	0.14

^aValues are presented as percentage (fraction) of patients unless specified otherwise.

SD, standard deviation.

The effect of the intervention is presented in Table 2. Test completion rates increased more than sevenfold, from 3.3% to 23.7%. Accounting for potential site differences, the intervention group had a higher completion rate (odds ratio [OR], 8.31; 95% confidence interval, 4.68–14.80; p < 0.001). The odds of test completion varied marginally among the three study sites when accounting for intervention (p = 0.12). However, the effect of intervention was replicated across all sites (p < 0.002 for each site; OR range, 7.32–11.00; where ORs did not differ significantly, p = 0.89). These findings show no interaction between site and intervention. As well, no significant difference in test completion rate was found between family medicine providers and internal medicine providers by site (p = 0.96). The odds of test completion did not depend on patient age (OR, 1.004 per decade; p = 0.97), limited proficiency in English language (OR, 1.51; p = 0.65), years of education (OR, 1.06; p = 0.48), black or African American race/ethnicity (OR, 1.06; p = 0.92), or being uninsured (OR, 1.14; p = 0.09).

Table 2.

Papanicolaou Test or Papanicolaou/Human Papillomavirus Cotest Completion Rate Overall and by Clinic Site

	Test completion ratea
Site	Intervention group	Control group	OR (95% CI)	p
Overall	23.7 (61/257)	3.3 (17/516)	8.27 (4.64–14.75)	<0.001
Northeast Clinic	31.8 (21/66)	4.9 (4/81)	8.79 (2.78–27.78)	<0.001
Northwest Clinic	23.2 (22/95)	2.7 (2/74)	11.03 (2.44–49.74)	0.002
Baldwin Clinic	18.8 (18/96)	3.0 (11/361)	7.32 (3.35–16.03)	<0.001

^aTest completion rates are presented as percentage (fraction) of patients.

CI, confidence interval; OR, odds ratio.

The difference in patient volume between the control and intervention groups for the Baldwin Clinic was due to implementation of an information technology system in 2016 that changed how patients were assigned to providers. Because the new system was in the process of being implemented at the Baldwin practice during our study's intervention phase, not all patients had been assigned to providers when the CDSS was run to identify the Baldwin Clinic intervention group. This reduced the number of patients tracked per provider, which resulted in a decrease in the number of patients eligible for the intervention group. Because we had completed the implementation of this system at the two other clinics, we did not see a difference in patient volumes between control and intervention groups.

In the intervention group, 11 of the 61 women who responded to the reminder and completed Pap test or Pap/HPV cotest qualified for a subsequent colposcopy. Of that group, four women received a diagnosis by colposcopic biopsy of CIN grades 2–3 and required a loop electrosurgical excision procedure (LEEP) or cervical conization for treatment. The other seven women had negative or CIN grade 1 biopsy results. Among the 17 of 529 women in the control group who had Pap testing or Pap/HPV cotesting during the study period, 3 were referred for colposcopy. One patient had a diagnosis with CIN grade 3 and was treated with LEEP. The other two patients had CIN grade 1 and a negative colposcopic biopsy.

Discussion

Our intervention resulted in a significant increase in test completion for high-risk patients who were overdue for screening or follow-up of abnormal Pap test, HPV test, or colposcopy results. Patient reminders for routine cervical cancer screening have previously been proven effective to increase patient compliance.^20–23 However, studies of interventions to increase follow-up of abnormal Pap test findings have shown mixed results. In a qualitative meta-analysis of 10 studies designed to study the effect on patient compliance for follow-up of abnormal Pap results, phone-based education counseling resulted in the greatest improvement at 24%–26%, with behavioral interventions—including transportation vouchers and phone reminders—resulting in up to 18% improvement in compliance.²⁴ However, in the only study reviewed of patient letter reminders, the improvement rate was <1%—much lower than our improvement of greater than sevenfold in compliance.²⁴ Although the response rate was affected by our intervention, a point of concern is that slightly >75% of identified high-risk women in our study did not respond to a targeted recommendation of a clinic appointment for Pap testing. This finding warrants further study of interventions focused on this group of particularly at-risk women.

CDSS may enhance patient care in multiple ways, including preventive health and abnormal result follow-up reminders. A systematic review of the effect of CDSS applications on clinician performance demonstrated improvement with use of physician reminder systems for preventive services in 76% of 21 studies; those studies included but were not limited to Pap test reminders.²⁵ Systems-based interventions are recognized as useful in primary care but are often underused. Of the 2,475 physicians surveyed in the 2006–2007 National Survey of Primary Care Physicians' Recommendations and Practices for Breast, Cervical, Colorectal, and Lung Cancer Screening, <10% reported using all identified systems strategies for cancer screening identified by the study authors.²⁶ In a survey of 385 primary care clinicians in 2014, only 17.7% reported access to clinic systems capable of generating an automated prompt at an appointment for women overdue for follow-up of abnormal Pap results.²⁷

Recent applications of decision support for cervical cancer screening have focused primarily on incorporating educational alerts into the EHR to improve clinician compliance with guidelines-based recommendations for the appropriate age to start screening and to stop screening and for the correct use of Pap/HPV cotesting.^28–30 A study of an EHR tracking system to generate a paper record of women with abnormal Pap results and an EHR tracking table to document patient contact and subsequent colposcopy found no significant postintervention difference on bivariate analysis, although improvement was observed with multivariate analysis.³¹ Our study considered compliance as completion of a Pap test or Pap/HPV cotest within 4 months of receiving or qualifying for a reminder, reflecting the major change to abnormal Pap management in the updated ASCCP guidelines with far fewer algorithms recommending immediate referral for colposcopy. We are not aware of other studies using an automated CDSS to identify high-risk women who are overdue for screening or who are overdue for follow-up based on previous abnormal results.

Limitations of our study include the homogeneous demographic characteristics of the patient population (primarily white, insured, and educated) and the distinctiveness of our CDSS, which would limit applicability in other practice settings without access to this tool. However, our goal is to ultimately provide our software solution as an open-access resource to enable dissemination to other clinical practices.

The complexity of the current cervical cancer screening test management algorithms necessitates creative application of technology to guide clinicians in providing correct advice and management of abnormal results. The ASCCP app and website³² are helpful for managing an initial abnormal Pap or Pap/HPV result and subsequent colposcopic biopsy results, but these still require clinicians to review past medical history for other risk factors and more distant Pap, HPV, and colposcopic biopsy results and to decipher what stage a patient is in regarding the ASCCP algorithms for follow-up.

Conclusions

Cervical cancer guidelines for screening and algorithms for management are complex. Expecting clinicians to easily recall and correctly apply the recommendations over extended periods to their patient panels, many of whom are no longer seen annually, is unrealistic and unsafe. A CDSS with the ability to identify high-risk patients and generate patient lists for clinician review and subsequent patient reminders enables delivery of individualized care. It also provides a safety net for a patient group at increased risk for cervical cancer.

Acknowledgments

The Agency for Healthcare Quality and Research supported this research through grant R21HS022911, “NLP-enabled decision support for cervical cancer screening and surveillance.” The content is solely the responsibility of the authors and does not necessarily represent the official views of the U.S. Department of Health and Human Services.

Author Disclosure Statement

No competing financial interests exist.