Use of electronic health record data to evaluate overuse of cervical cancer screening

Abstract

Background

National organizations historically focused on increasing use of effective services are now attempting to identify and discourage use of low-value services. Electronic health records (EHRs) could be used to measure use of low-value services, but few studies have examined this. The aim of the study was to: (1) determine if EHR data can be used to identify women eligible for an extended Pap testing interval; (2) determine the proportion of these women who received a Pap test sooner than recommended; and (3) assess the consequences of these low-value Pap tests.

Methods

Electronic query of EHR data identified women aged 30–65 years old who were at low-risk of cervical cancer and therefore eligible for an extended Pap testing interval of 3 years (as per professional society guidelines). Manual chart review assessed query accuracy. The use of low-value Pap tests (ie, those performed sooner than recommended) was measured, and adverse consequences of low-value Pap tests (ie, colposcopies performed as a result of low-value Pap tests) were identified.

Results

Manual chart review confirmed query accuracy. Two-thirds (1120/1705) of low-risk women received a Pap test sooner than recommended, and 21 colposcopies were performed as a result of this low-value Pap testing.

Conclusion

Secondary analysis of EHR data can accurately measure the use of low-value services such as Pap testing performed sooner than recommended in women at low risk of cervical cancer. Similar application of our methodology could facilitate efforts to simultaneously improve quality and decrease costs, maximizing value in the US healthcare system.

Introduction

Healthcare spending in the USA continues to increase more rapidly than inflation.¹ Most healthcare expenses are for services that improve patients' quality of life, longevity, or both. However, some tests are of low value—that is, they have marginal or no benefit, may harm patients, and waste financial resources. National organizations that have historically focused on increasing the use of effective services (eg, the American College of Physicians, the National Quality Forum, and the American Medical Association's Physician Consortium for Performance Improvement) are now attempting to decrease the use of low-value services.^2–6 To ultimately improve efficiency and reduce healthcare costs, methods will be needed to translate these recommendations into clinical practice.

Electronic health records (EHRs) have previously been utilized to increase the use of beneficial services, and they may similarly be utilized to decrease the use of low-value services.^{7 8} However, experience with using EHRs to identify low-value services is limited. Secondary analysis of EHR data has been used to examine if life expectancy (and therefore relative value) was associated with variation in cancer screening practices among older patients.^{9 10} In a randomized controlled trial by Tierney and colleagues, computerized predictive information improved targeting of laboratory tests to higher-risk patients while decreasing use in lower-risk patients.¹¹ Bates and colleagues used EHR alerts to decrease the overuse of low-value, redundant laboratory tests.¹²

EHR data could similarly be used to identify low-value preventive services, and Pap tests are a prime target. Professional guidelines are in relative agreement that annual Pap tests are a low-value service for women at low risk of cervical cancer and therefore recommend an extended screening interval in these low-risk women.^13–15 However, annual Pap tests are a high-value service for women at high risk of cervical cancer, and professional guidelines recommend continued annual screening in high-risk women.^13–15 Therefore efforts to minimize the use of low-value Pap tests could result in harm if high-risk women were to erroneously receive an extended screening interval. The information necessary to determine cervical cancer risk, and therefore Pap test value, is uniformly reported in searchable EHR fields. However, it is unknown if determinations of Pap test value based on this EHR data are sufficiently precise for use in efforts to measure and decrease the use of low-value Pap tests.

The aims of the study were to: (1) determine whether it is possible to use EHR data to accurately identify women eligible for an extended cervical cancer screening interval; (2) use EHR data to determine the proportion of low-risk women eligible for an extended screening interval who received a Pap test sooner than recommended by current guidelines; and (3) determine the number of low-risk women who underwent a colposcopy as a consequence of a Pap test performed sooner than recommended by current guidelines.

Methods

Definition of low-value Pap tests

Table 1 displays the cervical cancer screening guidelines at the time of the study.^14–16 If a woman eligible for triennial screening according to the guidelines received a Pap test sooner than recommended, that Pap test was considered to be a low-value Pap test.

Table 1.

Cervical cancer screening practice guidelines current as of 2007

	American College of Obstetrics and Gynecology (2003)	American Cancer Society (2002)	US Preventive Services Task Force (2003)
Screening interval	For women ≥30, may extend interval to: Every 2–3 years if three consecutive tests are NIL Every 3 years if NIL test and negative HPV	For women ≥30, may extend interval to: Every 2–3 years if three consecutive tests are NIL Every 3 years if NIL test and negative HPV	At least every 3 years

HPV, human papilloma virus; NIL, negative for intraepithelial lesion.

Practice setting and EHR

We used patient data from the Northwestern Medical Faculty Foundation (NMFF) General Internal Medicine Clinic, an urban, academic, primary care practice with 38 general internal medicine attending and 51 resident physicians and approximately 60 000 clinic visits yearly. All physicians perform liquid-based Pap tests and use the Hybrid Capture II human papilloma virus (HPV) test. All physicians use an EHR for all clinical encounters (EpicCare; Epic Systems Corporation, Madison, Wisconsin, USA). The EHR has discrete fields for medical history, surgical history, current and past medications (including date of order and discontinuation), encounter diagnoses, and a problem list. Diagnosis names are linked to International Classification of Diseases, Ninth Revision (ICD-9) codes.

The EHR includes data from all specialties within NMFF and includes comprehensive clinical decision support for preventive care and disease management. The clinical decision support system includes a point of care reminder to perform Pap testing. The default interval is set at 1 year after the last Pap test result was recorded; providers can manually change the alert interval to biennial or triennial screening when indicated (ie, the provider determines that a woman is low risk). As of 2010, 31 of 38 (81.6%) general internal medicine attending providers had changed the Pap test alert frequency from 1 to 3 years for at least one of their patients.

Eligibility criteria

This study was approved by the institutional review board at Northwestern University. An electronic query identified all women 30–65 years old with one or more visits to any NMFF general internal medicine provider between January 1, 2007 and December 31, 2007. We defined low-risk women according to American Cancer Society, American College of Obstetrics and Gynecology, and US Preventative Services Task Force guidelines at the time of the study (table 1).^14–16 The electronic query identified women with a Pap test read as ‘negative for intraepithelial lesion’ (NIL) in 2007 and two prior NIL Pap tests in 2004–2006. These women were considered to be low risk and therefore eligible for an extended screening interval. In addition, women with a NIL Pap test and a negative HPV test in 2007 were considered to be low risk and therefore eligible for an extended screening interval (box 1). At the time of this study, manual chart review was necessary to determine HPV results. The medical history, problem list, encounter diagnoses, medication list, orders, and Pap test reports from January 1, 2004 to December 31, 2009 were queried to characterize these women, the care they received, and the results of testing. The electronic query can be found in the online appendix.

Box 1. Inclusion and exclusion criteria.

Eligible for extended interval

Women 30–65 years old

2007 Pap test read as NIL

One of the following:

• Two NIL Pap tests between January 1, 2004 and December 31, 2006

• Negative HPV test in 2007

Ineligible for extended interval

Past medical history, encounter diagnosis, or problem list code for:

• Immunosuppression

  • Solid organ transplant

  • HIV

  • Administration of chemotherapy

  • Immune deficiency disorder

  • Neutropenia

• Previously abnormal Pap tests

  • History of CIN II, III, or cervical cancer

  • Abnormal Pap test between 2004 and 2007

  • Colposcopy between 2004 and 2007

• Diethylstilbestrol exposure Medication codes for:

  • Azathioprine

  • 6-Mercaptopurine

  • Methotrexate

  • Entanercept

  • Tacrolimus

  • Sirolimus

  • Infliximab

  • Adilimumab

  • Muromonab-CD3

  • Basiliximab

  • Daclizumab

  • Atgam

  • Cyclophosphamide

  • Cyclosporin

  • Anakinra

  • Mycophenolate mofetil

  • Granulocyte colony-stimulating factor

CIN, cervical intraepithelial neoplasia; HPV, human papilloma virus; NIL, negative for intraepithelial lesion.

Exclusion criteria

We excluded women at high risk of cervical cancer because they are not eligible for extending the screening interval. High-risk women were identified via electronic query. As per professional guidelines, high risk was defined as a history of an abnormal Pap test between 2004 and 2007, cervical intraepithelial neoplasia (CIN) II or III, cervical cancer, HPV positive in 2007, diethylstilbestrol exposure in utero, immunosuppressive medication in 2007, or diagnosis associated with immunosuppression between 2004 and 2007 (box 1).^{14 16}

Analysis

To ensure patients were included appropriately, we manually reviewed the charts of 100 randomly selected women who the electronic query classified as low risk (ie, eligible for an extended screening interval). Data including age, race, marital status, number of visits per year, and number of chronic illnesses were extracted from the EHR database to characterize those women eligible for an extended screening interval.

Using EHR data, we identified all Pap tests performed during 2008 and 2009 in women who were eligible for an extended screening interval. These Pap tests were considered to be of low value because they were performed sooner than recommended. We identified all women in the cohort who had a colposcopy following the Pap test in 2008 or 2009. All colposcopies performed in these women were considered to have the potential to be adverse consequences of low-value Pap tests. We performed a separate manual review of the records belonging to those women receiving colposcopies, because we felt that these women were most likely to have been misclassified as low risk.

Results

Low-risk women eligible for an extended screening interval

The EHR query identified 4002 women who had a NIL Pap test in 2007. Of these, 1749 were not eligible for an extended screening interval because they had not received two prior normal Pap tests during 2004–2006 or a negative HPV test in 2007. An additional 548 women were excluded after electronic query revealed an excluding diagnosis, excluding medication, or abnormal Pap test between 2004 and 2007. Ultimately, 1705 women were identified as eligible for an extended screening interval (figure 1). Patient characteristics are shown in table 2; 53.1% of eligible women were married, and 92.5% had ≤1 chronic condition.

Figure 1.

Identification of eligible women, low-value Pap testing, and its consequences. *These two women were excluded after manual chart review of all patients receiving colposcopy as a consequence of low-value Pap testing. One had a history of cervical intraepithelial neoplasia (CIN) of unknown severity recorded only in provider notes. The other had a history of CIN II recorded only in provider notes. NIL, negative for intraepithelial lesion.

Table 2.

Characteristics of women eligible for an extended screening interval (n=1705)

Characteristic	n (%)
Marital status
Single	669 (39.2)
Married	904 (53.1)
Divorced	99 (5.8)
Widowed	26 (1.5)
Separated	5 (0.3)
Unknown	2 (0.1)
Race
White	856 (50.2)
African-American	379 (22.2)
Latino	101 (5.9)
Other	42 (2.5)
Unknown	327 (19.1)
Number of GIM visits in 2007
≤2	1190 (69.8)
3–5	416 (24.4)
≥6	99 (5.8)
Number of chronic conditions
0	1146 (67.2)
1–2	535 (31.4)
≥3	24 (1.4)
Chronic conditions by disease
Hypertension	367 (21.5)
Diabetes	79 (4.6)
History of cancer	9 (0.5)
Chronic pulmonary disease	222 (13.2)
Coronary artery disease	18 (1.1)
Congestive heart failure	5 (0.3)
Peripheral vascular disease	6 (0.4)
Chronic liver disease	3 (0.2)
Chronic kidney disease	4 (0.2)
Dementia	3 (0.2)

GIM, general internal medicine.

Confirming query accuracy

We found 99 of the 100 women randomly selected for manual chart review were appropriately classified as eligible for an extended screening interval (ie, low risk). The one high-risk woman misclassified as low risk had a history of CIN of unknown severity that was recorded only in free-text notes.

Use of low-value Pap testing

Of the 1705 women identified as eligible for an extended screening interval by electronic query, 1120 (65.7%) received a low-value Pap test in 2008 or 2009 (figure 1). In 2008, 839 low-value Pap tests were performed, and 712 in 2009. A total of 431 (25.3%) women received a low-value Pap test in both years.

Consequences of Pap testing sooner than recommended

The electronic query identified 23 women who underwent colposcopy. Review of providers' electronic notes (of these 23 women) revealed that two women were misclassified as low risk (ie, had indications for annual Pap testing). One had a history of CIN II, and the other had a history of CIN of unknown severity; this information was recorded only in free-text notes. After exclusion of these two women, a total of 21 women (1.2% of those eligible for an extended screening interval) had undergone a colposcopy as a consequence of a low-value Pap test in 2008 or 2009 (figure 1). Five women who underwent colposcopy as a consequence of low-value Pap tests had CIN I. The remaining women had normal colposcopic findings or koilocytotic atypia consistent with HPV. Despite the lack of indication for an annual Pap, one woman's colposcopy led to a diagnosis of adenocarcinoma in situ (AIS).

Results using an extended screening interval of 2 years

If providers were following an extended screening interval of 2 years (the shortest interval recommended by the American College of Obstetrics and Gynecology and American Cancer Society guidelines) instead of 3 years, then 839 (49.2%) women received a low-value Pap test in 2008. A total of 10 (0.6%) women received a colposcopy as a consequence of a low-value Pap test in 2008.

Discussion

Clinical guidelines for preventive services such as Pap testing must always balance the benefits, harms, and costs of screening. Nationally representative models have determined that the risk of cervical cancer attributable to extended interval screening of low-risk women is approximately three per 100 000 women.¹⁷ The identification of one additional case of invasive cervical cancer requires a large number of additional tests at a significant cost.^{17 18} On the basis of this information, professional societies have decided that the cost and harm of more frequent screening was not justified, and they recommended that low-risk women undergo triennial screening and avoid more frequent, low-value Pap testing.^{13–15 19} However, the guidelines maintain that annual Pap testing is of high value for women at high risk of cervical cancer.^{13–15 19} Therefore Pap tests are a good example of a high frequency service with variable value that providers and policymakers should include in efforts to improve healthcare efficiency, while simultaneously maintaining efforts to ensure that all women receive cervical cancer screening when appropriate.

Our study did not intend to challenge or validate national cervical cancer screening recommendations. Rather, we sought only to demonstrate how EHR data can be used to measure guideline-concordant care. However, our findings do bring the balance of risks and benefits of more frequent screening into sharp focus. A total of 21 women had negative colposcopies and likely experienced physical discomfort and psychological distress over possibly having cancer as a consequence of low-value Pap testing.^{20 21} On the other hand, we did identify a woman with adenocarcinoma in situ (AIS) who might have been harmed if she had not been screened for an additional year; this is a predictable consequence of less frequent screening.

In our study, 66% of low-risk women received Pap tests sooner than recommended, representing inefficient care. These low-value Pap tests had significant downstream consequences. The unavoidable false-positive results and follow-up colposcopies associated with overscreening can cause undue psychological stress for women without cervical cancer.^{20 21} In addition, the financial burden of overscreening is substantial. Assuming a Pap test cost of US$63 and a colposcopy cost of US$286, the cost of low-value Pap tests and their consequences in our practice was approximately US$100 000.²² Although the results of this single-practice study cannot be extrapolated to the entire population, it is likely that low-value Pap tests cost the US healthcare system approximately US$0.5–1 billion per year while achieving little or no improvements in health.^{22 23}

In order to eliminate this low-value spending, we must first have a reliable method to measure the use of low-value services. Our study successfully demonstrated that EHR data can be used for this purpose. The query accurately identified women at low risk of cervical cancer (and therefore low-value Pap testing), but it was not perfect. One percent of high-risk women were misclassified as low risk because information pertinent to assessment of cervical cancer risk was recorded only within free-text provider notes. Ideally, the query would be 100% specific to ensure that no harms could result from erroneously labeling a high-risk woman as low risk and advising an extended interval of 3 years. However, even if the specificity of queries can be improved, it is likely that there will always be a low rate of incorrectly labeling some women as low risk, which has potential harm.

This study had several limitations. First, we did not address all situations in which cervical cancer screening is of low value. National surveys have identified propensity for cervical cancer overscreening with respect to initiation of screening, screening in women after hysterectomies, failure to extend the screening interval when using a liquid-based Pap test, failure to stop screening in older women, and low-value use of the HPV test.^24–28 All could probably be identified with similar electronic queries and would be worthwhile investigating in the future. Second, we did not have results of HPV tests before 2007. If a woman had three NIL Pap tests and a positive HPV test in 2005 or 2006, she may have been misclassified as low risk. However, this is unlikely to significantly affect the results, as only a small minority of women seen received HPV testing at the time of the study. Third, by only reviewing Pap tests between 2004 and 2007, we may have inadvertently excluded those low-risk women not receiving Pap tests sooner than recommended. However, only 21 women had a NIL Pap test in 2004 with no subsequent Pap tests until 2007. Including this small number of low-risk women who are (presumably) receiving extended interval screening as recommended would not have significantly affected our results. Finally, the study was conducted in the general internal medicine practice of a single multispecialty group using a single EHR. Accuracy of the query may vary in other EHRs and at other practices.

Of note, the study practice's EHR includes a health maintenance prompt to perform Pap testing, and our practice has set the default frequency at 1 year (ie, the reminder occurs annually unless a clinician changes the reminder interval). This was done to ensure that high-risk women were not inadvertently screened every 3 years (ie, prevent underuse); this may have promoted overscreening in low-risk women. However, the rate of low-value Pap test use in our study was similar to that reported in national surveys, in which 65% of providers report performing Pap tests sooner than recommended.²⁵ Therefore we believe the default setting of 1 year used for the cervical cancer screening alert in our practice was not a major factor contributing to the high rate of low-value Pap tests performed in this study.

Nonetheless, it is important to acknowledge the ramifications of regularly using default settings for clinical reminder systems. Because default settings assume that a service has uniform risk and benefit across patients, the choice of a default setting can lead to overscreening (if the default setting is too short) or underscreening (if the default setting is too long). Algorithms such as ours can use EHR data to determine which setting is most appropriate for each individual patient. Such methods could be used to optimize preventive service use (ie, minimize both over- and under-use) and should be explored further as a means of ensuring the provision of necessary services while simultaneously decreasing costs.

Ultimately, the fundamental problem with increasing efficiency to cut healthcare costs is the inherent tension between wanting to make sure that patients receive all necessary tests (avoiding underuse) while simultaneously not using low-value services such as annual Pap testing in women at low risk of cervical cancer (avoiding overuse). To date, the problem of overuse has largely been ignored.²⁹ However, increasing healthcare costs and pressure from health policymakers have spurred initiatives by several professional organizations to identify low-value services and recommend against their use.^2–6

Ours and other studies suggest that these recommendations alone are insufficient to significantly reduce low-value service use in clinical practice.²⁵ We have demonstrated a method to leverage EHR data to guide interventions that could systematically reduce the use of low-value services identified by these professional societies while simultaneously maintaining efforts to reduce underuse and ensure that all patients receive necessary services.

However, any attempt to use EHR data in this manner is likely to be controversial, and steps should be taken to minimize potential harms. First, what constitutes a low-value service must be clearly defined and openly communicated to patients to ensure they do not feel necessary care is being withheld. Second, effective methods for preventing low frequency misclassification errors should be explored. Improved methods of data capture, such as natural language processing, may ultimately improve accuracy by identifying information buried in text fields. Even if such improvement can be achieved, real-time provider oversight is necessary to catch misclassification errors as they occur. For example, a provider reviewing a patient's newest Pap test results could be prompted to confirm the electronic query's determination that a woman has been newly determined to be at low risk of cervical cancer. Finally, interventions to decrease low-value service use should be implemented in conjunction with additional quality improvement efforts to ensure that the delivery of recommended preventive services continues.

Conclusions

We have shown that EHR data can be used to accurately identify low-value Pap test use, that low-value Pap testing occurs commonly, and that it has significant consequences. Similar application of our methodology could leverage increasingly available EHR data to systematically reduce low-value service use. However, the ramifications of using our methods in clinical practice are not well understood. Initial implementation should be coupled with research to identify optimal methods to address patient concerns, to minimize harms from misclassification errors, and to ensure continued delivery of recommended preventive services. Although efforts should proceed cautiously at first, our methodology could ultimately guide efforts to decrease low-value service use and maximize value in healthcare, simultaneously improving quality and decreasing costs in the US healthcare system.