Can automated alerts within computerized physician order entry improve compliance with laboratory practice guidelines for ordering Pap tests?

Abstract

Background:

The electronic health record (EHR) provides opportunity to improve health and enhance appropriate test utilization through decision support. Electronic alerts in the order entry system can guide test use. Few published reports have assessed the impact of automated alerts on compliance of Pap ordering with published screening guidelines.

Methods:

Programming rules for Pap test ordering were developed within the EHR (Epic, Madison, WI) of the University of California, Davis Health System using American College of Obstetrics and Gynecology's 2009 guidelines and implemented in primary care clinics in 2010. Alerts discouraged Pap orders in women <21 and >71 years and displayed when an order was initiated. Providers were not prevented from placing an order. Results were measured during four calendar periods: (1) pre-alert (baseline) (July 2010 to June 2011), (2) post alert (alerts on) (July 2011 to December 2011), (3) inadvertent alert turn-off (“glitch”) (January 2012 to December 2012), (5) post-glitch (alerts re-instated) (1/2013-7/2013). Metrics used to measure alert impact were between time and period seasonally adjusted relative frequency ratios.

Results:

Alerts were most effective in the <21 year old age group. During the baseline period 2.7 Pap tests were order in patients less than age 21 for every 100 Paps in those 21-71 years of age. This relative frequency decreased to 1.7 in the post-alert period and 1.4 during the glitch, with an even greater decline to 0.8 post-glitch when alerts were reinstated. Less impact was observed in the >70 year old group where the baseline relative frequency was 2.4 and declined to 2.1 post-alert, remained stable at 2.0 during the glitch period, and declined again to 1.7 post-glitch when alerts were reinstated. This likely reflects inclusion of women with a history of abnormal Pap tests for whom continued Pap testing is indicated, as well as reluctance by providers and patients to accept discontinuation of Pap testing for women with a history of normal Pap results. In both age groups, decreases in ordering were greatest when the alerts were functioning, indicating that the alerts had an effect beyond the influences of the environment.

Conclusions:

Discouraging alerts can impact ordering of Pap tests and improve compliance with established guidelines, thus avoiding unnecessary follow-up tests that can create potential patient harm and unnecessary expense. Alerts represent a potential model to address utilization of other lab tests. Longer study intervals are necessary to determine if provider compliance is maintained.

INTRODUCTION

There is a growing expectation that health care providers will control utilization of laboratory testing in order to improve quality of care and reduce the costs of unnecessary testing. In 1998, the Institute of Medicine's (IOM) National Roundtable on Health Care Quality documented three types of quality problems: Overuse, underuse, and misuse.[1] These issues were re-emphasized in the IOM's 2001 report Crossing the Quality Chasm which stated that “Many patients, doctors, nurses, and health care leaders are concerned that the care delivered is not, essentially, the care we should receive.[2]” Use of laboratory testing is included in these concerns. The IOM's statement can easily be rephrased as “… the lab tests delivered are not, essentially, the tests we should receive.” Inappropriate use of testing can often lead to false positive results and incidental findings which prompt additional diagnostic studies, invasive procedures, and even unnecessary treatment. Each of these is associated with potential morbidity, mortality, anxiety, and expense, all of which can be harmful to the patient. Laboratory directors, including directors of cytopathology laboratories, therefore have a responsibility to address test utilization as part of their quality programs to ensure that testing is delivered appropriately and only to those who need it. Insurers are also embracing utilization as a measure of quality and changing their payment models to incentivize quality of care, including efficient use of laboratory services. The Federal Government is also discouraging test overuse through the quality measures included in the health information technology (IT) meaningful use program.[3] New payment models are emerging which forego fee for service for new models, such as a bundled payment structure, which incentivize less testing.

Over the years, many professional organizations and other groups have created laboratory practice guidelines (LPGs) to aid providers in appropriate selection of laboratory tests for their patients. LPGs for Pap testing and other cervical cancer screening tests have been developed by several professional societies, and were sometimes in conflict. In 2012, guidelines for cervical cancer screening were developed jointly by the American Cancer Society, American Society for Cervical and Colposcopic Pathology, and the American Society of Clinical Pathology with the participation of many other organizations. These guidelines include no Pap testing in women under age 21 years or over age 65 if the latter has a history of normal Pap tests,[4] and are consistent with guidelines on cervical cancer screening published by the US Preventive Services Task Force in 2012.[5] Both the American College of Obstetricians and Gynecologists (ACOG) and the American Association of Family Physicians included these guidelines for Pap testing in their “choosing wisely” list of evidence-based recommendations that physicians and patients should discuss in order to assist in wise decisions about the most appropriate care. The Choosing Wisely campaign is an initiative of the American Board of Internal Medicine Foundation “to encourage physicians, patients and other stakeholders to think and talk about medical tests and procedures that may be unnecessary or cause harm.[6] Leading specialty societies have been invited to participate in this initiative and contribute a list of tests that physicians and patients should question. To date, over 50 organizations have joined this effort which began in 2012, and the recommendations regarding testing that each organization has provided appear on a posted list on their website.[7]

A recent study funded by the Centers for Disease Control and Prevention found many impediments to provider compliance with LPGs, including lack of awareness, unfamiliarity and perceived difficulty in application, inconvenience, cumbersomeness, confusion, lack of a reminder system, lack of time, and fear of delayed or misdiagnosis leading to lawsuits. They suggest computerized physician order entry (CPOE) as a method to address these impediments.[8] Our study describes a decision-support method within the CPOE of our electronic medical record (EMR) which automates discouraging alerts to guide Pap test ordering for women under age 21 and over age 70 (the ages within the published ACOG guidelines at the time this study was undertaken).[9] We share pilot findings regarding the alerts’ effectiveness in discouraging Pap tests orders in these age groups.

MATERIALS AND METHODS

This project was reviewed and approved by our university's institutional review board. Decision-support rules for interruptive alerts to discourage inappropriate Pap test orders were developed by an outside contractor and the EMR team for our health system. Our health system uses the Epic EMR (Epic, Verona, Wisconsin) which was first implemented in 2002. The Epic EMR is an integrated suite of health care software which combines chart review, order management and documentation for ambulatory and in-patient care, registration and scheduling, as well as systems for laboratory professionals, pharmacists, and radiologists; and billing systems for insurers.

Ages chosen for the Pap test alerts were under age 21 years under and over age 70 years, in accordance with ACOG 2009 cervical cancer screening guidelines when the project was initiated in 2010. These alerts were implemented within the primary care clinics, and displayed when an order was initiated for women < 21 years of age and over age 70 [Figure 1]. There was no “hard stop;” in other words, providers were not prevented from placing an order. Designated physician champions served as liaisons to the providers in the primary care clinics to share development of the alerts and receive feedback from users after implementation.

Figure 1.

Screen shot of interruptive alert discouraging Pap testing in women under age 21 years

Results were measured during four calendar periods: Baseline (prealerts) (July 2010 to June 2011), postalert (alerts on): (July 2011 to December 2011), inadvertent alert turn-off ("glitch") (January 2012 to December 2012), postglitch (alerts re-instated) (January 2013 to July 2013). For each month during these periods. The number of total Pap tests ordered for each age group, is expressed relative to the number ordered in the reference age group (21-71 year olds). The metrics used to measure impact of the alert were between time and period seasonally adjusted relative frequency ratios (ARFR), estimated using multinomial logistic regression models that included fixed effects for month (11 df) to adjust for seasonality. ARFR < 1 are consistent with a reduction in the relative frequency of Pap tests for inappropriate age groups, a beneficial result. Model fit was examined graphically and residuals (ln [observed relative frequency] – ln [fitted relative frequency]) were tested for autocorrelation up to the fourth order using Durbin–Watson statistics. For each age group, pairwise ARFR and 95% confidence intervals were constructed among the following periods: (1) The baseline period (reference category) before alerts were initiated (July, 2010 to June, 2011), (2) A postalert period following initiation of alerts (July, 2011 to December, 2011), (3) A period in which alerts inadvertently stopped functioning (January, 2012 to December, 2012), and (4) A period in which the alerts were re-instated (January, 2013 to July, 2013).

RESULTS

Table 1 summarizes the data from the 4 time periods. The observed and fitted monthly relative frequencies of Pap orders in both age groups during the 4 time periods are illustrated in Figure 2. The multinomial logistic regression model fit the data well and the residuals did not exhibit statistically significant autocorrelation (all four Durbin–Watson P values exceeded 0.20). During the baseline period, 2.7 Pap tests were ordered in patients less than age 21 for every 100 Pap tests ordered for patients ages 21-71. In the postalert period, the relative frequency decreased to 1.7, a large reduction in seasonally adjusted relative frequency (ARFR = 0.58, 95% CI: 0.47, 70). During the period in which the alert inadvertently stopped functioning, the relative frequency in this age group was 1.4 (ARFR = 0.51, 95% CI: 0.43-0.59). When the alert was re-instated, an even greater decline was observed, with the relative frequency reduced to 0.8 (ARFR = 0.28, 95% CI: 0.21, 0.36), a 72% relative reduction in seasonally adjusted monthly relative frequencies compared to the baseline period.

Table 1.

Absolute and standardized relative frequencies of Pap tests by age group and time period

Figure 2.

(a) Observed and fitted monthly relative frequencies of Pap tests ordered in women under age 21 during 4 time periods. (b) Observed and fitted monthly relative frequencies of Pap tests ordered in women over age 70 during 4 time periods

For the over 70-year-old age group, the relative frequency of Pap test ordered during the baseline period was 2.4. In the postalert period, the relative frequency was 2.1 (ARFR = 0.95, 95% CI: 0.79, 1.15). During the period in which the alert inadvertently stopped functioning, the relative frequency of orders in this age group remained stable at 2.0 (ARFF = 0.85, 95% CI: 0.74, 0.98) When the alerts were re-instated, the relative frequency of Pap tests ordered was 1.7 (ARFR = 0.69, 95% CI: 0.57, 0.83), a large reduction from the baseline period.

DISCUSSION

In crossing the quality chasm, the IOM suggests that quality of care could be improved through the creation of an infrastructure to support evidence-based practice, and by facilitating the use of IT.[2] The interruptive alerts that we created to discourage Pap test ordering in women under 21 and over 70 years old are an example of such an IT intervention that supports evidence-based practice. Our study suggests that these alerts are effective in discouraging Pap test ordering in these age groups since substantial decreases in the relative frequency of tests ordered occurred when the alerts were functioning, and a lesser decrease was observed during the interval with nonfunctioning alerts. Though it is possible that growing provider awareness of LPGs contributed at least in part to the decrease in Pap test ordering, the period in which the alerts inadvertently ceased to function provided a fortuitous opportunity to consider this potentially confounding variable. During the nonfunctioning alert period, Pap test orders relative frequencies in the under 21-year-old age group were slightly but not statistically significantly lower than in the previous functioning alert period (ARFR = 0.88, 95% CI: 0.71, 1.10), a reduction which was much less than the decline in orders before or after the nonfunctioning alert period. In the over age 70 age group, there was also no significant decline in Pap test ordering during the nonfunctioning alert period compared to the previous period (ARFR = 0.89, 95% CI: 0.73, 1.08), but monthly percentage of Pap tests ordered in this age group declined when alerts were first initiated and) and once again after the alerts were instated re-instated (alert-re-instated vs. alert-off ARFR = 0.89, 95% CI: 0.66, 0.99). We therefore believe that these interruptive discouraging alerts influenced compliance with guidelines beyond general awareness of LPGs for Pap testing alone.

It is interesting to note that our alerts had a greater influence on Pap test ordering in women under age 21 than in those over age 70. We speculate that providers may be more comfortable with LPGs that delay the start of cervical cancer screening to age 21 than they are with discontinuing screening in older women. Additionally, older women may not be very accepting of discontinuing Pap testing since the importance of this test has been strongly emphasized throughout their lifetime. Older patients may therefore pressure their providers to perform this test, making it difficult for providers to deny this test. A recent survey of obstetrician-gynecologists showed that only about half followed the 2009 ACOG recommendations to begin Pap testing at age 21 and discontinue at age 70. Physicians surveyed expressed concern that patients wouldn’t come for regular exams without receiving a Pap test.[10] The ability of providers to effectively communicate with patients about changes in screening recommendations as well as the importance of having health care visits for other reasons may therefore limit the impact of an alert.

Alerts, as a form of decision-support in CPOE, have been shown to be effective in influencing a variety of physician ordering patterns. There are many published studies on prescribing alerts. These include reports of alert's effectiveness in reducing inappropriate medications for elderly patients in the emergency department and other settings,[11,12,13] reducing prescribing of heavily marketed hypnotic medications in ambulatory care settings,[14] and reducing overall prescribing of antimicrobials in addition to improving the appropriateness of antimicrobial prescribing.[15,16,17] According to a review of the literature on prescribing alerts in the elderly, a wide variety of CPOE interventions show clear potential to reduce inappropriate prescribing and polypharmacy in this population, but none of these have yet been widely adopted.[15] CPOE with decision-support has also shown effectiveness in reducing inappropriate transfusions.[18,19] Relatively few studies, however, have addressed alerts for orders for laboratory tests. Levick et al. showed a 21% decrease in repeat and serial orders in brain natriuretic peptide in intensive care patients after a “soft-stop” e-advisory to the ordering MD.[20] Similarly, unnecessary repeat viral serology tests decreased following implementation of alerts in a cardiovascular surgery clinic.[21] In contrast, alerts within the CPOE system that reminded providers about ordering tests for drug monitoring were not shown to be effective, according to a randomized controlled trial by Palen et al.[22] The literature on decision-support has to be interpreted with caution as many of the interventions described, including our study depend on institution-specific workflows and culture. Furthermore a disproportionate number of reports come from a small group of HIT focused academic institutions.[23]

Alerts as part of CPOE decision-support must be implemented thoughtfully. Alerts can be irritating since they interrupt workflow, create extra steps, and lead to “alert fatigue” in which individuals pay less attention to all alerts due to habituation.[24] Pearson et al. found that the most effective alerts were those that occurred after order selection, as were the alerts in our study.[25] Physicians and nurses responding to alerts as part of CPOE report that they prefer nonintrusive alerts, and that interruptive alerts should be selectively used for high-severity events.[26,27] We were fortunate to have strong clinical champions in the implementation of our project who consulted with clinical users to identify their issues and achieve buy-in as the project developed.

Dedicated support from the EMR team is also important to the success of this type of project. We are fortunate to have a strong health system IT team with a culture of innovation that we could leverage for this project. Our hospital is one of 118 hospitals nationally designated as a “Most Connected Hospital” by US News, and one of 11 hospitals in California designated as “Most Wired” by Hospitals and Health Networks magazine.[28,29] Our health system is a recent recipient of the Nicholas Davies Award which recognizes health care organizations who demonstrate the value of the EMR in supporting patient care, improved patient outcomes, addressing challenges and describing solutions that can be replicated by others.[30] Nonetheless, IT resources are often stretched thin within the institution, and this project was initiated using a contracted outside programmer. This made it difficult for our in-house EMR team to re-trace steps and resolve issues later.

Limitations to this study include the limited sample size and time frame, and the fact that this is a single-site study. We cannot fully assess the influence of national LPG education campaigns on ordering patterns or other system changes on the study effects. The observational nature and single-site of our study limit conclusions that can be drawn relative to a randomized controlled trial. While needed to confirm our findings, such a trial may be difficult to accomplish due to many logistic and economic challenges, including difficulty implementing alerts within different EMR systems at different institutions. As mentioned earlier, we believe that the period in which the alerts ceased functioning provided a comparison period to evaluate the influence of growing provider awareness of LPGs related to Pap testing.

In summary, our study suggests that interruptive alerts can effectively discourage ordering of Pap tests and improve compliance with established LPGs, thus providing the opportunity to avoid unnecessary follow-up tests that can create potential patient harm and unnecessary expense. Longer study intervals are necessary to determine if provider compliance is maintained. Additionally, our experience with Pap test alerts within CPOE provides an effective model worthy of consideration for controlling utilization of other lab tests. We encourage other laboratories and healthcare systems to consider this important method as a part of laboratory quality programs.