Utilization of computerized clinical decision support for potentially inappropriate medications

Abstract

Background: Electronic medical record (EMR) alerts may inform point of care decisions, including the decision to prescribe potentially inappropriate medications (PIM) identified in the Beers criteria. EMR alerts may not be considered relevant or informative in the clinician context, leading to a phenomenon colloquially known as “alert fatigue.”

Objective: To assess the frequency of clinical interaction with EMR alerts and associated deprescribing behaviors in ambulatory settings.

Methods: This is a retrospective observational study in two ambulatory clinics (the Kaye Edmonton Clinic Senior’s Clinic and the Lynnwood Family Practice Clinic) in Edmonton over an observational period of 30 months. Statistical analysis was done using descriptive statistics, chi-square and regression analysis.

Results: The reminder performance for interactions with the alert was 17.2% across the two clinics. The Number Needed to Remind (NNR) or mean number of alerts shown on clinician screens prior to a single interaction of any kind with the alert was 5.8. When actions were defined as a deprescribing (ie discontinuation) event that was related to the alert and that particular interaction in the EMR, the reminder performance was 1.2%, for an NNR of 82.8.

Conclusion: The configuration of alerts in the EMR was not associated with a clinically detectable increase in the uptake of the Beers criteria for high hazard medications.

Introduction

Polypharmacy is conventionally described as a total of five or more prescribed medications at once.¹ Polypharmacy increases the risk of adverse drug events (ADEs), potentially inappropriate medications (PIMs), and hospitalizations.² Efforts to quantify the rates of ADEs have been widely reported and well publicized, although efforts to reduce them have met with limited success.³ A natural starting point to reduce polypharmacy is to discontinue or reduce the dose of potentially inappropriate and minimally effective medications, a process known as deprescribing.⁴ A widely accepted screen for such opportunities is the Beers Criteria, an internationally recognized list of medications that are potentially inappropriate for older adults.^5,6 In the USA, health care expenditure related to PIMs estimated at $7.2 billion in 2001⁷ and in Canada, it is estimated that $419 million is spent on inappropriate medication use.⁸ The Institute for Safe Medication Practices included the Beers criteria in its Ambulatory Care Action Agenda in 2012, suggesting that practitioners use the criteria “to help improve the selection of prescription drugs” in the older population.⁹ Despite risks that PIMs pose to older adults, hazardous prescribing practices continue to exist in ambulatory settings.^10,11 A recent retrospective review of Canadian prescribing practices reported 49% of older adults filled at least one PIM.⁸

Electronic Medical Records (EMRs) can enhance patient safety by presenting relevant information and decision support to front-line prescribers. Computerized physician order entry systems have been shown to markedly decrease prescribing errors, likely by providing standard drug doses and decision support and by eliminating errors from poor handwriting.¹² According to the Agency for Healthcare Research and Quality, a Clinical Decision Support (CDS) provides timely information, usually at the point of care, to help inform decisions about a patient’s care. CDS tools and systems help clinical teams by taking over some routine tasks, warning of potential problems, or providing suggestions for the clinical team and patient to consider.¹³

Most EMRs have a system of pop-up alerts for different purposes. These help with CDS in many areas of medicine including medication management. To mitigate inappropriate prescribing, a computerized decision support (CDS) using a guideline that targets inappropriate medications in older adults (the Beers criteria) was developed in an EMR in our center, and is described elsewhere.¹⁴ This CDS uses alerts known as Best Practice Advisories (BPAs) to direct providers to a navigator where orders management, clinical information and educational materials are available. The most recent Beers criteria is designed to help alert clinicians to therapies that may pose a risk for older adults, and thereby can help them make evidence-informed decisions and enhance patient safety. If clinicians judiciously review and apply the BPAs, there would be a real potential for enhanced use of the Beers criteria and improved patient safety.

Electronic medical record (EMR) alerts are an automated process that continuously analyze EMR data in real time and warn about drug–drug interactions. However, overly frequent and/or non-specific alerts in EMRs are a common source of frustration among clinicians¹⁵ and this was an anticipated barrier. Physicians who are busy or see the same patient frequently may become desensitized to automated alerts and consequently ignore or fail to respond appropriately to such warnings, a phenomenon sometimes known as alert fatigue.^16,17 A perception by clinicians of diminishing returns can result in many, or even most alerts being ignored.

Within some of the ambulatory care settings in Edmonton, Alberta, Canada, an electronic medical record (eCLINICIAN) was introduced in 2008, and included a basic EMR alert feature. BPAs to advise clinicians of PIMs among their patients' medication lists or new orders were introduced in 2014. The effectiveness of alert strategies to address risks of PIMs and the impact of these alerts on physician workflows is not well known. In 1988, Laupacis et al. proposed a measure of clinical benefit intended to capture the value of certain clinical interventions over a period of time – the number needed to treat (NNT),^18–20 calculating the inverse of the absolute risk reduction. In a recent study, a new quality measure called Number Needed to Remind (NNR) was defined, for a single patient, as the mean number of alerts shown on clinician screens prior to a single interaction by the clinician EMR user.²¹ NNR can also be used to evaluate the effectiveness of the CDS to prompt a clinical response by clinicians for PIMs.

Here, we evaluate the relationship between alerts (ie BPAs for PIMs in eCLINICIAN) and the prescriber response and deprescribing over a period of 30 months. This interaction, reported as both reminder performance and NNR, will be compared between a primary care and geriatric specialty clinic. We also report the alerting burden (ie the number of unique alert presentations within different electronic charts). Finally, we will report the reminder performance of medications or classes with the highest alerting burden. We hypothesize that the alert presentations will influence clinician interactions with the alert, but not deprescribing.

Methodology

Design

This is a retrospective observational study in two ambulatory clinics (the Kaye Edmonton Clinic Senior’s Clinic and the Lynnwood Family Practice Clinic) in Edmonton over an observational period of 30 months. The KEC Senior’s Clinic provides specialized geriatrician consultation and the Lynnwood Family Practice Clinic is oriented to primary care. Both clinics use the same EMR system but are geographically separate and not affiliated, and both were separately involved in the development and implementation of the BPA alert feature in 2014 as described previously.¹⁴ Information about the prevalence of inappropriate medications, alert presentations and clinician user responses were obtained from the EMR data. This study was approved by the University of Alberta health research ethics board (Approval: Pro00068219). This is a retrospective observational study and for this study the physicians were not required to provide consent.

Patient population

We include all consecutive encounters (new or follow-up) of adults age 65 or older seen in the two clinics from July 1, 2015 to December 31, 2017, inclusive. Participants were seen by a physician prescriber who belongs to one of the two clinics. We excluded all encounters involving patients age 64 or younger at the time of their visit, encounters that do not involve a physician in one of the two target groups and the few patients seen in both locations.

Prescriber population

All physician prescribers (n=18) eligible to prescribe medications at the two clinics were included as participants in the research study. All physicians in the primary care clinic (n=7) had completed certification in Family Medicine. Among the physicians in the geriatrics clinic, two had completed this same credential, and 9 had completed specialty training in Geriatric Medicine. None of the prescribers was a resident, junior physician, pharmacist, nurse practitioner or physician assistant. Other prescriber characteristics have been previously described.¹⁴ In both clinics, clinician prescribers were at liberty to order medications without interacting with the BPA alerts, and there is currently no process for individual review of prescribing patterns within the clinic. All had completed the mandatory training and demonstrated competency to use the eCLINICIAN system. We excluded BPA interactions, medication entries or electronic orders by any supervised nurse, supervised medical students or medical residents, using eCLINICIAN for patient care.

Computerized provider order entry system

The clinical context and design of the EMR including the provider order entry aspect have been previously reported.¹³ Briefly, the eCLINICIAN system in use in the two clinics participating in the study is based on the 2014 version of the commercially available suite of software applications from Epic Systems Corporation ©. During the study period, computerized provider order entry functions and clinic scheduling functions were available in the Epic Care product. Prescribers charting using Epic Care in the two participating clinics consulted with patients and conducted their own documentation tasks associated with ordering, including managing the BPAs. The BPAs composing the Beers Criteria medication alerts were first active in the two clinics beginning in 2014. In an effort to avoid contamination with data artifacts related to variability in clinician familiarity with the alerts alongside development and training, the study period began on July 1, 2015, after these activities were complete.

In the two clinics, BPAs are generated for any patient when a Beers List medication is entered into the EMR either as part of the medication list or as a new order. The BPA alert is presented prominently on the main page. A prescribing physician would then respond by opening the alert and resolving it. The BPA provides two components. The first component includes the generic drug name, a short description of the possible concern, the level of evidence, a brief recommendation and a link to an evidence synopsis from the 2015 Beers criteria. The second component of the alert is designed to resolve the alert. Response options include “acknowledged”, “safety established”, “no alternatives”, benefit outweighs risk”, “patient acknowledged risk”, “low risk” and “see notes”, which is linked with a text entry box. Finally, to encourage a deprescribing action, the BPA includes direct links to the medication list and the orders section of the EMR.

Data extraction

EMR data related to visit volume, frequency of alert presentations (ie alerting burden) and ordering behaviors were extracted related to the two clinics participating in the study. Visits included in the analysis were in-person encounters marked as complete (ie telephone encounters, and no-show visits were excluded from the analysis).

Number needed to remind calculation

To evaluate reminder performance, we used a definition where the numbers of interactions recorded with an alert (ie the BPA) were divided by the total number of alert presentations for that specific BPA.²¹ In our case, the mean NNR, ie the mean number of presentations of a single alert prior to a prescriber interaction with the alert, was calculated as the reciprocal of the reminder performance. Specifically, NNR was the total number of alert presentations specific to a particular medication in a particular patient presented to the physician user before a response by the user occurred. We used a simple approach to evaluate whether interaction with the alerts influenced prescribing behavior over time. Specifically, over the 30 months of data collected and analyzed, reminder performance was calculated for each calendar month.

Number needed to deprescribe calculation

For the purpose of our study, we include discontinuation events and not dose reduction events as evidence of deprescribing. Similarly, we further evaluated deprescribing activities by quantifying the Number Needed to Deprescribe (NND) or the number of alert presentations specific to a medication and patient presented to a physician user before there was a deprescribing event. For the purpose of our retrospective study, a deprescribing event was defined as documented evidence of complete discontinuation of a medication within the same class as the alert found in the ordering section of the patient chart on the day the alert was presented. Deprescribing events from the patient record potentially related to the alert presentation were identified by locating discontinued medication orders from the eCLINICIAN data warehouse. Reasons for discontinuing the medication order were also filtered to avoid orders indicating a subsequent resumption of the medication (ie discontinue orders related to reorder and refill activities were excluded).

Outcomes

The patient populations (by clinic, total) were described in terms of age (including mean, median, and 25th, 75th percentiles), sex, total number of patients seen, total number of visits and mean number of visits per patient. Visit volumes, alert presentation volumes and reminder performance (both in terms of any interaction and any deprescribing event) were described in total and compared by clinic in quarterly intervals over the study period. Alerting burden and reminder performance was calculated for the top 20 medications that led to alert presentations for each clinic. Finally, for all interactions with the BPA, physician responses by category were described and compared by clinic.

Statistical analysis

Statistical comparisons were performed to identify how patient characteristics varied across the two clinics. To evaluate the extent to which alert burden and alert performance varied across the two clinics, reminder performance (and corresponding NNR) for the two prescriber actions (interaction with alert and discontinue medication) were calculated and compared using chi-square tests. The extent to which alert burden and alert performance changed over time was evaluated using linear regression. Patient characteristics of age and gender were compared with linear regression and chi-square testing, respectively. Statistical analysis was completed using R (version 3.4.3). Alpha was set at 0.05.

Results

Clinical context of PIM prescribing

The median patient age in the geriatrics clinic was significantly higher (81, Inter Quartile Range (IQR): 75–86) than in the family medicine clinic (78, IQR: 68–81, p<2.2×10^–16). As seen in Table 1, females comprise a larger proportion of the patients in the geriatrics clinic (60.6%) compared with the primary care clinic (54.1%; p=0.0002095). The total number of visits was much higher for the primary care clinic compared to the specialty clinic (17,613 vs 3,334). Response pattern to Beers criteria alerts is shown in Table 2.

Table 1.

Baseline characteristics comparing two clinic study subjects.

Patient characteristics	Overall	Primary care	Geriatrics
All patients	7,385	5,713	1,694
Patients excluded (<65 years of age)	4,142	4,056	86
Patients excluded (seen in both clinics)	22
Study population size	3,221	1,635	1,586
Female (n,%)	1,845 (57.3)	884 (54.1)	961 (60.6)
Age (median, IQR)	78 (71–84)	74 (68–81)	81 (75–86)
Diagnoses recorded per patient (median, IQR)	5 (2–10)	9 (5–15)	3 (2–5)
Patients on 5 or more medications (n)	1,678	1,244	434
Patients on 5 or more medications (percent)	52.1	76.1	27.4
Medication orders per patient (median, IQR)	6 (2–15)	13 (6–24)	3 (1–6)
Patients on 1 or more Beers Criteria medications (n)	466	274	192
PIMs per patient (median, IQR)	0 (0–1)	1 (0–2)	0 (0–0)
Visits	20,947	17,613	3,334
Visits per patient (median, IQR)	3 (1–9)	8 (4–15)	2 (1–3)
Visits with alerts (n)	7,429	6,223	1,206
Visits with alerts (%)	35.5	35.3	36.2

Table 2.

Override reasons recorded when interacting with the Beers criteria EMR alert.

Override reasons	Overall		Primary care		Geriatrics
	N	Percent	N	Percent	N	Percent
Alert Information Acknowledged	34	0.41%	1	0.02%	33	2.09%
Benefit Outweighs Risk	147	1.79%	74	1.11%	73	4.62%
Low risk	62	0.75%	52	0.78%	10	0.63%
No Alternatives	19	0.23%	4	0.06%	15	0.95%
Patient Acknowledged Risk	646	7.86%	473	7.12%	173	10.95%
Safety Established	91	1.11%	25	0.38%	66	4.18%
See Notes	172	2.09%	28	0.42%	144	9.11%
No Information Recorded	7051	85.76%	5985	90.11%	1066	67.47%
Total alert presentations	8222		6642		1580

Abbreviation: EMR; electronic medical record.

During the study period, the BPA alerts related to Beers Criteria PIMs were presented to clinicians a total of 8,222 times during 20,947 different patient visits to 18 different prescribing physicians. From the cohort of 3,221 patients included in the study, 7,429 visits included one or more reminder presentations. Trends related to visit volume, alerting burden, reminder performance for alert interactions and deprescribing activities during the study period are shown in Figure 1. The reminder performance across both clinics was 17.3%, which corresponds to an NNR of 5.8. When reminder performance was calculated separately for the two clinics, a statistically significant difference was found (primary care 13.4% vs geriatrics care 37.1%; p<0.05). The NNR in the primary care clinic was 7.4 and NNR in the specialty clinic was 2.7.

Figure 1.

Legend: Patient visits, alert presentations, interaction rates and discontinuation rates over the 30-month study period. (A) Volume of patient care visits at two clinics, primary care (black bars) and a geriatrics clinic (grey bars). Each bar represents a month of data. (B) Volume of alert presentations in the two clinics. (C) Percent of alert presentations with any interaction (reminder performance) in primary care practice clinics (squares), geriatrics clinic reminder performance (circles). (D) Percent of alert presentations associated with a possible deprescribing event in primary care clinic (squares), or geriatrics clinic (circles).

The reminder performance for deprescribing events was predictably lower at 1.2%. There was no statistically significant difference in deprescribing between the clinics (primary care 1.3%, geriatrics clinic 1.0%, p=n.s.). The data extracted from the EMR record suggest a value for Number Needed to Deprescribe of 82, with values for the primary care clinic of 80 and the geriatrics clinic of 96. Put simply, typically 82 alert presentations happened prior to a deprescribing event that could be detected retrospectively. All parameters were compared for changes for each month of the study period. No changes were detected (Volumes of visit F_1,29=0.02, p=1, rates of alerting F_1,29=0.07, p=1, interaction rates F_1,29=0.4529, p=0.981 and medication discontinuing practice rates F_1,29=0.69, p=0.84).

Medication class characteristics

The 20 most common medications and classes of medication orders that led to alert presentations are shown in Table 3. The top two medications or classes that generated the most alert presentations were the same for the primary care clinic and specialty clinic: anticholinergics and zopiclone. The medications or classes with the best reminder performance for BPA interaction in the primary care clinic was naproxen (36.4%), cyclobenzaprine (20.1%) and ketorolac (19.8%), and in the specialty clinic, it was ibuprofen (54.1%), risperidone (52.0%) and naproxen (48.3%). By comparison to reminder performance for any interaction, the reminder performance for deprescribing was much lower for all medications or classes in both clinics. The most frequent alerts in the primary care clinic were naproxen (3.3%), clonazepam (2.8%), amitriptyline (2.5%) and quetiapine (2.5%), while in the specialty clinic they were cyclobenzaprine (5.6%), ibuprofen (5.3%) and amitriptyline (5.2%). The most frequently deprescribed medication classes relative to alert burden included NSAIDs, pain medications, muscle relaxants and anti-psychotic medications.

Table 3.

Alerting burden and reminder performance measured by rates of interaction and discontinuation of alerting rules grouped by medication classes.

Medication classes	Patients	Alert presentations	Alert interactions	D/C events	Interaction rate	D/C rate
	N	N	N	N	Percent	Percent
Nonbenzodiazepine hypnotics	478	2538	353	32	13.91	1.26
Anticholinergics	471	2113	400	12	18.93	0.57
Benzodiazepines	355	2000	313	29	15.65	1.45
NSAIDs	280	839	258	24	30.75	2.86
Endocrine	262	1474	203	8	13.77	0.54
Antipsychotics	183	564	140	12	24.82	2.13
Antiarrhythmics	93	499	60	2	12.02	0.40
Muscle relaxants	82	319	67	7	21.00	2.19
Pain medications	64	289	60	2	20.76	0.69
Gastrointestinal	58	268	30	5	11.19	1.87
Alpha blockers	31	126	31	1	24.60	0.79
Antiparkinson agents	5	47	5	0	10.64	0.00
Antispasmodics	5	31	2	0	6.45	0.00
Antithrombotics	4	6	0	0	0.00	0.00
Barbiturates	4	11	3	0	27.27	0.00
Antiepileptics	1	1	0	0	0.00	0.00
Total	2376	11,125	1925	134	17.3	1.20

Abbreviation: D/C; discontinuation.

Discussion

Incorporating alerts into EMRs is intended to support clinicians in recognizing and acting on PIMs, but in two very different ambulatory settings, physician interaction with the alerts occurred in a minority of alert presentations, and the downstream impact on deprescribing was quite small, regardless of medication class. The number of clinic visits and alerts was much higher in the primary care clinic, and reminder performance for physician interaction was higher in the specialty clinic. There appeared to be no improvement in reminder performance for physician interaction in either clinic over the time of the study, which may indicate other factors beside clinician awareness are important for experienced physicians. By comparison, reminder performance for deprescribing was comparably low in both clinics but did improve over time. The overall value of NNR for interactions with the reminders was 5.8. This value varied from 2.7 for the geriatrics clinic to 7.4 for the primary care clinic. This may indicate that geriatrics prescribers found the reminders more useful in their clinics, or that they had more time to participate in the BPA interaction. The Number Needed to Deprescribe was more similar between groups, but much higher than the NNR. The low reminder performance for both interaction and deprescribing was evident across all medication classes in both settings.

Several important caveats should be considered around these data when deciding if these values are relevant to other clinical settings or CDS applications. First, it is possible that the deprescribing rate was underestimated. During the study period, not all instructions by physicians to their patients are necessarily captured electronically using eCLINICIAN. For example, physicians in the specialty clinic may function more as consultants, making a recommendation to primary care physicians to deprescribe medications rather than doing it themselves, which is not captured in this study. Electronic orders are printed and provided to patients to fill manually at a community pharmacy. Prescribers may have prescribed or deprescribed other medications outside eCLINICIAN/EpicCare, which may result in under-reporting of PIMs or of reminder performance for deprescribing, respectively. Prescribers may advise patients to discontinue medications without recording this in the medication record. Further, we note that a more broad definition of deprescribing would include changes in medication by dose or within a class. Due to the limitations of our dataset, we have applied a more narrow definition of deprescribing as discontinuation only. In the future, the adoption of e-prescribing technology and future versions of technologies related to the eCLINICIAN alert technology may help to better reflect the true rate of deprescribing of PIMs.

Second, the retrospective cross-sectional design complicates the comparison of clinics with very different prescriber characteristics, populations, processes and remuneration patterns. These differences make it difficult to make inferences about the relative impact of the common EMR alert strategy on physician/alert interactions, deprescribing behavior and the influence of specific medication classes.

A potential source of bias resulted from the choice to use alert presentations, rather than patients as a unit of analysis for the study. The rate of alerting and total burden of alerts may be inflated in the family physician clinic compared to the geriatrics clinic. That may be due in part to the higher frequency of visits by patients who are taking PIMs in the family clinic. The reported lower rates of reminder performance may be partly the result of this epiphenomenon. We chose this method of analysis to track the clinician experience with alert fatigue which can be aggravated by this epiphenomenon. It may be important for system designers to take this aspect of clinician practice into account when deciding to configure content such as the Beers criteria to present alongside clinical visits as it may contribute to alert fatigue.

Another caveat is that there was no prior attempt to set documentation standards in relation to BPAs. As such, the low reminder performance for interaction may reflect factors other than alert fatigue such as uncertainty of expectations, or a calculated decision to conserve cognitive focus for other tasks deemed to be more important. We anticipate that future work to deploy alerts such as these BPAs would be accompanied by clearer documentation standards, including more robust rationale for deciding to override an alert. The typical explanation by clinicians for low reminder performance is alert fatigue. However, we did not see an expected pattern of worsening reminder performance over time, as expected for alert fatigue. We felt that the thirty-month period in our study was a reasonable time to evaluate for alert fatigue. To begin to address clinician reports of alert fatigue more effectively, and to better understand how to use medication alerts in an EMR, developing valid quantitative measures of alert effectiveness that are grounded in clinical experience and parallel implementation science techniques are needed. Re-engagement of the user group through a process of quality improvement to develop a mutually invested goal, explore root causes for disengagement with the BPAs, implement measures to improve and continually track improvement would significantly raise confidence in our interpretation of reminder performance data. Clinicians may not entirely agree with all the criteria listed in the Beers Criteria and the applicability to their specific patient, which is another factor potentially impacting reminder performance. It is noted that the Beers Criteria should be considered when individualizing therapy, but are not designed to be rigid rules for prescribing.

For example, the involvement of other allied health professionals such as pharmacists may have contributed to better deprescribing practices. Also, deprescribing is a multistep process, ensuring discussion about patient values and goals, agreement to stop a medication, then a tapering process in some cases. Ideally, a deprescribing study should be viewed over a longer period of time, as we did in this study, but the complexity of patient-centered decision-making was difficult to capture in this study. Some medications such as benzodiazepines cannot be stopped abruptly, so appropriate efforts to slowly taper rather than a hazardously discontinue would not be captured in our study.

A 2013 systematic review of information technology interventions aimed at improving medication safety found that CDS interventions have been used to significantly reduce the use of PIMs in the elderly and pregnant populations.¹¹ Raebel et al. studied the use of a computerized tool to alert pharmacists when patients were prescribed a PIM. Over the one-year trial, 543 (1.8%) intervention-group patients vs 644 (2.2%) control-group patients were prescribed PIMs (P=0.002).²²

Alert pattern in an EMR and the response by physicians over time provides information of the possibility of alert fatigue in physicians who care for an older population at risk for polypharmacy. We hope these data will inform future EMR-based strategies to minimize alert fatigue, and thus make the reduction of polypharmacy and ADEs in the elderly more effective and sustainable. More specifically, we anticipate that the results of this study will help to increase alert specificity by reducing or eliminating clinically inconsequential alerts and make only high-level alerts interruptive. Knowledge of appropriate and specifically identified high-risk alert helps to improve time management and workflow coordination in EMR.

To our knowledge, it is the first study that directly compares the impact of an electronic alert on recognition of inappropriate medications using standardized criteria for high-risk medications in the elderly such as the Beers criteria. In relation to the impact on reminder performance, the impact of patient populations, prescriber populations, training or educational strategies or quality assurance techniques should be tested. Qualitative research that uses focus groups could define novel approaches to improve deprescribing and explore user experience with alerts.

Conclusions

The use of an electronic alert in individuals at both specialty and primary care clinic EMRs was not associated with a significantly increased uptake of Beers criteria for inappropriate medication use. If electronic reminders are tested to improve deprescribing behavior by physicians and minimize PIMs in vulnerable older adults, then research including quality improvement will be needed to enhance reminder performance. Because of the potential for broad impact at low cost, reminders should be further studied to integrate their use to address the growing problem of PIM use.

What is already known on this topic

• Clinical decision support (CDS) has the potential to influence clinician behavior at the point of care, including the decision to prescribe or continue orders for potentially inappropriate medications (PIMs).

• Different barriers exist to effective CDS, including alert fatigue.

• Some measures of CDS effectiveness have been initially defined including Number Needed to Remind (NNR).

What this study adds to our knowledge

• Effective measurement of CDS performance should include informative process measures of clinician behavior, including interaction with alerts and documented decisions to deprescribe PIMs.

• Future valid CDS performance measures such as the Number Needed to Deprescribe would be supported by e-prescribing systems and effective clinician use.

Glossary

• Polypharmacy – total of five or more concurrently prescribed medications.

• PIM – Potentially Inappropriate Medications – medications prescribed without a valid indication or with a contraindication, where risks outweigh the benefits where there is an increased risk of adverse drug reactions, or when a safer alternative is available.

• EMR – Electronic Medical Record – the digital version of the patient medical record.

• Computerized Physician Order Entry – an application that enables providers to enter medical orders into a computer system.

• BPA – Best Practice Advisory – an EMR alert presented to health care professionals that provides patient-specific information, intelligently filtered, triggered for pre-determined situations to help improve the delivery of care.

• CDS – Clinical Decision Support – a health information technology that analyzes data to help health care professionals make decisions and improve patient care.

• Navigator – A series of sections in an electronic medical record meant to follow a particular workflow such as an office visit or medication reconciliation.

• Orders management – a computer system that provides entry and storage of orders for prescriptions, tests and other services in order to enhance legibility, reduce duplication and improve the speed with which orders are executed.

• Reminder performance – the result of the numbers of alert interactions recorded with an alert (ie the BPA) divided by the total number of alert presentations.

• Number Needed to Remind (NNR) – the mean number of alert presentations specific to a medication and patient presented to a health professional user before there was an interaction with the alert or a deprescribing event.

• Number Needed to Deprescribe (NND) – the mean number of alert presentations specific to a medication and patient presented to a health professional user before there was a deprescribing event.

• Alerting burden – the frequency of alert presentations.

• Alert presentation – a previously designed patient-specific electronic message that is inserted based on pre-set triggers into the workflow of a health care professional in the context of patient care.

• Alert interaction – Any electronic response by the health care professional to an alert presentation in the context of patient care.

• Electronic Medical Record (EMR) Alerts – an alert presentation in an EMR.

• Deprescribing event – evidence of complete discontinuation of a medication within the same class as the alert as found in the ordering section of the correct patient chart on the day the alert was presented.

• Alert Fatigue – a phenomenon in which health professionals who are busy or see the same patient frequently may become desensitized to automated alerts and consequently ignore or fail to respond appropriately to such warnings.

Disclosure

Dr Allen Ausford reports consulting fees for clinical development work on the province of Alberta HIE(netcare)and CIS(eCLINICIAN- the ambulatory program that was used in this study) from Alberta Health Services, during the conduct of the study. Dr Jacques Romney received grants from University of Alberta eClinician EMR Innovation Grant, during the conduct of the study. The authors report no other conflicts of interest in this work.