Summary
Objective : To determine the impact of the introduction of new pre-written orders for antimicrobials in a computerized provider order entry (CPOE) system on 1) accuracy of documented indications for antimicrobials in the CPOE system, 2) appropriateness of antimicrobial prescribing, and 3) compliance with the hospital’s antimicrobial policy. Prescriber opinions of the new decision support were also explored to determine why the redesign was effective or ineffective in altering prescribing practices.
Methods : The study comprised two parts: a controlled pre-post study and qualitative interviews. The intervention involved the redesign of pre-written orders for half the antimicrobials so that approved indications were incorporated into pre-written orders. 555 antimicrobials prescribed before (September – October, 2013) and 534 antimicrobials prescribed after (March – April, 2015) the intervention on all general wards of a hospital were audited by study pharmacists. Eleven prescribers participated in semi-structured interviews.
Results : Redesign of computerized decision support did not result in more appropriate or compliant antimicrobial prescribing, nor did it improve accuracy of indication documentation in the CPOE system (Intervention antimicrobials: appropriateness 49% vs. 50%; compliance 44% vs. 42%; accuracy 58% vs. 38%; all p>0.05). Via our interviews with prescribers we identified five main reasons for this, primarily that indications entered into the CPOE system were not monitored or followed-up, and that the antimicrobial approval process did not align well with prescriber workflow.
Conclusion : Redesign of pre-written orders to incorporate appropriate indications did not improve antimicrobial prescribing. Workarounds are likely when compliance with hospital policy creates additional work for prescribers or when system usability is poor. Implementation of IT, in the absence of support or follow-up, is unlikely to achieve all anticipated benefits.
Citation: Baysari MT, Del Gigante J, Moran M, Sandaradura I, Li L, Richardson KL, Sandhu A, Lehnbom EC, Westbrook JI, Day RO. Redesign of computerized decision support to improve antimicrobial prescribing. Appl Clin Inform 2017; 8: 949–963 https://doi.org/10.4338/ACI2017042017040069
Keywords: Clinical decision support, Alerting, Order entry, Medication management, Hospital information systems
1. Background and Significance
Inappropriate prescribing of antimicrobials is a key contributor to antimicrobial resistance. Inappropriate prescribing includes antimicrobials which are prescribed when not needed; in doses, frequencies, routes or for durations that are not appropriate; or prescription of a broad spectrum antimicrobial when a more appropriate narrow spectrum alternative is available [ 1 ]. In Australia and internationally, studies have shown that up to 50% of antimicrobials are prescribed inappropriately [ 2 – 4 ]. Many interventions designed to improve antimicrobial prescribing in hospitals have been trialed with varying levels of success [ 5 , 6 ]. A recent systematic review and meta-analysis showed that computerized decision support embedded in computerized provider order entry (CPOE) systems can significantly increase appropriate prescribing of antimicrobials [ 7 ].
A common form of decision support in CPOE is computerized alerts. Alerts are generated at the point of prescribing to warn clinicians of potentially inappropriate or unsafe orders. Some studies have shown that when well designed and targeted, computerized alerts can be effective in improving antimicrobial prescribing. For example, in a hospital in Iran, a rule-based clinical decision support system examined the dose and dosing interval of antimicrobials, and generated an alert if these were not within the “normal” range. This alert resulted in a significant reduction in dosing errors from 38% to 22% and a reduction in dosing interval errors from 25% to 20% [ 8 ].
In addition to reports of positive impact, there are an increasing number of studies demonstrating that computerized alerts are frequently ignored by clinicians. Studies have found that clinicians override (i.e. click past) 49%–96% of drug alerts encountered [ 9 ]. Alert fatigue, now a well-established phenomenon, results from the excessive generation of computerized alerts [ 10 ] and has been identified as the primary reason for alerts being ignored and overridden by users [ 9 , 11 ]. A less interruptive and so often more effective form of decision support in CPOE is pre-written orders. Prewritten orders eliminate the requirement for prescribers to select/enter components of an order (e.g. dose, frequency) as they are pre-populated with recommended values. Unlike alerts that detect potential errors following order entry, pre-written orders guide clinicians in making appropriate selections during the ordering process. Research has shown that pre-written orders for antimicrobials in CPOE can also be effective in improving antimicrobial prescribing. For example, a number of studies have shown that weight-based pre-written orders for vancomycin improve vancomycin dosing [ 12 , 13 ]. Introduction of order sets [groups of pre-written orders] have also been shown to improve antimicrobial use for severe sepsis [ 14 ], improve antimicrobial use in critically ill children [ 15 ], and increase timely discontinuation of post-operative antimicrobials [ 16 ].
At our study hospital, antimicrobial formulary restrictions were communicated to prescribers via the presentation of a computerized alert in a CPOE system. However, previous observational studies undertaken at the hospital revealed that prescribers were being presented with a large number of computerized alerts, the majority of which were not read by doctors [ 17 ]. In an attempt to minimize dependence on computerized alerts to inform prescribers of antimicrobial policy recommendations, a series of new pre-written orders for restricted antimicrobials were developed.
2. Objectives
In this study, we set out to determine the impact of this decision support redesign on antimicrobial prescribing. We examined the impact of the implementation of new pre-written orders on three outcome measures:
accuracy of antimicrobial indications documented by prescribers in the CPOE system,
appropriateness of antimicrobial prescribing, and
compliance with the hospital’s antimicrobial policy.
We also aimed to explore prescriber opinions of the new decision support to determine why the change was effective or ineffective in altering prescribing practices.
3. Methods
3.1. Study site and antimicrobial policy
The study was conducted at a teaching hospital with 320 beds in Sydney, Australia. Hospital policy restricted the use of certain antimicrobials based on recommendations made in the Australian Therapeutic Guidelines: Antibiotic [ 1 ], local resistance patterns and locally endorsed unit-specific protocols. All antimicrobials were classified according to a ‘traffic light system’, as shown in Table 1 . ‘Green’ antimicrobials were not restricted in their prescription and did not require approval for use. All ‘red’ antimicrobials required approval. ‘Orange’ antimicrobials needed approval where use was outside pre-specified indications. This study focused on only orange antimicrobials as previous audits at the hospital revealed that compliance with the policy for orange antimicrobials was very poor [ 18 ].
Table 1.
Hospital antimicrobial policy
| Antimicrobial | Description | Examples |
|---|---|---|
| Red | These antimicrobials required antimicrobial stewardship doctor approval before they could be prescribed | Daptomycin Linezolid |
| Orange | These antimicrobials could be used without approval for specific indications but required antimicrobial stewardship doctor approval if prescribed outside of this pre-defined list | Azithromycin IV is pre-approved for severe CAP * (in combination with IV ceftriaxone or cefotaxime) or for severe sexually-acquired pelvic inflammatory disease |
| Green | These antimicrobials were not restricted | Flucloxacillin Cephalexin |
* CAP=Community Acquired Pneumonia; IV=Intravenous
3.2. Computerized provider order entry (CPOE) system and decision support
The CPOE for medication management (CSC, MedChart®) allows electronic prescribing, pharmacy review, and medication administration. Most medications, including antimicrobials, are available to prescribers as pre-written orders. That is, prescribers select an order that is pre-populated with recommended values. The medication management CPOE system interfaces with other hospital clinical information systems, including a CPOE for ordering and reporting of laboratory and imaging tests, paging, rostering and clinical documentation. Patient progress notes were not electronic.
Prior to decision support redesign, MedChart® informed prescribers of antimicrobial restrictions via the presentation of a computerized alert at the point of prescribing. Prescribers were not able to move past the alert screen without entering text into the comment field of the alert. As shown in ► Figure 1 , prescribers wishing to order an orange antimicrobial entered the number corresponding to the clinical indication in a comment field at the bottom of the alert screen. If they wished to prescribe the antimicrobial for a non-approved indication, they were instructed to input the indication into the comment field and contact the antimicrobial stewardship doctor to gain approval. The number of indications listed in an alert was dependent upon the antimicrobial and ranged from one to fourteen approved indications. For example, as shown in Figure 1 , vancomycin is pre-approved for five indications.
Fig. 1.
Computerized alert that triggers when prescribers select vancomycin. Doctors must enter the number corresponding to the indication in the alert comment field.
The intervention: Half the orange antimicrobials were randomly selected to be ‘Intervention antimicrobials’ (► Table 2 ). For these intervention antimicrobials, decision support was redesigned so that approved indications were incorporated into pre-written orders. In this way, prescribers were not required to read alert text to determine what indications were pre-approved. Instead, prescribers were required to select an antimicrobial order-indication combination (e.g. Azithromycin 1g Oral regularly every 7 days for 2 doses – mild-moderate urethritis; Azithromycin 250mg Oral once daily – pulmonary mycobacterium avium complex [MAC]) from the list of pre-written orders within the CPOE system (► Figure 2 ). Lists were ordered based on route of administration. The number of prewritten orders available for selection was dependent upon the antimicrobial and ranged from one to 20 pre-written orders. Note that the unit of randomization for this study was antimicrobial, independent of the number of orders prescribed of each antimicrobial.
Table 2.
Orange antimicrobials were randomized to control or intervention
| Control antimicrobials | Intervention anti- microbials |
|---|---|
| Aciclovir | Amphotericin Nebulised |
| Cefepime | Azithromycin |
| Ceftazidime | Cefotaxime |
| Gentamicin | Ceftriaxone |
| Itraconazole | Ciprofloxacin |
| Moxifloxacin | Clarithromycin |
| Control antimicrobials | Intervention anti- microbials |
| Piperacillin/tazobactam | Fluconazole |
| Terbinafine Oral | Foscarnet |
| Tobramycin | Norfloxacin |
| Valaciclovir | Oseltamivir |
| Vancomycin | Ribavirin |
| Voriconazole | Rifampicin |
| – | Sodium Fusidate |
Fig. 2.
Pre-written orders for azithromycin. These orders incorporate approved indications.
Opportunities to incorporate decision support into the CPOE (at the local level) were limited and the decision to incorporate indications into pre-written orders was based on the availability of this functionality within the system. Decision support was redesigned by the hospital’s electronic medication management pharmacists (those with expertise in the operation of the prescribing functionality) and was approved by the hospital’s Antimicrobial Stewardship Committee, comprising doctors, both senior and junior, and pharmacists. Accompanying the change to the system, flyers were posted around the hospital and antimicrobial stewardship staff, pharmacists and prescribers received training on the two possible mechanisms for recording antimicrobial indications in the CPOE. This training was delivered by the Project Lead pharmacist and comprised a 15-minute demonstration of the two methods, including how to select and record indications, as well as practical examples. It is important to note that pre-written orders were not unique to antimicrobials and all doctors were familiar with using this method of prescribing for other medications.
3.3. Study design
This study comprised two parts
a controlled pre-post study (► Figure 3 ) and
qualitative interviews.
Fig. 3.
Study design and numbers of antimicrobials (abx) audited in Part 1.
As described above, decision support was redesigned for only half the antimicrobials (the ‘Intervention antimicrobials’). No change was made to ‘Control antimicrobials’.
3.4. Procedure
Relevant details of orange antimicrobials prescribed on any ward in the hospital before (September – October, 2013) and after (March – April, 2015) the intervention were extracted from the CPOE system. This included information related to the antimicrobial (e.g. drug, dose, frequency, and route), indication for use [as recorded in the alert comment field or pre-written order], prescriber details (name, specialty and prescriber level) and patient details [medical record number, ward and age]. Two experienced clinical pharmacists then reviewed patient progress notes to determine the indication for use of each antimicrobial in our sample based upon the clinical information presented in each patient’s record.
3.4.1. Determining accuracy of indications documented in the CPOE system
To determine accuracy of indications documented in the CPOE, we adopted a similar approach to previous studies [ 19 – 21 ] and compared antimicrobial indications recorded in the CPOE to indications evident in patient progress notes. If concordant, the indication documented in the CPOE was deemed to be accurate. If no indication was documented in the CPOE or in patient progress notes, accuracy was classified as ‘un-assessable’.
3.4.2. Determining appropriateness and compliance of antimicrobial use
The indication recorded in patient progress notes was viewed to be the gold standard against which appropriateness and compliance was assessed. To determine whether the antimicrobial was appropriate, the Australian Therapeutic Guidelines: Antibiotic [ 1 ] and locally endorsed guidelines were consulted. Antimicrobial orders were classified as appropriate, suboptimal (if antimicrobial was appropriate, however dose or frequency were not optimal), inappropriate or un-assessable (if no indication was documented in both the CPOE and in patient progress notes). To determine whether compliant with the hospital policy, pharmacists first established whether the antimicrobial had been prescribed for a pre-approved indication, and if not, whether approval from the antimicrobial stewardship doctor had been sought. If either of these conditions were met, the prescription was designated ‘compliant’. If no indication was documented in the CPOE or in patient notes, compliance was classified as ‘un-assessable’. Cases where it proved difficult to determine appropriateness or compliance were discussed with a clinical microbiologist.
Prior to commencement of formal data collection, inter-rater reliability (IRR) was undertaken to ensure the two study pharmacists were consistent in their assessments of appropriateness. This involved pharmacists independently reviewing the same antimicrobials and making an assessment of appropriateness. The two pharmacists and clinical microbiologist then came together to discuss inconsistent cases. Three rounds of IRR were undertaken with pharmacists, where 12, 35 and 26 antimicrobials were independently reviewed. In the last round of IRR, the pharmacists achieved 81% agreement (kappa =0.69).
3.4.3. Exploring prescriber opinions of decision support
Prescribers were opportunistically recruited via direct approach (i.e. were directly approached while on the wards) and invited to participate in a short semi-structured interview. Junior doctors were targeted for recruitment as previous research indicated that senior doctors are less likely to use the CPOE system [ 17 ]. All prescribers who were invited agreed to take part. Interview questions were developed with input from a clinical pharmacologist, pharmacist and a human factors researcher, and were piloted with a junior prescriber prior to being finalized. Prescribers were asked to
describe the two ways of documenting indications in the CPOE (i.e. in the comment field of alerts and in pre-written orders),
explain which approach they preferred and why, and
suggest possible reasons for why indications recorded in the CPOE may not always be concordant with the indications evident in patient notes.
The interviews were audiotaped and transcribed. The de-identified content was analyzed by three investigators, two pharmacists and a human factors researcher, to identify factors contributing to inaccurate documentation of indications and non-compliant or inappropriate antimicrobial use. A general inductive approach [ 22 ] was used for analysis, whereby contributing factors were extracted with no a-priori framework or model to guide analysis. This involved reviewers reading each statement made by participants and coding each as a contributing factor, if relevant. Investigators came together periodically to compare factors they had identified. Any disagreements in contributing factors were resolved via discussion. Regular meetings also allowed researchers to determine when theme saturation had been achieved. Interviews were continued until no new themes were apparent, which is an appropriate and standard approach for qualitative research [ 23 ]. This occurred following eleven interviews with doctors. Doctors included one senior doctor (an anesthetist) and ten junior doctors (interns, residents and registrars) who rotated through various specialties.
3.5. Statistical analysis
The analyses were conducted for our three main outcomes: concordance, appropriateness, and compliance. For the descriptive statistics, we calculated the proportions of concordant, appropriate, and compliant prescribing by study period and intervention group. Multivariate logistic regression models were applied to examine the impact of the intervention on the three outcomes (interaction term of study period and study group), and then adjusted for prescriber specialty (34 specialties), prescriber level (3 levels) and the number of indication options available for selection. In the regression model for appropriateness, the suboptimal category was grouped with the inappropriate category. The significance level was set at 5% and SAS version 9.4 was used for data management and analyses.
4. Results
4.1. Accuracy of indications documented in the CPOE system
Overall, prior to decision support redesign, accurate indications were documented in the CPOE system for 63% of assessable antimicrobial prescriptions (► Table 3 : 80% (60/75) for the control group; 58% (150/257) for the intervention group; Total: 210/332). Although antimicrobials were randomized to control or intervention group, the groups differed at baseline with respect to accuracy of indication documentation – 58% of intervention antimicrobials and 80% of control antimicrobials included an accurate indication.
Table 3.
Number (percentage) of control and intervention antimicrobial orders where the indication in the CPOE was concordant and non-concordant with the indication in patient progress notes in the pre and post periods
| Control antimicrobials | Intervention antimicrobials | |||
|---|---|---|---|---|
| Pre period | Post period | Pre period | Post period | |
| Concordant (i.e. accurate) | 60 (80%) | 59 (77%) | 150 (58%) | 106 (38%) |
| Non-concordant (i.e. inaccurate) | 15 (20%) | 18 (23%) | 107 (42%) | 175 (62%) |
| Total | 75 | 77 | 257 | 281 |
Redesign of decision support had no impact on accuracy of indications documented in the CPOE system. No significant difference was found between intervention and control antimicrobials in the proportion of antimicrobials with concordant indications across the study periods (p=0.1). Adjusting for prescriber specialty, prescriber level and the number of indications available for selection using the logistic regression model, did not alter this result (p=0.1).
Interestingly, from the logistic regression model, each additional indication available for selection in alerts or pre-written orders was associated with a 9% (95% CI: 6%-13%) decrease in the odds of an antimicrobial indication recorded in the CPOE being accurate (p<0.0001).
In a large number of cases, concordance between the indication evident in patient progress notes and the indication recorded in the CPOE system was un-assessable (399/1089, 37% of all antimicrobial orders). This was because no indication was recorded in the CPOE (n=290), or no indication was documented in patient progress notes (n=141). In some of these cases, there was no indication documented in both the CPOE and progress notes. In instances where prescribers failed to record or select an indication in the CPOE, they entered largely un-interpretable (e.g. “ICU”), nonsensical text (e.g. “fsdf”) into the comment field of the alert, or manually removed/edited the indication in prewritten orders.
4.2. Appropriateness of antimicrobial prescribing
Prior to decision support redesign, 55% (► Table IV : [100+191]/[143+388] = 291/531) of assessable antimicrobials were identified to be appropriate. There was a difference at baseline between groups, with 49% (191/388) of intervention antimicrobials and 70% (100/143) of control antimicrobials classified as appropriate in the pre period.
Table 4.
Number (percentage) of control and intervention antimicrobial orders that were appropriate, suboptimal and inappropriate in the pre and post periods
| Control antimicrobials | Intervention antimicrobials | |||
|---|---|---|---|---|
| Pre period | Post period | Pre period | Post period | |
| Appropriate | 100 (70%) | 80 (67%) | 191 (49%) | 205 (50%) |
| Suboptimal | 23 (16%) | 29 (24%) | 99 (26%) | 107 (26%) |
| Inappropriate | 29 (14%) | 10 (8%) | 98 (26%) | 94 (23%) |
| Total | 143 | 119 | 388 | 406 |
Redesign of decision support had no impact on appropriateness of antimicrobial prescribing (► Table 4 ). Despite the differences at baseline, no significant difference was found between intervention and control antimicrobials in appropriateness across the two time periods (p=0.6). Adjusting for prescriber specialty and the number of indication options available for selection did not alter this result (p=0.6). Each additional indication available for selection(in alerts and in pre-written orders) was associated with an 8% (95% CI: 5%–11%) decrease in the odds of an antimicrobial being appropriate (p<0.0001).
4.3. Compliance to the hospital antimicrobial policy
Prior to decision support redesign, 48% (► Table 5 : [87+171/[144+388] = 258/532) of assessable antimicrobials were compliant with the hospital’s antimicrobial prescribing policy. There was a difference at baseline between groups, with 44% (171/388) of intervention antimicrobials and 60% (87/144) of control antimicrobials found to be compliant with the policy.
Table 5.
Number (percentage) of control and intervention antimicrobial orders that were compliant and non-compliant with the hospital’s antimicrobial prescribing policy in the pre and post periods
| Control antimicrobials | Intervention antimicrobials | |||
|---|---|---|---|---|
| Pre period | Post period | Pre period | Post period | |
| Compliant | 87 (60%) | 72 (61%) | 171 (44%) | 171 (42%) |
| Non-compliant | 57 (40%) | 47 (40%) | 217 (56%) | 235 (58%) |
| Total | 144 | 119 | 388 | 406 |
As was found with accuracy and appropriateness, compliance with the hospital policy was not influenced by redesign of decision support (► Table 5 ). Despite the differences at baseline, no significant difference was found between intervention and control antimicrobials in compliance across the two study periods (p=0.8). Adjusting for prescriber specialty and the number of indication options available for selection did not alter this result (p=0.54). Each additional indication available for selection (in alerts and in pre-written orders) was associated with a 7% (95% CI: 5–10%) decrease in the odds of an antimicrobial being compliant with the hospital policy (p<0.0001).
4.4. Clinicians’ views about decision support redesign
Via our interviews with prescribers we identified five main factors that contributed to inaccurate documentation of indications in the CPOE, non-compliance to hospital policy and inappropriate antimicrobial use.
Dose and frequency took priority over indication
Regardless of the method used to record an indication in the CPOE, it became immediately apparent that prescribers did not view documenting an indication for antimicrobial use as a priority.
They are still getting the right dose, it’s still the right thing, it’s still on the computer, I guess what difference does it make. (D6)
Prescribers explained that when selecting a pre-written order from the list available, they searched for an appropriate dose and frequency of an antibiotic, not the indication. Identifying an order with an appropriate dose was viewed as more important than selecting the correct indication.
I think lots of people, they just like choose the same dose and frequency that they’re after and they just click it, regardless of what the qualifier [indication] says. (D1)
From anecdotal evidence I know that a lot of people will just click ceftriaxone regardless of what quicklist [pre-written order] it is… for example they’ll just look at dosages rather than the indication and say yes I want the 1g daily for 7 days, I’ll click that one. (D2)
Long lists of pre-written orders facilitated errors in selection
Prescribers were generally very positive about pre-written orders but explained that in some cases there were large numbers of orders available for a single antimicrobial and this led to unintentionally selecting the wrong order.
Um, sometimes there are a lot of options and so I don’t think I’ve personally done it but I know my colleagues have accidently clicked the wrong dose just because there are a million different regimens or dosages... I’d hate to see the quicklists [pre-written orders] go; perhaps you could take out a few. (D7)
For example, ceftriaxone, there is an enormous list of quicklists [pre-written orders] and you sort of have to scroll through to find the one that you want. (D2)
Lack of monitoring of indications entered into the CPOE system
One of the most common beliefs expressed by prescribers was that information recorded in the CPOE system was not being read or used. Therefore, there were no consequences for individual pre-scribers when they recorded an incorrect indication.
The pharmacy is not going to be any wiser to the fact that the indication is wrong and it is not the indication. (D5)
You wonder who checks them, with some of those pop ups, I know they are not checked or I think they are not checked and so it is just irrelevant. (D11)
Doctors explained that the CPOE system and the approval process in general was easy to override.
No one ever follows up and no one ever gets pulled up. (D1)
By doing a pre-approved indication, even though it is false, they are not going to get chased up to investigate it further. (D5)
The system lets you move, click forward even if you haven’t given an appropriate indication or any indication at all- you just have to hit a letter. (D11)
Antimicrobial approval process was time consuming and poorly integrated
Several participants explained that seeking approval for non pre-approved indications was time consuming and interrupted their workflow. When time pressured and rushed, prescribers often entered or selected an approved indication, even if not completely accurate. This removed the requirement for prescribers to step away from the prescribing process to contact the antimicrobial stewardship doctor and saved time. Many doctors reported choosing indications that were partly consistent with what they intended to prescribe the antimicrobial for.
I reckon maybe, if they are in a really big hurry and they just want to prescribe a drug and it fits the criteria in terms of dose and time but might not fit the actual indication, so people might just take that short cut if there is a long list. (D6)
But I know some of my colleagues will just type something in which won’t make any sense. They just type a few keys just to proceed to the next level…I don’t think that that is done to challenge the system, it is just done so we can give the patients the right antibiotics. Cause you know we are never going to chart something that we don’t think is appropriate but…I guess you could say that the on-line system could be viewed as a barrier. (D7)
Pressure from senior doctors to prescribe without obtaining approval
Related to time pressure, junior doctors reported that they felt pressure from more senior doctors to prescribe antimicrobials without gaining approval from the antimicrobial stewardship doctor. Junior doctors recorded inaccurate indications to bypass the approval process and to avoid being reprimanded by consultants. Several junior doctors also reported feeling intimidated by more senior staff.
As a junior doctor you do try to game the system and do what you are told to do and the less number of people trying to chase you up, making sure your prescribing is appropriate...makes life easier. (D5) It’s a physician telling you to do it and as a junior doctor you are not going to say well maybe that’s not the correct indication, arrhh whether it is wrong or right…you’ve got to do it regardless…because an expert in the field supposedly has told you to do it. (D7)
5. Discussion
We found that redesign of computerized decision support did not result in more appropriate or compliant antimicrobial prescribing, nor did it improve accuracy of indication documentation in the CPOE system. Prescribers reported intentionally selecting/recording inaccurate indications in the CPOE to bypass the approval process, save time and avoid reproach from more senior doctors. This workaround was possible because the system was easy to override and indications for use were not monitored and followed-up.
We have identified no previous study assessing accuracy of indication documentation for antimicrobials in a CPOE system. Our results are comparable to those reported in other studies that assessed accuracy of indication (or ‘problem’) selection for non-antimicrobial prescribing (29–95%) [ 19 – 21 ]. However, documentation of indications for antimicrobial use represents a unique case because restriction to use an antimicrobial is often dependent on what the antimicrobial is being used for [ 6 ]. In this study we learned that inaccurate documentation of indications was often deliberate to avoid the associated approval process.
“Gaming” the approval system, and the resultant non-compliance to hospital policy, appeared to be due to several factors, a key influence being that the approval process did not align well with pre-scriber workflow. That is, if ordering an antimicrobial for an indication that is not pre-approved, prescribers were required to step away from the prescribing process, make contact with the antimicrobial stewardship doctor and obtain approval before returning to their prescription. Research has shown that users adopt workarounds in response to systems that are perceived to adversely impact on their workflow [ 24 , 25 ]. Streamlining approval for antimicrobials and minimizing disruption to workflow have been identified as factors associated with long-term effectiveness of other decision support interventions, such as web-based antimicrobial restriction programs [ 6 , 26 ]. Thus, to facilitate compliance with approval processes (i.e. seeking and gaining approval for non pre-approved uses) hospitals should ensure that procedures are efficient and seamlessly integrate into pre-scriber workflow.
Another important factor contributing to prescribers ‘working around’ the system was the absence of monitoring of documented indications. The inclusion of a mandatory indication field in alerts resulted in prescribers entering nonsensical text or punctuation into the system to move past the alert window. This was possible because no verification or validation of content was performed. Although computerized systems often enable greater visibility, and as a consequence, accountability than traditional paper-based systems, this benefit is realized only if computerized data are reviewed and followed-up. Thus, implementation of IT, although often viewed as a solution to overcome inefficiencies associated with paper-based records, may require ‘human’ support and resources to achieve all desired benefits.
Accuracy of indication documentation was not assessable in over a third of the antimicrobial prescriptions in our sample because no indication was recorded in the CPOE or in patient progress notes. Of a particular concern was the failure of prescribers to document antimicrobial indications in patient progress notes. During auditing of paper records, determining an indication for antimicrobial use proved to be a resource intensive and difficult process because prescribers did not often explicitly list an indication for use. The recording of indications in electronic systems has been proposed as a panacea to this documentation problem [ 18 ], but our results suggest that accuracy of electronic documentation is likely to be dependent on a number of work process and system factors.
One system design aspect that influenced appropriateness, compliance and accuracy of indication documentation was the number of indication options available to prescribers for selection. During interviews, prescribers reported that long lists of options potentially increased the risk of incorrect selection. Our quantitative findings provide some support for this claim. Each additional indication available for selection in alerts or pre-written orders was associated with a reduction in our three outcome measures. One possible explanation for this finding is “choice overload” – offering people too many choices may impede decision making via the increase in cognitive load associated with choosing from a larger assortment [ 27 ]. Providing decision-makers with limited time to make a decision has been shown to exacerbate the detrimental effect of choice overload on decision-making [ 28 ]. Thus, in a busy clinical environment, where doctors have limited time to place orders, providing prescribers with fewer options is likely to lead to more accurate selection.
This study had a number of limitations. Firstly, we relied on prescriber documentation in patient notes to make our assessments of antimicrobial appropriateness and compliance and this resulted in number of cases being classed as ‘unassessable’. We assumed that the diagnosis recorded in patient notes was accurate. Secondly, we investigated the impact of the decision support redesign on appropriateness, compliance and recording of indications for orange antimicrobials, but did not monitor other unintended consequences of redesign, including for example, changes in use of red antimicrobials. To more fully understand the impact of new decision support on prescriber work, a comprehensive investigation of all antimicrobial prescribing is necessary, preferably in prospective research designs.
6. Conclusion
Overall, we found that implementation of pre-written orders with appropriate indications did not improve antimicrobial prescribing. Workarounds are a risk when compliance with hospital policy creates additional work for prescribers or when system usability is poor (i.e. too many options are provided for selection). Importantly, implementation of IT, in the absence of support or follow-up, is unlikely to achieve all anticipated benefits. Incorporating antimicrobial monitoring and approval into the CPOE system to better align with prescriber workflow, and undertaking audit and feedback of documented indications would be an ideal way forward.
7. Clinical relevance statement
Following implementation of new decision support, workarounds are likely when system use creates additional work for prescribers or when system usability is poor. Implementation of decision support, in the absence of human support or follow-up, is unlikely to achieve all anticipated benefits.
Multiple Choices Question
Integrating an antimicrobial approval process into a computerised order entry (CPOE) system is likely to be unsuccessful if:
A. The system is easy to override
B. Gaining approval via the system does not align with prescriber workflow
C. Antimicrobial prescribing and approvals are not monitored by antimicrobial stewardship staff
D. All of the above
Correct choice is D All of the above . Based on our interviews with prescribers, we determined that doctors were intentionally recording inaccurate indications in the CPOE to avoid gaining approval to use an antimicrobial. The approval process was time consuming and required prescribers to step away from their ordering workflow. Indications recorded in the CPOE system were not regularly reviewed or followed up and the system was easy to override. This combination of factors made it possible to work around the approval process.
Acknowledgements
This research was funded by HealthShare NSW and NHMRC Program Grant 1054146.
Conflict of Interest The authors declare no conflict of interest.
Conflicts of interest
The authors declare no conflict of interest.
Human subjects protection
This research was approved by the hospital’s Human Research Ethics Committee.
References
- 1.Antibiotic Expert Group. . Melbourne, Australia: Therapeutic Guidelines Limited; 2014. Therapeutic Guidelines: Antibiotic (Version 15) [Google Scholar]
- 2.US Department of Health and Human Services. Antibiotic resistance threats in the United States 2013Centers for Disease Control and Prevention; 2013.
- 3.Radford JM, Cardiff LM, Pillans PI, Fielding DIK, Looke DFM. Drug usage evaluation of antimicrobial therapy for community-acquired pneumonia. The Australian Journal of Hospital Pharmacy. 1999;29(06):317–20. [Google Scholar]
- 4.Willemsen I, Groenhuijzen A, Bogaers D, Stuurman A, van Keulen P, Kluytmans J. Appropriateness of antimicrobial therapy measured by repeated prevalence surveys. Antimicrob Agents Chemother. 2007;51(03):864–7. doi: 10.1128/AAC.00994-06. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Davey P, Brown E, Charani E, Fenelon L, Gould IM, Holmes A. Interventions to improve antibiotic prescribing practices for hospital inpatients. Cochrane Database of Systematic Reviews. 2013;4:CD003543.. doi: 10.1002/14651858.CD003543.pub3. [DOI] [PubMed] [Google Scholar]
- 6.Venugopal V, Lehmann CU, Diener-West M, Agwu AL. Longitudinal evaluation of a World Wide Web–based antimicrobial stewardship program: Assessing factors associated with approval patterns and trends over time. Am J Infect Control. 2014;42(02):100–5. doi: 10.1016/j.ajic.2013.09.018. [DOI] [PubMed] [Google Scholar]
- 7.Baysari MT, Lehnbom EC, Li L, Hargreaves A, Day RO, Westbrook JI. The effectiveness of information technology to improve antimicrobial prescribing in hospitals: A systematic review and meta-analysis. Int J Med Inform. 2016;92:15–34. doi: 10.1016/j.ijmedinf.2016.04.008. [DOI] [PubMed] [Google Scholar]
- 8.Kazemi A, Ellenius J, Pourasghar F, Tofighi S, Salehi A, Amanati A. The effect of Computerized Physician Order Entry and decision support system on medication errors in the neonatal ward: experiences from an Iranian teaching hospital. J Med Syst. 2011;35(01):25–37. doi: 10.1007/s10916-009-9338-x. [DOI] [PubMed] [Google Scholar]
- 9.van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(02):138–47. doi: 10.1197/jamia.M1809. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak. 2013;13:111. doi: 10.1186/1472-6947-13-111. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.van der Sijs H, van Gelder T, Vulto A, Berg M, Aarts J. Understanding handling of drug safety alerts: a simulation study. Int J Med Inform. 2010;79(05):361–9. doi: 10.1016/j.ijmedinf.2010.01.008. [DOI] [PubMed] [Google Scholar]
- 12.Devabhakthuni S, Gonzales J, Tata A, Lee S, Shah P, Offurum AI. Evaluation of Vancomycin Dosing and Monitoring in Adult Medicine Patients. Hosp Pharm. 2012;47(06):451–9. [Google Scholar]
- 13.Hall AB, Montero J, Cobian J, Regan T. The effects of an electronic order set on vancomycin dosing in the ED. Am J Emerg Med. 2015;33(01):92–4. doi: 10.1016/j.ajem.2014.09.049. [DOI] [PubMed] [Google Scholar]
- 14.Thiel SW, Asghar MF, Micek ST, Reichley RM, Doherty JA, Kollef MH. Hospital-wide impact of a standardized order set for the management of bacteremic severe sepsis. Crit Care Med. 2009;37(03):819–24. doi: 10.1097/CCM.0b013e318196206b. [DOI] [PubMed] [Google Scholar]
- 15.Karsies TJ, Sargel CL, Marquardt DJ, Khan N, Hall MW. An empiric antibiotic protocol using risk stratification improves antibiotic selection and timing in critically ill children. Ann Am Thorac Soc. 2014;11(10):1569–75. doi: 10.1513/AnnalsATS.201408-389OC. [DOI] [PubMed] [Google Scholar]
- 16.Haynes K, Linkin DR, Fishman NO, Bilker WB, Strom BL, Pifer EA. Effectiveness of an information technology intervention to improve prophylactic antibacterial use in the postoperative period. J Am Med Inform Assoc. 2011;18(02):164–8. doi: 10.1136/jamia.2009.002998. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Baysari MT, Westbrook JI, Richardson KL, Day RO. The influence of computerized decision support on prescribing during ward-rounds: are the decision-makers targeted? JAMIA. 2011;18:754–9. doi: 10.1136/amiajnl-2011-000135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Baysari MT, Oliver K, Egan B, Li L, Richardson K, Sandaradura I. Audit and feedback of antibiotic use. Utilising electronic prescription data. Appl Clin Inform. 2013;4(04):583–95. doi: 10.4338/ACI-2013-08-RA-0063. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Falck S, Adimadhyam S, Meltzer DO, Walton SM, Galanter WL. A trial of indication based prescribing of antihypertensive medications during computerized order entry to improve problem list documentation. Int J Med Inform. 2013. [DOI] [PubMed]
- 20.Galanter WL, Hier DB, Jao C, Sarne D. Computerized physician order entry of medications and clinical decision support can improve problem list documentation compliance. Int J Med Inform. 2010;79(05):332–8. doi: 10.1016/j.ijmedinf.2008.05.005. [DOI] [PubMed] [Google Scholar]
- 21.Walton SM, Galanter WL, Rosencranz H, Meltzer D, Stafford RS, Tiryaki F. A trial of inpatient indication based prescribing during computerized order entry with medications commonly used off-label. Appl Clin Inform. 2011;2(01):94–103. doi: 10.4338/ACI-2010-11-RA-0072. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006;27(02):237–46. [Google Scholar]
- 23.Patton MQ.Qualitative evaluation and research methods3rd ed.NewBury Park, CA: Sage Publications Inc; 2001 [Google Scholar]
- 24.Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals. BMJ Quality & Safety. 2016. [DOI] [PubMed]
- 25.Koppel R, Wetterneck T, Telles JL, Karsh B-T. Workarounds to Barcode Medication Administration Systems: Their Occurrences, Causes, and Threats to Patient Safety. J Am Med Inform Assoc. 2008;15(04):408–23. doi: 10.1197/jamia.M2616. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Agwu AL, Lee CK, Jain SK, Murray KL, Topolski J, Miller RE. A World Wide Web-based antimicrobial stewardship program improves efficiency, communication, and user satisfaction and reduces cost in a tertiary care pediatric medical center. Clin Infect Dis. 2008;47(06):747–53. doi: 10.1086/591133. [DOI] [PubMed] [Google Scholar]
- 27.Chernev A, Böckenholt U, Goodman J. Choice overload: A conceptual review and meta-analysis. J Consum Psychol. 2015;25(02):333–58. [Google Scholar]
- 28.Haynes GA. Testing the boundaries of the choice overload phenomenon: The effect of number of options and time pressure on decision difficulty and satisfaction. Psychology and Marketing. 2009;26(03):204–12. [Google Scholar]