Skip to main content
AMIA Annual Symposium Proceedings logoLink to AMIA Annual Symposium Proceedings
. 2005;2005:6–10.

Physician Perspective on Computerized Order-sets with Embedded Guideline Information in a Commercial Emergency Department Information System

Phillip V Asaro 1,, Amy L Sheldahl 1, Douglas M Char 1
PMCID: PMC1560614  PMID: 16778991

Abstract

Computerized provider order entry (CPOE) is a promising conduit for medical knowledge in support of guideline-consistent decision-making at the point of care. While there are many published examples of successful implementations of CPOE with decision support, there remain questions about the effectiveness of commercially available information system products, particularly in the emergency department (ED). We describe an attempt at using the available CPOE functionality in a commercial ED information system to deliver guideline knowledge and report the results of physician surveys regarding paper-based guideline/order-sets and the corresponding CPOE order-sets that replaced them. Physicians reported that they liked the CPOE order-sets better than the paper version and did use the order-sets, but guideline compliance did not improve. Cultural and organizational issues as well as limitations in the functionality of the commercial system appear to have limited the effectiveness of this implementation.

Keywords: Practice Guidelines, Computerized Provider Order Entry, Emergency Medicine

Introduction

Clinical practice guidelines are widely recognized as having potential for improving clinical care. On the other hand, slow acceptance and poor compliance with published guidelines in clinical practice are well documented.1

Computerized provider order entry (CPOE) is an attractive vehicle for delivery of evidence-based clinical guideline information at the point of care.2 Various methods of presentation of guideline information have been reported including context-specific links to intranet or Internet information sources,3 immediate feedback during the ordering process,4 alerts and reminders triggered by orders,5 and pre-constructed order-sets that are consistent with guidelines.6 Reports of successful implementations typically describe CPOE systems with integrated decision support functionality. Furthermore, much of the evidence for CPOE benefit arises from studies with homegrown systems in academic inpatient environments. Questions have been raised regarding generalizability of this evidence to commercially available systems and to other environments such as the emergency department.7

Institutional Background

In an effort to improve the care of patients with acute coronary syndrome (ACS) in our emergency department (ED), a multidisciplinary team including emergency physicians and cardiologists, on the basis of current evidence and national guidelines, arrived at consensus on a guideline with order-sets for our institution. Following six months of development, the guideline took shape as a set of four paper forms, each consisting of 2–3 pages of check-off orders with interwoven guideline information. By design, the first form would be used at the initiation of evaluation and treatment for all patients with chest pain or otherwise suspected ACS. The guideline provides criteria for assignment of patients to one of four risk categories based on initial evaluation. The risk assignment would then guide selection of a risk-specific order-set for 1) acute ST elevation MI, 2) intermediate or high risk ACS (including non-ST elevation MI), or 3) low risk ACS. The rationale for and the structure of the guideline and order-sets were explained and use was promoted in meetings of attending and resident physicians.

Completely independent of the ACS guideline development effort, the ED information system project team was moving toward implementation of CPOE using a commercially available ED information system. The ED was set to implement CPOE, with abrupt and complete cessation of all paper orders, just 2 ½ months after the paper ACS order-sets were implemented. In an attempt to carry the effort of the ACS guideline/order-set development forward into the CPOE system, order-sets were constructed that provide easy access to orders consistent with the guideline.

The ED information system supports pre-defined problem oriented order-sets for common ED problems. Order-set organization is limited to simple listing of orders in a three-level hierarchy including the order-set name, with a width constraint of about 20 characters for lines at the third level of hierarchy. Significant adaptation of the ACS guideline/order-sets was necessary to fit the available format.

The chest pain/ACS order-sets in the CPOE system follow the same logic as their paper counterparts with an initial order-set intended to be used for all ACS patients and three sets specific to risk category. Correct use of the order-sets would include opening of the initial order-set plus one of the risk-category-specific sets. Information-only lines, based on guideline text, were inserted between the actual order items in the order-sets (e.g. “consider clopidogrel if ASA allergic” or “preferred for age > 75”). ACS risk stratification criteria were incorporated into the system in two ways. First, we placed PDF (portable document format) versions of the paper forms online, accessible via a link at the bottom of the order entry screen. Secondly, we created “information-only order-sets” listing the criteria for each of the risk stratification categories, using abbreviated versions of the criteria to fit the available format. A provider can open the STEMI criteria list, for example, to see the criteria for ST elevation MI. Another information-only order-set lists the recommended criteria for selecting from the various reperfusion/thrombolysis options.

Objectives

1) Gain insight into ED physician perspectives on ACS guideline information embedded in order sets in a commercial CPOE system without more robust decision support functionality. 2) Determine whether physician survey responses regarding their perspectives correlate with level of training. 3) Discuss the survey results in light of data on actual order-set use and β-blocker ordering.

Methods

Anonymous surveys were administered to ED resident and attending physicians. In a survey prior to CPOE implementation, physicians were asked about the paper version of the guideline/order-sets. After CPOE implementation, a second survey asked about the CPOE version.

The surveys were distributed at weekly ED conferences, which are attended by most of the 51 resident emergency medicine physicians and many of the 30 attending physicians. In addition, survey forms were placed in mailboxes with a request that physicians who had not already completed the survey do so. The surveys requested responses using a 10-point Likert scale.

We performed retrospective chart reviews to obtain data regarding order-set use and compliance with the guideline recommendation for β-blocker use. We examined charts of patients with an ED diagnosis of MI, ACS, or unstable angina during each of four one-month periods: 1) before the implementation of the paper guideline, 2) during use of the paper guideline just before CPOE implementation, 3) beginning one month after CPOE implementation, and 4) beginning three months after CPOE implementation. An order-set use score was assigned to each case (see Table 3).

Table 3.

Use of order-sets by phase

None (0) Partial (1) Initial or Advanced (not both) Nonspecific (2) Initial and Multiple Advanced Sets Specific (3) Initial and One Advanced (simple preprinted form in Phase 1)
Phase 1 33/45 (73%) 12/45 (27%)
Phase 2 36/66 (55%) 20/66 (30%) 10/66 (15%)
Phase 3 12/48 (25%) 17/48 (35%) 3/48 (6%) 16/48 (33%)
Phase 4 8/49 (16%) 14/49 (29%) 8/49 (16%) 19/49 (39%)

Statistical comparison of means of responses to questions between groups was performed with the independent samples t-test. Spearman’s rho was used to examine correlations between ordinal variables including responses to questions vs. level of training of physicians, order-set use score vs. treatment with a β-blocker, and between responses to different questions.

Results

Respondents to the pre-CPOE survey represented 30% of physicians (35% of residents, 20% of attendings). The response rate to the post-CPOE survey was 44% overall (50% of all residents, 37% of attendings). The effective response rate of residents was a little higher than 50% for the second survey in that a few first year residents had not yet rotated into the ED, had no experience with the ordering system, and did not complete the survey.

It should be noted that a residency-year transition took place between these two surveys. For example, residents in their first year at the time of the first survey were in their second year at the time of the second survey. First-year residents in the second survey could not answer the question asking for comparison of the paper and CPOE guidelines as they entered the residency after CPOE implementation; there are therefore fewer data points in answers to that question.

Survey responses are summarized in Table 1. In general, responses of resident physicians tend to be more positive than those of attending physicians. The resident physician responses in the second survey (regarding the CPOE version) were more positive than those to the first survey (regarding the paper version). Attending physician responses on the two surveys were similar with lower means. However, when asked specifically whether the CPOE version was better than the paper version, a somewhat positive response resulted.

Table 1.

Survey responses – mean response (10-point Likert scale) with standard deviation (SD)

Pre- or Post- CPOE Attending mean (SD) Resident mean (SD) Significance of Attend - Resid Difference
Please rate the ease of use of the current guideline Pre 5.2 (4.0) 5.9 (2.3) p = 0.70
Post 5.4 (2.3) 7.4 (1.6) p = 0.02
The current CP/ACS guideline (order-set) assists me by prompting organized and complete orders Pre 4.8 (3.5) 6.3 (2.8) p = 0.39
Post 5.5 (2.5) 7.0 (2.1) p = 0.10
The current CP/ACS guideline (order-set) leads to better patient care Pre 4.6 (2.3) 6.2 (2.6) p = 0.17
Post 6.2 (2.5) 7.1 (2.4) p = 0.30
Compare the ease of use of the current CPOE order-set guideline to the previously used paper order sheets/guideline Post 6.6 (2.0) 7.3 (2.2) p = 0.46
How useful are the abbreviated stratification criteria on the order-set tab? Post 3.8 (2.4) 6.4 (2.0) p = 0.01
How useful are the expanded risk stratification criteria available from the “globe”? Post 4.1 (3.0) 5.8 (2.4) p = 0.13
How useful are the intermingled prompts that are imbedded in the orders? Post 4.2 (2.7) 6.6 (2.3) p = 0.02

Table 2 shows correlations between responses to questions as well as correlations between questions and the level of training of physicians. There are several positive correlations between questions. There is a negative correlation between responses and training level for many of the questions. One notable exception is the question regarding comparison between the CPOE and paper versions of the order-sets.

Table 2.

Correlations between responses to questions and to level of training -- Spearman’s rho (p-value)

Pre- or Post-CPOE Level of Training Ease of Use Organized Orders Better Patient Care
Ease of Use Pre −0.18(0.40) -- 0.51(0.01) 0.45(0.03)
Post −0.42(0.01) -- 0.57(0.001) 0.63(0.001)
Organized Orders Pre −0.37(0.08) 0.51(0.01) -- 0.81(0.001)
Post −0.42(0.01) 0.57(0.001) -- 0.84(0.001)
Better Patient Care Pre −0.49(0.02) 0.45(0.03) 0.81(0.001) --
Post −0.26(0.13) 0.63(0.001) 0.84(0.001) --
CPOE Better Than Paper Post −0.17(0.37) 0.53(0.01) 0.68(0.001) 0.66(0.001)
Abbrev Criteria Useful Post −0.38(0.02) 0.43(0.01) 0.40(0.02) 0.37(0.03)
Expanded Criteria Useful Post −0.35(0.06) 0.41(0.03) 0.66(0.001) 0.62(0.001)
Integrated Prompts Useful Post −.046(0.01) 0.65(0.001) 0.72(0.001) 0.50(0.01)

Rates of use of paper forms and opening of CPOE order-sets are shown in Table 3. CPOE order-set opening was more frequent than was use of paper forms.

Table 4 shows rate of β-blocker use in high-risk patients in each of the four phases. Overall percentage is shown as well as a breakdown by order-set use. There was a statistically significant correlation between β-blocker use and the order-set use score only during the 4th phase.

Table 4.

Orders for β-blocker by order-set use for high risk patients without contraindication

N % β-Blocker None (0) Partial (1) Nonspecific (2) Specific (3) Spearman(Sig)
Phase 1 21 81% 88% of 16 60% of 5 −0.30 (p=0.19)
Phase 2 37 68% 68% of 22 70% of 10 60% of 5 −0.03 (p=0.86)
Phase 3 26 69% 33% of 6 88% of 8 100% of 1 73% of 11 0.24 (p=0.23)
Phase 4 26 77% 67% of 3 43% of 7 80% of 5 100% of 11 0.49 (p=0.01)

Discussion

ACS is a common and potentially serious problem in the ED. About 8 % of patient visits in our ED have chest pain as chief complaint and at least 20% of these patients are given an ED diagnosis consistent with either MI or ACS. There is good evidence for benefit with use of recommended medications and there is consensus at a national level regarding early aggressive treatment.8 There remains significant variance between the recommendations and actual care, both locally and nationally. Care of ACS patients in the ED is therefore a good target for guideline implementation.

We observed an interesting pattern in order-set use and β-blocker orders. Use of the paper forms was quite low (Phase 2). Initial use of the CPOE order-sets was also low (Phase 3), but did increase after several months (Phase 4). Opening of the order-sets correlated with ordering of a β-blocker, but overall β-blocker use did not increase significantly. This pattern suggests that the correlation arises from the use of the order-sets for the convenience of their pre-formed orders rather than from prompting of orders resulting from use of the order-sets.

Correlations between level of training and responses to questions concerning the usefulness of various aspects of the electronic version of the guideline suggest that inexperienced physicians consider the prompting by the guideline to be more useful than do more experienced physicians. Even so, physicians at all levels of training indicate that the electronic version of the guideline/order-set is superior to the paper version. Not surprisingly, physicians who consider some aspects of the guideline in a positive light also feel positively about other aspects, as evidenced by the significant correlations between responses to several pairs of survey questions.

Responses to questions about ACS criteria and integrated prompts are also interesting. The attending physicians gave decidedly negative responses to these questions in contrast to the responses of resident physicians. Based on click counts in the information system, the abbreviated criteria lists are rarely viewed. In view of the minimal actual use of the criteria lists and the overall lack of change in ordering behavior, the positive responses of resident physicians may indicate that they conceptually view decision support as potentially useful rather than indicating that they find the current functionality helpful. Comments on survey forms and informal discussion with residents seem to support this view.

An open-ended final question on the surveys yielded a number of comments related to “cookbook medicine” and some expression of concern regarding the adverse effect of predefined order-sets on learning by physicians in training. These are often-discussed concerns in the context of CPOE and decision support. Interestingly, the physicians who made these comments generally gave otherwise positive responses on the survey.

Structural and organizational issues are prominent barriers to successful implementation in our setting. Variability in attending physician agreement with guideline recommendations is often cited by resident physicians as a contributing factor. Physician attitude toward an implementation is certainly a critical factor for success.9 In an academic institution, attending physician variability may have a trickle-down effect resulting in inconsistency in guideline adherence among resident physicians, particularly in the emergency department where the attending physician is typically more directly involved in decision-making throughout the patient encounter. Our data, similar to those of some other studies,10 demonstrate an inverse relationship between the perceived value of decision support and training level of the user. Attending physicians may be slower to adopt such input into their clinical practices.

Another source of decision-making influence in the ED is interaction with other services, particularly the inpatient internal medicine and cardiology services for ACS patients. Inconsistencies in hospital-wide adoption of the guideline may further contribute to lower than expected use of recommended treatments in the ED. These problems are consistent with published discussions of CPOE and guideline implementation that mention organizational and cultural issues requiring substantial organizational efforts to overcome.11, 12

A number of studies have shown computerized decision support using alerts and reminders to be effective. Our implementation differs in that we are unable to require a response to recommendations, provide patient-specific recommendations, or provide ready access to justification of recommendations. Our data are consistent with the view that at least lesser-experienced physicians are interested in assistance with “doing the right thing.” It seems likely that our implementation was limited by our inability to provide direct assistance requiring minimal additional time and effort along with readily available supportive evidence. Few ED clinical information systems have significant decision-support functions. Further research on the efficacy of decision support in the ED is needed, but it is very likely that some degree of decision support will be feasible and efficacious. Therefore, purchasers of ED systems should be attentive to decision support functionality. Given the demand, vendors are more likely to develop the functionality required.

Limitations

The poor response rate to the surveys, especially from attending physicians, raises questions regarding possible sampling bias.

Conclusion

Survey data indicate conceptual openness to decision support, at least by lesser-experienced physicians. However, guideline-consistent CPOE order-sets alone failed to improve adherence to a clinical practice guideline in our ED. The primary contributions to this failure appear to be cultural and organizational, but the lack of decision support functionality is a likely contributor. Further research is needed on the efficacy of decision support in the ED, but purchasers of ED information systems should be very attentive to these issues, thereby encouraging vendors to further develop the required functionality in their products.

References

  • 1.Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. Jama. 1999;282(15):1458–65. doi: 10.1001/jama.282.15.1458. [DOI] [PubMed] [Google Scholar]
  • 2.Rosenbloom ST, Talbert D, Aronsky D. Clinicians' perceptions of clinical decision support integrated into computerized provider order entry. International Journal of Medical Informatics. 2004;73(5):433–41. doi: 10.1016/j.ijmedinf.2004.04.001. [DOI] [PubMed] [Google Scholar]
  • 3.Cimino JJ, Li J. Sharing infobuttons to resolve clinicians' information needs. AMIA Annu Symp Proc. 2003815 [PMC free article] [PubMed] [Google Scholar]
  • 4.Bindels R, Hasman A, Derickx M, Van Wersch JW, Winkens RA. User satisfaction with a real-time automated feedback system for general practitioners: A quantitative and qualitative study. Int J Qual Health Care. 2003;15(6):501–8. doi: 10.1093/intqhc/mzg076. [DOI] [PubMed] [Google Scholar]
  • 5.Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW. Effects of computerized physician order entry on prescribing practices.[see comment] Archives of Internal Medicine. 2000;160(18):2741–7. doi: 10.1001/archinte.160.18.2741. [DOI] [PubMed] [Google Scholar]
  • 6.Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. Journal of the American Medical Informatics Association. 2003;10(4):322–9. doi: 10.1197/jamia.M1090. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Handler JA, Feied CF, Coonan K, et al. Computerized physician order entry and online decision support. Acad Emerg Med. 2004;11(11):1135–41. doi: 10.1197/j.aem.2004.08.007. [DOI] [PubMed] [Google Scholar]
  • 8.Hoekstra JW, Pollack CV, Jr, Roe MT, et al. Improving the care of patients with non-st-elevation acute coronary syndromes in the emergency department: The crusade initiative. Acad Emerg Med. 2002;9(11):1146–55. doi: 10.1111/j.1553-2712.2002.tb01569.x. [DOI] [PubMed] [Google Scholar]
  • 9.Weir C, Lincoln M, Roscoe D, Moreshead G. Successful implementation of an integrated physician order entry application: A systems perspective. Proceedings - the Annual Symposium on Computer Applications in Medical Care. 1995:790–4. [PMC free article] [PubMed] [Google Scholar]
  • 10.Tierney WM, Overhage JM, McDonald CJ, Wolinsky FD. Medical students' and housestaff's opinions of computerized order-writing. Acad Med. 1994;69(5):386–9. doi: 10.1097/00001888-199405000-00013. [DOI] [PubMed] [Google Scholar]
  • 11.Ash JS, Gorman PN, Lavelle M, et al. A cross-site qualitative study of physician order entry. Journal of the American Medical Informatics Association. 2003;10(2):188–200. doi: 10.1197/jamia.M770. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Kuperman GJ, Gibson RF. Computer physician order entry: Benefits, costs, and issues. Annals of Internal Medicine. 2003;139(1):31–9. doi: 10.7326/0003-4819-139-1-200307010-00010. [DOI] [PubMed] [Google Scholar]

Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association

RESOURCES