Using Computerized Provider Order Entry and Clinical Decision Support to Improve Referring Physicians' Implementation of Consultants' Medical Recommendations

Abstract

Objectives

Only half of consultants' medical recommendations are implemented. We created a tool that lets referring providers review and implement electronic recommendations made by consultants, with the hypothesis that facilitation with our tool could improve implementation.

Measurements

The tool was piloted among geriatrics consultants and hospitalists. Pre-post evaluation was done with control (before pilot; N = 20) and intervention (after pilot; N = 20) patients. Consultants wrote notes containing recommendations for all study patients, and entered electronic recommendations only for intervention patients. We analyzed all recommendations and surveyed hospitalists.

Results

A total of 249 recommendations were made for intervention patients versus 192 for controls (p < 0.05). Of all recommendations about intervention patients, 78% were implemented, compared to 59% for controls (p = 0.01). Of the intervention recommendations, 77% were entered electronically using our tool; of these, 86% were implemented. All 24 survey respondents indicated that the system improved quality, saved time, and should be expanded.

Conclusion

Consultant recommendations were implemented 30% more often when there was electronic facilitation of recommendations.

Introduction

Specialty consultants often make recommendations to be addressed by referring providers. Studies show that in some cases only about half of these recommendations are actually implemented. ¹ Although ideal implementation rates will vary depending on the specific circumstance, the low rates suggest that appropriate recommendations are not being implemented. Factors contributing to such low adoption of recommendations include faulty communication, ^2,3 inappropriate timing, inadequate detail in the recommendations, ⁴ difference in opinions, ^5,6 lost paperwork, and administrative or systems-based barriers. Approaches to improve implementation of recommendations and quality of consultations are needed, and these interventions should respect referring physicians' knowledge, preferences, opinions, and autonomy for writing orders on their patients, especially in settings in which consultants provide advice rather than direct treatment. ⁷

Berwick suggests that improving quality may require changing the overall flow of work and patterns of interaction among providers. ⁸ Although many have found that recommendations are best communicated in face-to-face meetings or through phone conversation, additional, asynchronous communications ^9,10 may facilitate care while not overburdening the highly interruptive work environment. ^4,11 In particular, asynchronous communications about consultation offer a viable approach for improving implementation rates.

Our local preliminary work in this area found that only 51% of recommendations made by an inpatient geriatrics consultation team were implemented. ¹² Although implementation varied by physician, multivariate analysis showed that implementation was most likely when the consultant started the process of care in some way (odds ratio > 4, p < 0.001). This finding is similar to observations in other studies in which facilitation by consultants, either through direct implementation of recommendations ^13,14 or by assisting with case management, ¹⁵ led to higher implementation rates. In focus groups at our institution, both referring and consulting physicians cited lack of prioritization of recommendations, illegibility, and inefficiency of paper-based communication as contributors to these low rates. Both groups viewed a computerized provider order entry (CPOE) ¹⁶ system as one way to facilitate the implementation of appropriate recommendations. A CPOE system could be used to improve legibility, timeliness, and specificity in making recommendations. It could also help consultants to leverage CPOE-based clinical decision support (CDS) to increase the number of recommendations made without duplicating existing orders.

In response to this feedback, we sought to integrate electronic facilitation of recommendations into an inpatient CPOE system. We created a tool to enable inpatient consultants to create detailed electronic consultant-recommended orders (CROs) and allow referring hospitalists to review, modify, and directly implement these CROs. In this article, we describe the design, creation, and pilot implementation of this system; the results of a pre-post analysis of the effect of CROs on the number of recommendations made and implemented; and findings of a satisfaction survey for referring care hospitalists who used this system. We hypothesized that this system could improve implementation of care suggestions and would be valued by clinicians.

Methods

Setting and Electronic Health Record

We performed this study at a 264-bed, urban, tax-supported, Midwestern hospital on the campus of an academic medical center. The hospital is served by a comprehensive electronic medical record (EMR) system. ¹⁷ The EMR has a local CPOE system, The Medical Gopher, ¹⁷ through which all orders—ranging from medications, diagnostic studies (including laboratory and radiology studies), ancillary services, nursing orders, and consultation requests—must be entered. Orders become active when they are electronically signed by the clinician. The CPOE system provides alerts and routine care suggestions and checks for drug–allergy and drug–drug interactions. The CDS in the EMR is implemented using the G-CARE language. ¹⁸ At the time of this study, all orders were being entered by primary teams via CPOE. Also at baseline, most consultants hand-wrote their clinical chart notes and recommendations, and the referring team was responsible for implementing these recommendations.

Intervention: CRO Tool

The new CRO tool enabled providers to enter orders or CROs (i.e., recommendations). Maintaining a clear distinction at all levels between actual orders and CROs was a key design consideration. We started by engaging an interdisciplinary team of providers, to ensure that the CRO tool was compatible with existing workflow (▶).

Figure 1.

Workflow for consultant-recommended orders. RMRS = Regenstrief Medical Records System for electronic medical records.

The CRO tool complemented consultants' notes and direct communication with referring providers. To enter CROs, a consultant logged into the CPOE system, which launched the CRO interface on demand. There, the consultant could enter new CROs or view, edit, or remove existing ones (▶). New CROs included suggested diagnostic tests, drugs, other therapies, activity or diet, nursing procedures, or additional consultations. Consultants could also recommend revising or discontinuing orders that were already active for a patient. The tool took full advantage of the CDS system: it provided usual alerts for duplicates of active orders and checked for interactions and appropriate dosing of drugs. Consultants could also type their notes through the CRO interface. The CROs and notes were stored in the EMR and could be printed or viewed from any workstation (▶).

Figure 2.

Interface for consultants to enter consultant-recommended orders and notes.

Figure 3.

Consultant's note with appended consultant-recommended orders.

When referring team providers logged into a patient's record, they were automatically presented with the list of CROs. They could then modify, delete, or approve them (▶) or could simply leave the CRO screen by pressing the ESC key. The CROs that the referring team approved were implemented directly, as if created de novo by the referring provider, and CROs that had already been implemented were not displayed on subsequent views. ▶ outlines considerations in developing the CRO tool. To avoid confusion between CROs and regular orders, all documentation containing CROs clarified that these were recommendations and not yet active orders (▶). In the data repository, orders and CROs were stored separately. We set privileges for editing and acting on CROs on a provider and team basis. All CROs were inactivated when a patient was transferred or discharged from the hospital. At any time during a regular CPOE session, providers could access the CRO tool by selecting a new menu option entitled Consultant Recommended Orders.

Figure 4.

Referring provider reviewing and acting on consultant-recommended orders.

Table 1.

Table 1 Considerations when Incorporating CROs into CPOE

Technical

How to avoid confusion with active orders in user interface, notes, and in data repository

How to avoid confusion with student orders

How to use existing CPOE functions for CROs

How to take advantage of existing Clinical Decision Support system

How to recommend discontinuation or revision for items implemented without active orders in the CPOE

Administrative

Who has privileges to enter, view, or act on CROs?

What happens to CROs at discharge?

What happens to CROs when a patient is transferred to the ICU or to another team?

Should CROs be prioritized, and if so, how?

Should CROs be purged after a pre-established number of days?

Whether to allow for CROs to be printed or displayed as notes

Workflow

When should CROs be displayed to providers?

Should display be automatic or should users decide when to view CROs?

CPOE = computerized provider order entry; CRO = consultant-recommended orders; ICU = intensive care unit.

Study Design

We conducted a before-and-after pilot study of the implementation of the CRO tool, evaluating 40 hospital admissions (20 before and 20 during implementation). The hospital's general internal medicine service has eight hospitalist teams, each staffed with an attending physician and three staff members. Most ordering—and hence, most use of CPOE—occurs by staff members. The geriatrics consulting team consists of an attending geriatrician, a nurse practitioner, a geriatrics fellow, and staff members rotating on the service. All recommendations by the geriatrics team were approved by the attending geriatrician. Patients in the study were those who were admitted to the hospitalist service and underwent geriatrics consultation. Patients excluded from the study included those who were discharged <24 hours after geriatrics consultation, died in the hospital, or were transferred to another service. The study was approved by the institutional review board and was conducted between December 2007 and February 2008.

All medicine staff members received personal instruction about the new CRO feature at the beginning of their monthly rotation. Providers were encouraged to practice with the CRO feature before using it with their hospitalized patients. To maximize consistency of training, a training video was produced, shown, and made available for 24-hour access on a secure intranet site. Round-the-clock support was offered by the investigator team, but no providers used this service during the pilot. We trained the geriatrics nurse practitioner to enter CROs on behalf of the geriatrics consulting team; the team discusses and agrees on recommendations during daily rounds.

Before introduction of the CRO tool, geriatrics consultants conveyed their recommendations as part of a free-text note. After introduction, this practice continued, but in addition, they entered recommendations as CROs using the tool. During the pilot implementation, individual uses of the CRO tool occurred at consultants' discretion for intervention patients. Consultants were encouraged, but not required, to use the system to generate all CROs for intervention patients.

Survey

To assess referring providers' attitudes about the effect of using CROs, we designed a survey targeting the primary care staff members as the main users of CPOE. An anonymous, self-administered questionnaire was administered to a convenience sample of 29 (81%) of the 36 medicine staff members who had used the CRO tool. The questionnaire asked about clarity, usefulness, and quality of consultations when there were CROs as opposed to no CROs. Responses were on a 5-point Likert scale, with ratings of 1 (much worse) to 5 (much better). Relative ease of implementing recommendations when there were CROs was rated as 1 (much harder) to 5 (much easier). Providers also responded to questions about the future of the CRO tool. The questionnaires were administered during the routine morning report and educational conferences.

Data Analysis

One investigator reviewed medical records of all study patients and documented information about all recommendations made by the geriatrics team, including whether they were implemented. An approved order that corresponded to a recommendation was considered equivalent to implementation of the recommendation. Recommendations meant to be implemented only after discharge were excluded. To assess reliability of identifying recommendations and their implementation, a second investigator independently re-reviewed 40% of the study records.

The total number of recommendations per patient was compared between groups using an extra-Poisson model. Inter-rater reliability was estimated with the kappa statistic. We then compared implementation rates of the recommendations between the intervention and control admissions using a nonlinear mixed model with a binomial outcome; predictors included intervention as a fixed effect and patients as random effects, to account for multiple recommendations per patient. Although consultants were not required to use the CRO tool for intervention patients, we conducted an intention-to-treat analysis, which provided conservative estimates of associations with the intervention. We computed descriptive statistics for the survey.

Results

Patient Characteristics

Gender, race, age, and hospital length of stay were not statistically different between the intervention and control groups (▶).

Table 2.

Table 2 Characteristics of Participants (N = 40)

		Study Group			Implementation of Recommendations (%)
Characteristic	Frequency (%)	Control (%)	Intervention (%)	P Value
Gender				0.311
Female	27 (68)	75	60		68
Male	13 (33)	25	40		68
Race				0.276
African American	21 (53)	55	50		68
White	17 (43)	35	50		69
Other	2 (5.0)	10	0		56
Age group				0.329
65–74 years	12 (30)	40	20		67
75–84 years	18 (45)	35	55		75
≥85 years	10 (25)	25	25		57
Length of stay ∗				0.406
1–2 days	10 (25)	25	25		59
3–5 days	11 (28)	35	20		64
6–7 days	8 (20)	10	30		79
8–19 days	11 (28)	30	25		73

^∗ Categorized by quartiles.

Inter-rater Reliability

Seventeen medical records were reviewed by two investigators for inter-rater reliability. Of 202 recommendations identified by either reviewer, 169 (84%) were identified by both. Among these, reviewers agreed about implementation in 146 recommendations (86%; kappa = 0.7018), regardless of study group (control = 82%, intervention = 89%, p = 0.185 by chi-square analysis). This analysis established confidence about the primary reviewer's assessments for the study.

Implementation of Recommendations

More total (247 vs. 192, p < 0.05) and per-admission (mean 12.4 ± 4.2 vs. 9.6 ± 3.7) recommendations occurred among intervention patients compared to controls. Among intervention patients, consultants entered 190 (77%) of the recommendations electronically as CROs; 164 (86%) of these were implemented. Overall implementation of recommendations was significantly higher in the intervention group (78% vs. 59%, p = 0.01). All types of recommendations were made for both the control and intervention groups. We observed variations in implementation by type of recommendation, with generally higher implementation rates in the intervention group (▶).

Table 3.

Table 3 Implementation by Type of Recommendation

Recommendations		Implementation
Type	No. (%)	Control N (%)	Intervention N (%)
Consultative	41 (9.7)	17 (94)	21 (91)
Rehabilitative	49 (12)	15 (75)	24 (83)
Diet	24 (5.7)	4 (67)	14 (78)
Laboratory	34 (8.0)	12 (60)	13 (93)
Directive	3 (0.7)	0 (0)	2 (100)
Pharmacologic	217 (51)	56 (54)	87 (77)
Fluids	7 (1.7)	N/A	4 (57)
Devices, durable	22 (5.2)	7 (58)	5 (50)
Social	2 (0.5)	N/A	1 (50)
Skin hygiene	16 (3.8)	1 (17)	5 (50)
Solitary functional	6 (1.4)	1 (25)	1 (50)
Physical examination	2 (0.5)	0 (0)	0 (0)

N/A = not applicable.

Survey Results

The response rate was 83% (24 of 29). On a 5-point Likert scale, respondents agreed that CROs were more useful (4.29 ± 0.69) and clearer (4.46 ± 0.59) and that the quality of consultation (4.25 ± 0.68) and communication between primary and geriatrics teams (4.08 ± 0.88) improved when compared to recommendations made without CROs. Respondents also stated that it was easier to implement recommendations made as CROs (4.79 ± 0.41). All respondents stated that CROs saved time and that all consultants should use the CRO tool.

At the end of the pilot study, geriatrics consultants asked to continue using the CRO tool. They stated that the CRO tool helped them to keep track of which of their recommendations had not been addressed, and they valued the tool's integration with CDS.

Discussion

This study described and studied an innovative tool that enables specialty consultants to enter recommendations electronically via CPOE and allows referring providers to review and act directly on these recommendations. Our pilot study shows that the use of this collaborative, facilitative tool was associated with a significantly higher rate of implementing recommendations (78% vs. 59%). Referring physicians also indicated a strong preference for using this tool for all inpatient consultations.

The implementation rate was higher in the intervention group, despite a significantly higher number of recommendations per admission in this group. This goes against reports in several prior studies indicating that implementation rates decrease when more recommendations are made. A study of cardiology consultation showed that nonconcordance with diagnostic suggestions was highest when the most requests were made. ¹⁹ In addition, Maly et al. ²⁰ reported that the implementation rate was higher when there were fewer than five recommendations, and referring physicians were unhappy when the length “of the list of recommendations seemed disproportionate to the complexity of the problems.” ²¹ In the present study, the CRO tool seems to reduce the relative effort needed to implement recommendations, hence diminishing the burden otherwise associated with higher numbers. Recommendations made using the tool rarely duplicate active orders and are specific, legible, and easy to access from any institutional terminal. As an example, medications recommended through the CRO tool contain exact doses and frequencies, and this kind of specificity from consultants is known to improve implementation. ⁴ With the CRO tool, referring providers do not need to spend valuable time figuring out how to order unfamiliar items in recommendations, and they can implement the recommendations with a few clicks of the keyboard or mouse. The CRO tool makes active orders available to consultants as they are making recommendations, and they can take advantage of built-in decision support in the CPOE system. In addition, consultants can easily track recommendations that the referring team has not yet addressed.

Our work shows the potential role of informatics to improve the consultation process. Targets of informatics-based interventions should be chosen carefully, ²² and their implementation must respect existing workflow. At our hospital, like many others especially in academic medical centers, consultants do not provide direct treatment or implement recommendations directly; instead, they make recommendations, and referring clinicians are responsible for implementation. This approach respects the referring teams' knowledge of the patient, their opinions, and their preference to have order-writing authority and care-coordination responsibilities for their patients. ⁷ Physicians will sometimes disagree about the appropriateness of particular recommendations, and it is the role of the referring physician to balance recommendations made across multiple specialties against the patient's total clinical condition. The CRO tool was created with special attention to the existing workflow, and it can readily be extended for use by additional referring and consulting teams or services. By providing a shared, detailed specification of recommendations, the tool complements and strengthens communication among providers and effectively improves the handoff process.

Several limitations of our study deserve mention. The before-and-after design may introduce bias, our reviewers were not blinded, and the small sample size did not allow for adequate control for clustering by provider, length of stay, or patients' characteristics. Two other factors may have partly explained improvement. First, consultants may have inadvertently started to pay more attention to recommendations. Second, the level of detail needed for recommendations in the intervention phase may have been greater than in the control period and so may have artificially inflated both the number of recommendations made and the percentage implemented. Nevertheless, the relative large increase in implementation suggests that these are not the only effects. Another limitation of the study is that it was conducted using one consulting service at a single academic center, and the findings may not necessarily translate across all consulting services or institutions. The tool would also have an uncertain role in institutions where consultants implement their recommendations directly, and the survey results may not be representative of the entire group.

Through this pilot study, we were able to show the feasibility and potential benefit of recommended orders made via the CPOE system. The approach we used to create recommended orders can be adapted to facilitate implementation of pre-admission orders, for making recommendations during the inpatient-to-outpatient transition, and for managing consultant recommendations in the outpatient setting. We intend to evaluate the impact of the CRO tool on clinical outcomes.

Conclusion

Implementation of recommendations improved by more than 30% when recommendations were entered and processed via CPOE, and referring clinicians strongly preferred to have this tool for all inpatient consultations.