Changes in end-user satisfaction with Computerized Provider Order Entry over time among nurses and providers in intensive care units

Abstract

Background

Implementation of Computerized Provider Order Entry (CPOE) has many potential advantages. Despite the potential benefits of CPOE, several attempts to implement CPOE systems have failed or met with high levels of user resistance. Implementation of CPOE can fail or meet high levels of user resistance for a variety of reasons, including lack of attention to users’ needs and the significant workflow changes required by CPOE. User satisfaction is a critical factor in information technology implementation. Little is known about how end-user satisfaction with CPOE changes over time.

Objective

To examine ordering provider and nurse satisfaction with CPOE implementation over time.

Methods

We conducted a repeated cross-sectional questionnaire survey in four intensive care units of a large hospital. We analyzed the questionnaire data as well as the responses to two open-ended questions about advantages and disadvantages of CPOE.

Results

Users were moderately satisfied with CPOE and there were interesting differences between user groups: ordering providers and nurses. User satisfaction with CPOE did not change over time for providers, but it did improve significantly for nurses. Results also show that nurses and providers are satisfied with different aspects of CPOE.

Introduction

In a paper-based hospital order-management system, a physician or mid-level provider examines the patient, if necessary writes an order, hands the order off to the nurse, who in turn hands it off to the unit clerk, who in turn makes sure that it is delivered to the pharmacy or other appropriate services. For medication orders, the pharmacist double checks the order and prepares and dispenses the medication. Finally, the medication is delivered to the unit, and the nurse administers the medication to the patient. For other orders, the appropriate service arranges for the test or other intervention and the patient receives it. This description of the order management process does not reflect the complexity of the actual process, such as the variety of people and disciplines involved and the many opportunities for errors as well as error recovery.^1–3 The introduction of a complex technology such as Computerized Provider Order Entry (CPOE) into the complex order-management process is likely to have both positive and negative effects on the quality and safety outcomes of the process, as well as the clinician executing the process.

CPOE is a technology used by clinicians to directly enter medication and other orders into a computer system; orders are then directly transmitted electronically to the pharmacy and to other services. CPOE technology supports standardized, legible, and complete orders and can reduce medication errors at the ordering, transcribing, and pharmacy review stages of the order-management process.^{4 5} CPOE technology can have additional benefits, such as speeding up the ordering process and reducing the number of people who are required to participate in the workflow, thereby reducing delays and errors due to miscommunication.^6–9

CPOE technology may include Clinical Decision Support (CDS), which supports the quality and safety of ordering activities.¹⁰ Basic CDS may display default values for the drug dose, frequency, and route of administration, and alert providers to drug allergies, drug–drug interactions, and duplicate orders. Advanced CDS may include medication dosing support for kidney disease and drug–disease checking.

While CPOE technology can improve the quality of care and patient safety,^{4 5 11–14} in particular medication safety,¹⁵ some attempts to implement CPOE technology have failed, either meeting with high user resistance or contributing to safety problems.^{14 16–24} CPOE is available in less than 20% of US hospitals,^{14 25 26} but implementation is accelerating, with many hospitals and other facilities planning an implementation.^{26 27} CPOE implementation efforts have stumbled for a variety of reasons, including lack of attention to users’ needs that often led to dissatisfaction of users with the technology.^{20 21 28–32}

Studies have examined the impact of CPOE implementation on end users, but primarily on providers and rarely on nurses. Some studies have compared the impact of CPOE on providers and nurses. For example, Callen et al³³ examined CPOE implementation in two Australian hospitals and showed that physicians and nurses have different attitudes towards CPOE: overall nurses were more positive about CPOE than physicians. Few studies have examined the effect of CPOE on intensive care units (ICUs). Given the high number of medication orders and other orders for ICU patients³⁴ and the high risk of medical errors in ICUs,³⁵ it is important to examine the impact of technology on ICU clinicians.

Few studies have examined the long-term impact of CPOE. Results of a recent study by Khajouei et al⁹ showed that end-users were satisfied with CPOE ease of use, efficiency, workflow, and patient safety, nearly 10 years after implementation. This long-term positive effect of CPOE technology was identified after an earlier assessment of the technology showed issues with usability and lack of functionalities. To our knowledge, no study has examined how end-user satisfaction with CPOE changes over time. Most studies use one round of data collection after CPOE implementation. Other studies have used a pre–post design to examine the impact of CPOE implementation on a variable such as perception of patient safety. But no study has examined end-user satisfaction with CPOE in the short term (eg, 3 months after CPOE implementation) and longer term (eg, 12 months after CPOE implementation). Because factors that predict end user satisfaction or acceptance of a technology (such as perceived usefulness and ease of use) change over time with greater use of the technology,^{32 36 37} it is important to examine end-user satisfaction with CPOE at different times post-implementation.

The objectives of this study are to evaluate satisfaction with CPOE among ICU nurses and ordering providers and to examine whether CPOE satisfaction changes over time.

Methods

Setting

The study was performed in four ICUs in a 400-bed rural, tertiary care teaching hospital in the northeast USA: the 24-bed adult intensive care unit, the 18-bed cardiac intensive care unit (CICU), the 38-bed neonatal ICU (NICU), and the 11-bed pediatric ICU. The electronic health record (EHR) under study was the EpicCare Inpatient Clinical System version Spring 2006 (Epic Systems, Madison, Wisconsin, USA). CPOE with basic CDS, clinical documentation (nurse, physician, and mid-level provider), pharmacy system, and the electronic medication administration record (eMAR) were implemented organization-wide on the same day. Before CPOE implementation, nurses and providers had 2 h of competency-based online training, followed by 8 h classroom training for nurses (4 h of CPOE and 4 h of eMAR) and 4 h classroom training for providers (CPOE). Physicians and mid-level providers (except for NICU staff) had been using the EHR and CPOE in the ambulatory setting for 5 or more years.

Design

We used a repeated cross-sectional study design with two rounds of data collection: 3 months post-CPOE implementation (round 1 or R1), and 1 year post-CPOE implementation (round 2 or R2).

Sample

Nurses, physicians (attendings, fellows, and residents), nurse practitioners, and physician assistants working in all four ICUs were asked to fill out the same survey for both rounds of data collection. A total of 177 respondents filled out the survey at R1 (response rate: 47%), and 220 respondents at R2 (response rate: 68%). The overall response rate was 56%. We compared CPOE satisfaction between ICU nurses and ordering providers (physicians and mid-level providers) in the ICU. Ordering providers enter orders in the CPOE technology; nurses use CPOE technology to enter verbal orders (although this is uncommon) and to review and verify orders. In the remainder of this paper we use the term providers for ordering providers.

Data collection procedures

Researchers distributed paper questionnaires to nurses and providers in the ICUs. Therefore not all nurses and providers had a chance to fill out the survey if they were not present at any of the times that surveys were distributed. Respondents returned completed surveys in locked mailboxes in each ICU conference room. The questionnaires were filled out anonymously. Participation was voluntary and the study was approved by the institutional review boards at the University of Wisconsin-Madison and the study hospital.

Questionnaire

We used a valid and reliable questionnaire to assess the effects of CPOE implementation.³⁸ The measure of end-user satisfaction with CPOE implementation was based on the Provider Order Entry User Satisfaction and Usage Survey (POESUS)¹⁷ (see Hoonakker et al³⁹ for an evaluation of POESUS). One item (#7) was removed from the original POESUS: ‘Order entry gives me the information I need to write better orders’, because nurses do not write orders. Each item was ranked on a scale from 1 (never) to 7 (always). The R1 questionnaire also included two open-ended questions: ‘What are the three things you like most about order entry (CPOE)?’ and ‘What are the three things you would like to change about order entry (CPOE) to make it better?’

Data analysis

SPSS V.18.0 and NCSS⁴⁰ were used to analyze the data. First we analyzed missing values in the survey data using Little's test.⁴¹ Missing values were completely at random. Second, we created an overall scale of end user satisfaction with CPOE based on the individual POESUS items. A scale score was created for survey participants who responded to at least 75% of the questions. We recoded the four negatively worded items (Q3, Q6, Q10, and Q12; see table 2) in order to create a user satisfaction scale ranging from 0 (lowest satisfaction) to 100 (highest satisfaction). Third, we used a general linear mixed model for repeated assessments to test differences between groups on the two different assessments of end-user satisfaction assuming independence between the repeated measures. We also included the interactions between group (nurses vs providers) and round of data collection (R1 vs R2) at scale level in this analysis. Fourth, we used a Wilcoxon ranking order test of items across time and groups and calculated a mean item different effect size (d).^{32 42 43}

Table 2.

End-user satisfaction with CPOE by group (nurses vs providers) and round of data (R1 vs R2) collection on a scale from 1 (never) to 7 (always)

	Nurses R1 (N=121)	Nurses R2 (N=163)	Providers R1 (N=54)	Providers R2 (N=57)
1. The order entry system is reliable—it does its job consistently	(2) 4.25 (1.05)	(3) 4.67 (1.16)	(1) 4.94 (1.00)	(1) 4.85 (1.27)
2. Order entry improves my productivity	(11) 3.62 (1.48)	(7) 4.47 (1.49)	(10) 4.25 (1.64)	(12) 4.3 (1.71)
3. Order entry has a negative impact on patient care*	(8) 3.82 (1.47)	(4) 4.61 (1.53)	(13) 4.11 (1.53)	(5) 4.56 (1.28)
4. Order entry reduces patient care errors	(7) 3.83 (1.10)	(11) 4.32 (1.32)	(8) 4.53 (1.16)	(6) 4.55 (1.23)
5. The order entry system is easy to use	(9) 3.80 (1.33)	(12) 4.29 (1.43)	(11) 4.22 (1.54)	(13) 4.20 (1.48)
6. Compared to paper ordering, order entry slows me down*	(15) 3.45 (1.67)	(8) 4.40 (1.77)	(15) 3.38 (1.80)	(15) 3.87 (1.97)
7. I feel I had adequate training on order entry	(13) 3.58 (1.38)	(15) 3.94 (1.51)	(3) 4.85 (1.20)	(3) 4.73 (1.56)
8. Order entry improves the quality of patient care	(11) 3.62 (1.35)	(10) 4.39 (1.33)	(5) 4.67 (1.14)	(4) 4.59 (1.46)
9. System response time on order entry is slow*	(4) 4.20 (1.31)	(4) 4.61 (1.31)	(14) 3.92 (1.48)	(10) 4.33 (1.58)
10. When I have a problem with order entry, I just ask someone for help	(1) 5.24 (1.36)	(1) 5.19 (1.25)	(2) 4.90 (1.13)	(1) 4.98 (1.49)
11. I feel that I can benefit from refresher classes on order entry*	(5) 4.09 (1.81)	(13) 4.23 (1.76)	(9) 4.33 (1.72)	(9) 4.38 (1.71)
12. When I need help on order entry, I can find it	(3) 4.22 (1.37)	(2) 4.72 (1.41)	(6) 4.62 (1.32)	(10) 4.33 (1.35)
13. Overall, order entry improves the safety of care I provide	(6) 3.90 (1.30)	(6) 4.49 (1.33)	(4) 4.71 (1.24)	(7) 4.54 (1.52)
14. Overall, order entry saves me time	(14) 3.46 (1.56)	(13) 4.23 (1.49)	(12) 4.21 (1.71)	(14) 3.94 (1.83)
15. Overall, I am satisfied with the order entry system	(9) 3.80 (1.44)	(8) 4.40 (1.40)	(7) 4.56 (1.26)	(8) 4.39 (1.58)
Scale end user satisfaction with CPOE (0–100)	48.58 (15.6)	57.78 (13.5)	56.81 (15.3)	57.26 (16.5)

Results expressed as: (rank orders of the item), mean, (SD).

*Note that items 3, 6, 9, and 11 are reversed scored.

CPOE, Computerized Provider Order Entry.

We used a thematic analysis to analyze the two open-ended questions. First, data were cleaned. For example, negatively worded comments were removed in the question about what respondents liked about CPOE. Then two researchers (PH and PC) independently categorized the comments in major categories, and the results were compared. If the two researchers did not agree on how to categorize a certain comment, this was discussed and if needed, a third researcher (TW) was consulted. Only if consensus was achieved among the team members was a category accepted.

Results

Table 1 describes the study population. A greater proportion of nurses were female as compared to providers. Nurses had longer tenure than providers (R1); and providers had more years of computer experience (R1 and R2) and more computer expertise (R1) than nurses. There were no statistically significant changes in the study population's characteristics over time.

Table 1.

Respondent characteristics by group (nurses vs prescribing providers) and round of data collection (R1: 3 months post-implementation; R2: 12 months post-implementation)

	Nurses R1 (N=121)	Nurses R2 (N=163)	Providers R1 (N=54)†,‡	Providers R2 (N=57)‡,§
Gender (female)*,**	90%	88%	33%	33%
Age (years):
≤34	43%	44%	52%	44%
35–44	24%	27%	23%	24%
45–54	30%	22%	15%	24%
≥55	4%	8%	10%	7%
Average tenure at hospital in years*	10.2	9.6	5.1	7.1
Average tenure at unit in years*	8.4	7.7	4.3	6.7
Unit
AICU	23%	28%	37%	21%
CICU	29%	29%	30%	42%
NICU	30%	30%	15%	12%
PICU	18%	13%	18%	25%
Average years of computer experience*,**	10.2	∼9.8¶	14.4	∼14.0¶
Computer expertise (1=never use it; 7=regular and expert user)*	5.1	5.5	6.0	5.9

*Differences between nurses and providers in R2 are statistically significant at p<0.05.

**Differences between nurses and providers in R2 are statistically significant at p<0.05.

†R1: 3 physician assistants, 3 nurse practitioners, 15 attendings, 10 fellows, 13 residents, 9 interns, 1 unknown.

‡Differences between providers (attendings, fellows, etc) are not statistically significant in R1 or R2.

§R2: 7 physician assistants, 4 nurse practitioners, 21 attendings, 8 fellows, 9 residents, 8 interns.

¶In R2 we changed the response category of the question. Number reflects an estimate.

AICU, adult intensive care unit; CICU, cardiac intensive care unit; NICU, neonatal intensive care unit; PICU, pediatric intensive care unit.

End-user satisfaction with CPOE (POESUS)

Table 2 describes end-user satisfaction with CPOE for the two groups (ICU nurses and providers) and the two rounds of data collection. Rank orders for the items in the questionnaire are provided: from (1) (highest/most satisfied) to (15) (lowest/least satisfied).

Results in table 2 show that nurses’ satisfaction with CPOE at R1 is below the scale mid-point (mean=48.6 on a 0–100 scale, N=121). At R1 nurses are most satisfied with getting help when having problems with order entry (Q10 and Q12) and with the reliability of the system (Q1). At R1 nurses are most dissatisfied with the time consuming effects of CPOE (Q6 and Q14) and the (lack of) training they received (Q7).

Nurses’ satisfaction increased over time: 1 year after implementation (R2), the mean of overall nurses’ satisfaction with CPOE was 56.8 on a scale from 0–100 (N=163), a significant increase from R1 (see figure 1). At R2 nurses were still most satisfied with the availability of support (Q10 and Q12), and with the reliability of the system (Q1), and most dissatisfied with the training received (Q7 and Q11). Although they still reported that CPOE did not save them time (Q14), apparently it did not slow them down as much (Q6).

Figure 1.

End user satisfaction (scale) by group (nurses vs providers) and round of data collection (R1 vs R2). POESUS, Provider Order Entry User Satisfaction and Usage Survey. This figure is only reproduced in colour in the online version.

Providers were moderately satisfied with CPOE at both 3 months (R1) and 12 months (R2) after implementation (57.8 vs 57.3 on 0–100 scale). Providers’ satisfaction did not change over time. At R1, providers were most satisfied with the reliability of the system (Q1), the availability of support (Q10), and the training on CPOE they received (Q7). They were least satisfied with the impact of CPOE on their work speed (Q6), CPOE system response time (Q9), and impact of CPOE on patient care (Q3). Although providers’ satisfaction did not change over time, the rank order of aspects of CPOE that they were (dis)satisfied with changed slightly (see figure 3 and table 2). For example, at R2 providers were much more positive about the impact of CPOE on patient care (Q3).

Figure 3.

Changes in providers’ end-user satisfaction with Computerized Provider Order Entry (CPOE) between R1 and R2 (mean item difference effect sizes).

Providers and nurses reported different levels of satisfaction with CPOE over time: we found statistically significant differences between groups (nurse and providers) (F=5.46, df=1, p<0.05), and between the two rounds of data collection (F=7.14, df=1, p<0.01). In addition, the interaction between group and round was statistically significant (F=5.68, df=1, p<0.05). At R1, providers were more satisfied with CPOE than nurses, but at R2, the difference between nurses and providers was no longer significant (see figure 1).

Results of a sub-group analysis show that there are no significant differences in overall CPOE satisfaction between the different groups of providers at R1 and R2, nor in changes in overall CPOE satisfaction over time. Results of this analysis are not reported in this paper; for details please contact the first author.

Changes in specific aspects of CPOE end-user satisfaction over time

For nurses, 12 of the 15 aspects of end-user satisfaction changed between R1 and R2 (see figure 2). Items 7, 10, and 11 did not change over time. The highest increases occurred for the impact of CPOE on improving productivity (Q2 and Q6) and the role of CPOE in improving quality of care (Q3 and Q8). The overall effect size was 0.38.

Figure 2.

Changes in nurses’ end-user satisfaction with Computerized Provider Order Entry (CPOE) between R1 and R2 (mean item difference effect sizes).

None of the differences in providers’ satisfaction with CPOE between R1 and R2 was statistically significant.

Qualitative data

A total of 130 remarks were made by 44 nurses and 33 providers in the open-ended question about the things they like about CPOE in R1. Results of a thematic analysis are shown in table 3.

Table 3 .

What are the three things you like most about order entry (CPOE), by major category and group (nurses vs providers)

Category	Nurses	Providers	Total
Easy access to patient information	14	14	28
Readability	18	5	23
Efficiency	19	4	23
Usability	8	12	20
Better quality of orders	8	5	13
Order sets	0	8	8
Safety	3	3	6
Reduced need for verbal orders	5	0	5
Better documentation	1	3	4
Total number of respondents providing comments	44	33	77
No comments given	77	21	98
Total	121	54	175

CPOE, Computerized Provider Order Entry.

Respondents most often mentioned easy access to patient information (remote ordering, always having the chart available at any computer, not having to search for a paper chart, etc), followed by readability (the legibility of the orders, the ability to read orders, etc); efficiency (the timeliness of the ordering process, the fact that orders are ‘instant’, that it saves time, etc); the usability of the system (easy to use, recommends medication dosage, automatic display in mg/kg for medications, automatic adjustment of medication for dosing weights, easy to view, etc); the better quality of the orders (more accurate and detailed description of order, more complete orders, orders includes the name of the ordering physician, etc); order sets (order sets for usual procedure, order sets streamline entry); safety (correct dosing, avoiding duplicate orders, avoiding transcription errors); the reduced need for verbal orders (minimal need for verbal orders); and better documentation (list all orders, can read everyone's notes). Results show interesting similarities and differences between nurses and providers. Overall, both nurses and providers like the easy access to patient information; nurses like the readability of the orders and the efficiency of the ordering process; and providers like the usability and the order sets.

A total of 124 remarks were made by 54 nurses and 31 providers to the open-ended question about the things they would like to change about CPOE in R1 (see results of the thematic analysis in table 4).

Table 4 .

What are the three things you would like to change about order entry (CPOE) to make it better, by major category and group (nurses vs providers)

Improvements	Nurses	Providers	Total
Usability	17	22	39
Order modification	15	0	15
Finding information	5	8	13
Interface with the lab system	8	3	11
Communication	11	0	11
Efficiency	3	4	7
Sign, hold, and release orders	6	1	7
Safety	5	2	7
Duplicate orders	2	4	6
Time adjustment of orders	6	0	6
More specificity	2	3	5
Drug alerts	0	3	3
Order sets	2	0	2
Total number of respondents providing comments	54	31	85
No comments given	67	23	90
Total	121	54	175

CPOE, Computerized Provider Order Entry.

Most often mentioned, by both nurses and providers, were usability issues (system is badly designed, too complicated or user-unfriendly, too much info displayed, important buttons not easily seen; pattern/name match isn't always making things easy to find the order) followed by order modification (it should be easier to modify orders, the ability to clean out orders if they are old or no longer apply, need to discontinue old order when a new order is written); finding information (make things easier to find, it is difficult to find specific orders, hard to find MD orders, nursing communication is hard to find, some items are only found in certain order sets, doing a separate search for them doesn't find the order); the interface with the lab (better interface with the lab system, labs are always messed up, lab sheets are confusing); communication (doctors being able to write orders and not letting the nurses know, better face-to-face communication about what is ordered, doctors and nurse practitioners must not order things without telling the nurse, notification when orders are written); sign, hold, and release orders (improve process to release signed and held orders, make signed and held orders easier to release, sign and pending order confusion); efficiency (faster processing time, system response time is slow, documenting takes a lot of time); safety (order reconciliation is hard; too many medication errors are a problem); duplicate orders (hard to prevent duplicate orders, difficult to correct duplicate orders); time adjustment of orders (adjusting times on labs when written as q4, q6, etc, adjusting times for meds/labs is time-consuming); more specificity (make it NICU specific—not standardized across the hospital, many orders are not unit-specific, more options for pediatrics to write meds mcg/kg/min, mg/hr, mcg or mg, etc, update orders or order sets specific to CICU); drug alerts (drug alerts are often overridden because they don't seem relevant and often don't get paid close attention to, too may ‘drug interactions’ warnings); and order sets (make them ‘click’ what they want, ‘not unclick’ what they don't want; order sets—patients have different needs—they don't all need the same thing). Overall, both nurses and providers would like to improve the usability of the CPOE system; nurses would like to make it easier to modify orders (including time adjustment) and would like to improve communication; while providers would like to improve the ability to find specific information in the CPOE system.

Discussion

Despite the potential benefits of CPOE, many attempts to implement CPOE systems have been challenging, and the issue of CPOE acceptance and satisfaction has emerged as critical to the success of CPOE implementation. Our research makes an important contribution by examining user satisfaction with CPOE in a group of ICU nurses and providers. In addition, we examined those perceptions in the short term (3 months) and in the long term (12 months post-CPOE implementation). It seems likely that attitudes of users could change over time, for example because they develop a better appreciation of the potential benefits, become more familiar with the technology, adapt their use of the technology, or adapt workflows to constraints that the technology imposes. Our results show that ICU clinicians are moderately satisfied with CPOE, and that the satisfaction of ICU nurses, but not ICU providers, with CPOE increased over time (see figure 1).

Differences between ICU nurses and providers at R1

There are several possible explanations for the differences between nurses and providers at R1. The CPOE system was implemented hospital-wide on the same day, with the use of CPOE mandatory. Most physicians (except NICU staff) and most mid-level providers had experience with the outpatient CPOE module, provided by the same software company as the inpatient CPOE module reported here (the outpatient CPOE implementation was completed in 2002). Inpatient nurses’ lack of experience with CPOE may account in part for their more negative perceptions of CPOE at R1. A second explanation may be related to the fact that the two groups were trained differently and on different software versions than the go-live version. Despite a similar overall curricular structure, some details of CPOE training were different for nurses and providers. Both groups were required to complete role-appropriate online education (requiring about 2 h) and a 4-h face-to-face CPOE classroom training. In addition, the nurses received 4 h face-to-face classroom training in the use of the eMAR. All curricula were developed by an interdisciplinary team of clinicians, educators, software developers, and informaticians with specific focus on specific provider and nursing tasks. Providers were more satisfied with the training received than were the nurses (Q7 and Q11).

Positive aspects of CPOE

Overall, both groups were most satisfied with the reliability of the system (Q1) and the support provided (Q10 and Q12). They were moderately satisfied about the potential of CPOE to improve the safety of care (Q13) and reduce patient care errors (Q4). The providers were more satisfied than the nurses with the training received (Q7); the potentially positive impact of order entry on patient care (Q3); and the potential to improve the quality of patient care (Q8) and reduce patient care errors (Q4). Results of the thematic analysis on aspects that users like about CPOE provide more detail about what nurses and providers like about the ordering system. Respondents most often mentioned: easy access to patient information (both nurses and providers); the readability of the orders (nurses, but also some providers); the efficiency of the system (nurses, but also some providers); the ease of use (nurses and providers); the better quality of the orders (nurses and providers); and the order sets (providers).

Negative aspects of CPOE

Our results highlight the aspects of CPOE that both groups of end-users (providers and nurses) are less satisfied with. Nurses were not satisfied with the training they received (Q7). Neither nurses nor providers were convinced that CPOE helped them to improve care processes: they did not think it increased their productivity (Q2); they did not think it saved them time (Q15); and they did think it slowed them down (Q6), although providers and especially nurses were more positive about these aspects at R2 (see figures 2 and 3). Both groups reported that CPOE was not easy to use (Q5). Results of the thematic analysis of aspects of CPOE that need improvement provide more detail about what users (both nurses and providers) would like to change about the system. Results show that respondents most often mentioned the usability of the system (nurses and providers); the ability to modify orders (nurses); finding information in the system (providers); communication (nurses); and the interface with the lab system (both nurses and providers). Shortly after CPOE implementation there were problems with the interface with the lab; but this problem was later resolved.

Changes over time

An interesting result of this study relates to differences in the perceptions of the two groups between 3 months and 1 year after CPOE implementation and how they change over time. At R1 nurses were significantly less satisfied with CPOE than at R2 and than providers. At R2 they had a comparable level of end-user satisfaction with CPOE as providers. Nurses’ satisfaction with CPOE increased on nearly every measure (see figure 2)—apart from Q7 and Q11 (training) and Q10 (getting help)—with a pooled effect size of 0.37. The largest improvements in user satisfaction—with effect sizes >0.50—were on items Q2 (Order entry increases my productivity), Q3 (Order entry has a negative impact on patient care), Q6 (Compared to paper ordering, order entry slows me down), and Q8 (Order entry improves the quality of patient care). Overall satisfaction of providers did not change significantly over time.

Addressing CPOE satisfaction

A comparison of our results with results of the Veterans Administration (VA) system, using the same questionnaire,¹⁷ shows that VA users were more satisfied with their CPOE system than respondents in our study.³⁹ On the other hand, a comparison with a study conducted by Wilson et al⁴⁴ on the Department of Defense (DOD)'s Composite Health Care System, also using the same questionnaire, showed that both groups in this ICU study were significantly more satisfied with CPOE than the providers in the hospital and outpatient clinics of two military facilities. These differences may be due to organizational differences (VA vs DOD vs a tertiary-care hospital), differences in the settings studied (ICUs, general hospital, or outpatient clinics) or the health information technology system studied. By using the same questionnaire, we are able to compare results across organizations and systems.

Therefore, in future research on end-user satisfaction with CPOE, we recommend: (1) that all actors involved in the ordering process (physicians, mid-level providers, nurses, clerks, and pharmacists) are surveyed; (2) that the questionnaire used in this study and the VA and DOD studies is used; and (3) that end-user satisfaction is examined over time.

Study limitations

The repeated cross-sectional design of the study caused several study limitations, and there is no way to address these issues. Only a study with a truly longitudinal design can resolve the issues. First, there is probably some dependency in the nurses’ samples at R1 and R2, meaning that the same nurses have filled out the questionnaire in both rounds of data collection. This means that the way some nurses have responded to the questionnaire survey in R2, can have been influenced by the way they responded to the questionnaire in R1, 9 months earlier. Second, there is probably some independency between the ordering providers’ samples. Some of the respondents in the provider sample, specifically the attendings, fellows, physician assistants, and nurse practitioners in R1 are probably the same respondents as in R2, causing the same dependency problem as in the nurses sample. However, another part of the provider sample is probably different in R1 and R2: in particular the year 2 and year 3 residents, and the interns (first year residents). All residents rotate through the units. This means that up to a certain degree the providers’ samples in R1 and R2 are independent and that the average scores on the questionnaire can be influenced by either the time period that the providers have been using CPOE or by differences in the sample composition in R1 and R2. Further, in R1 (3 months post-implementation), nine interns (first year residents) filled out the questionnaire. All nine interns were working at the hospital when the EHR/CPOE system went live, and filled out the questionnaire 3 months afterwards. However, in the second round of data collection (R2, 12 months post-implementation), not all interns had worked with the CPOE system for 12 months. Some interns may have had experience with an EHR/CPOE system during their years as medical students, for example during clerkships and preceptorships.^{45 46} Results of additional analysis that compared the providers with and without interns did not show any statistically significant differences. A third limitation of the study is that the samples of providers in R1 and R2 were rather small, and consist of different job positions (interns, residents, fellows, attendings, nurse practitioners, and physician assistants). However, all providers can enter orders into CPOE. To summarize: some caution is needed with the conclusion that user satisfaction with CPOE did not change over time for providers, but that it did improve significantly for nurses.

Conclusion

Results of this study show: (1) significant differences in end-user satisfaction with CPOE between groups of users (nurses and providers); and (2) changes over time in end-user satisfaction with CPOE. This suggests that if end-user satisfaction with health information technology and CPOE specifically is only measured among certain users or at only one point in time, results may be deceptive.