ICU nurses' acceptance of electronic health records

Abstract

Objective

To assess intensive care unit (ICU) nurses' acceptance of electronic health records (EHR) technology and examine the relationship between EHR design, implementation factors, and nurse acceptance.

Design

The authors analyzed data from two cross-sectional survey questionnaires distributed to nurses working in four ICUs at a northeastern US regional medical center, 3 months and 12 months after EHR implementation.

Measurements

Survey items were drawn from established instruments used to measure EHR acceptance and usability, and the usefulness of three EHR functionalities, specifically computerized provider order entry (CPOE), the electronic medication administration record (eMAR), and a nursing documentation flowsheet.

Results

On average, ICU nurses were more accepting of the EHR at 12 months as compared to 3 months. They also perceived the EHR as being more usable and both CPOE and eMAR as being more useful. Multivariate hierarchical modeling indicated that EHR usability and CPOE usefulness predicted EHR acceptance at both 3 and 12 months. At 3 months postimplementation, eMAR usefulness predicted EHR acceptance, but its effect disappeared at 12 months. Nursing flowsheet usefulness predicted EHR acceptance but only at 12 months.

Conclusion

As the push toward implementation of EHR technology continues, more hospitals will face issues related to acceptance of EHR technology by staff caring for critically ill patients. This research suggests that factors related to technology design have strong effects on acceptance, even 1 year following the EHR implementation.

Introduction

The push toward implementation of electronic health records (EHR) has raised issues related to the acceptance of the technology.¹ This is particularly important in intensive care units (ICUs) where physicians and nurses experience high workload,^{2 3} patient care is critical and complex,^{4 5} decisions often need to be made quickly, and interventions must be implemented in a timely manner.⁴ Any change in the work system of ICUs such as the implementation of EHR technology can have important consequences for providers as well as patients.⁶ Understanding ICU staff perceptions of the EHR technology and its implementation can help EHR designers and implementers in their continuous effort to improve the design, implementation, and use of the technology. In this study, we report data on EHR acceptance by ICU nurses and analyze factors related to design and implementation of the technology that can contribute to acceptance.

Background

The implementation and use of EHR technology have raised numerous challenges, including end-user acceptance.⁷ In complex healthcare environments such as ICUs, it is important to understand end-user perceptions of the usability, usefulness, and acceptance of the technology.⁴ Most research on EHR acceptance has focused on physicians^{8 9}; less is known about nurses' acceptance of EHR technology¹⁰ and its different functionalities—for example, the electronic medication administration record (eMAR) or order entry. Some challenges to acceptance and use may be temporary and visible only during the short-term adaptation phase that immediately follows implementation. Other challenges to acceptance and use may arise only after extended use of the technology by users.¹¹ Therefore, we need to examine EHR acceptance at multiple periods of time.^{12 13}

Studies of EHR acceptance have rarely addressed ICU providers, and particularly nurses.¹⁰ A small-scale study of nurses' perceptions of EHR was limited to 46 nurses in one medical-surgical unit and one ICU.¹⁴ Most nurses (96%) preferred the EHR to paper patient records because it provided enhanced access to patient information, facilitated documentation and information retrieval, and improved organization of work. Because use of the EHR technology by ICU nurses participating in our study is mandatory, continued acceptance of the technology is important¹⁵; if nurses find the EHR technology neither usable nor useful and develop negative perceptions of and attitudes toward the EHR technology, it may be difficult to engage them continuously in using the full features of the system and in learning new features of it. Similar negative consequences have been documented in the context of mandatory usage by nurses of bar coding medication administration technology¹⁶ and smart intravenous pump technology.^{17 18} In this context of mandatory technology usage, perceptual measures of acceptance and use are therefore critical as system use data are unlikely to provide information about the system quality.

Various models of technology acceptance have been proposed.^{7 19} In this study, we draw on two bodies of knowledge to examine EHR acceptance among ICU nurses. First, according to Nielsen,²⁰ technology acceptance is influenced by (1) the usability of the technology (ie, ‘how well users can use the technology functionalities’) and (2) the utility or usefulness of the technology (ie, ‘whether the functionalities of the technology can do what is needed’). (The concepts of usability and usefulness/utility as proposed by Nielsen¹² are respectively similar to the concepts of perceived ease of use and perceived usefulness of the Technology Acceptance Model.¹³) We assume that ICU nurses' acceptance of the EHR technology is influenced by the usability and usefulness of the technology. We asked ICU nurses to evaluate the EHR usability, as well as the usefulness of three EHR functionalities, specifically computerized provider order entry (CPOE), eMAR, and a nursing documentation flowsheet. Second, the manner in which the technology is implemented can influence end-user satisfaction and acceptance.^{19 21–23} An extensive review of organizational design and management literature shows how characteristics of the technological change process can affect acceptance of the technology.²⁴ A key characteristic of the change process is the ability of end users to participate effectively in the implementation process.^24–26 When end users are given a chance to provide input into the design and implementation of the technology, they are more likely to accept and use it.²⁵

The EHR has been defined in many different ways.²⁷ Because EHR technology can include various functionalities, it is important to examine each of them specifically. In this study, we assess ICU nurses' perceptions of the usefulness of three EHR functionalities: CPOE, eMAR, and nursing documentation flowsheets. Our research question is: Do implementation method, technology usability, and usefulness affect nurses' acceptance of the EHR?

Methods

Data

This research is part of a larger study investigating the impact of EHR on the work of end users and various outcomes in four ICUs of a regional medical center (http://www.cqpi.engr.wisc.edu/cpoe_home). The EHR product studied was the EpicCare Inpatient Clinical System version Spring 2006 (Epic Systems, Madison, Wisconsin). Several functionalities of the EHR were implemented concurrently in October 2007, including CPOE and eMAR. The nurses must use CPOE to review and sign off on entered orders. Verbal orders are entered by nurses but are uncommon. The nurses use the eMAR to review and document medication administration, timing and comments about the administration. The eMAR uses color coding to let nurses know that a medication is currently due or overdue. The nursing flowsheet functionality, in which nurses record information such as vital signs, patient symptoms, and patient care performed, was implemented before the beginning of the study (June 2005), except in the neonatal ICU where it was implemented with the EHR. The EHR system was optimized for each specific ICU. Based on feedback from ICU clinicians (see below the list of implementation activities in which ICU nurses participated) and analysis of ICU care processes performed by the IT team, tools for information display (eg, ‘accordion report’ for presenting complex ICU patient data) were developed and provided coherent views of the complex data typically generated in the care of ICU patients. After implementation, the system evolved incrementally, as care processes, the EHR, and user knowledge, skills, and behaviors were improved. Nurses received 10 h of required competency-based training before implementation. In this paper, we analyze data from two cross-sectional surveys conducted after the October 2007 EHR implementation.

The research site is a 400-bed rural, tertiary care medical center located in the northeastern USA. Nurses in four ICUs were asked to participate in the study: (1) the Adult ICU, which is a 24-bed medical/surgical shock/trauma unit, (2) the 18-bed medical/surgical Cardiac ICU, (3) the 38-bed Neonatal ICU, and (4) the 11-bed Pediatric ICU.

In the two rounds of survey data collection, ICU nurses were invited to complete a questionnaire. Respondents were recruited through multiple means including posters about the study that were displayed in the unit, meetings in each unit describing the study and encouraging participation, email announcements of data collection, and the scheduling of specific dates and times when researchers came to the unit and distributed surveys. Surveys were returned by respondents to a locked mailbox in the unit break room. Researchers went to each ICU several times to continue recruiting participants and to distribute surveys to staff with varied work schedules. Data were collected 3 months (January–February 2008) and 1 year (October 2008) after EHR implementation. Participation was voluntary, and the study was approved by the institutional review boards at the university and the study hospital. The response rate for the 3-month data collection was 51% (121 participating nurses out of 237 eligible participants), while the response rate at 12 months was 72% (161 participating nurses out of 224 eligible participants). The different response rates can be explained by the time and resources involved in the distribution and collection of surveys in each round. For instance, at 12 months, we organized a greater number of meetings in each ICU during which we described the study and more actively recruited nurses for participation in the survey.

Variables

The paper questionnaire included items from established instruments to measure technology acceptance, EHR usability, and EHR usefulness.^{12 29 30} Several questions about the characteristics of respondents were also included, such as their level of participation in implementation and training activities. The questionnaire was pilot-tested with nine end users from nursing units of the medical center other than the ICUs. The objectives of the pilot test were (1) to evaluate the length of the survey and to measure the time needed to complete it; (2) to assess the flow and order of questions; (3) to ensure that the new questions on implementation activities were clear to the respondents; and (4) to gather input from nurses about effective methods for distributing and collecting surveys. In response to pilot test feedback, we eliminated several questions to make the survey shorter, changed the response categories for two questions, clarified the terms referring to specific implementation activities, and refined our recruitment strategy.

A single 10-point Likert-scale item measured the respondents’ overall acceptance of the EHR technology, ranging from (1) ‘dislike very much and don't want to use’ to (10) ‘like very much and eager to use.’ We elected not to use EHR system-usage data for several reasons. First, nurses have to use the EHR technology; therefore, system-usage data may be insensitive to capture the positive and/or negative features of the EHR technology. Second, system-usage data are complex to analyze and interpret.^{31 32} In particular, recent information systems research indicates the need to examine multiple levels of system use data and their interactions.^{33 34} Our study focuses on individual nurses' attitudes toward and perceptions of the EHR technology, which are an important focus of research.^{31 32}

Perceived usability measures were selected from the Questionnaire for User Interface Satisfaction (QUIS).³⁰ The seven items of the QUIS scale of learning were combined into a measure of the overall EHR usability by calculating the average response and rescaling it to range from 0 (negative) to 100 (positive).

Perceptions of usefulness were measured for CPOE, the eMAR, and the nursing flowsheet. The items for each functionality were combined into a multi-item scale indicating the perceived usefulness of that functionality. The acceptance and usefulness scales have been used in previous research.^{12 36 37}

In the 3-month postimplementation round, the survey included items on the information received by the end users about EHR implementation and their inputs in decision-making regarding EHR implementation. These items used semantic differential response categories such as ‘vague–precise’ (for information provided to users about the implementation), ‘meaningless–meaningful’ and ‘non-productive–productive’ (for users' opportunities for input) and ‘insufficient–sufficient’ and ‘useless–useful’ (for both). The items were drawn from published literature³⁸ and have been used in previous research.^{12 37 39} Another set of items asked respondents to indicate whether they participated in 11 EHR implementation activities, such as the health-system-level team, strategic design team, the nurse feedback team, the operations managers' meetings, and the project oversight committee. The implementation activities also included two prospective human factors assessments that were conducted as part of the larger research project, that is, usability evaluation and a proactive risk assessment. The other EHR implementation activities were a multidisciplinary feedback group, departmental meetings, pilot testing, and regulatory reviews.

Questions capturing survey respondent demographics and other characteristics include age, ethnicity, race, clinical work unit, the number of hours typically worked each week, the shift typically worked, years spent working for this hospital, and years spent working in the current ICU. Respondents were also asked to estimate their number of years of computer experience and their years of experience working with the outpatient EHR (which has been used throughout the health system since 2002).

Analyses

Descriptive analyses were performed to assess nurses' EHR acceptance, the perceived usability and usefulness of the EHR functionalities, and perceptions of the EHR implementation process. Because data were collected with a repeated cross-sectional design, longitudinal analyses could not be performed. Instead, overtime comparisons indicate whether the average responses of nurses differed significantly between short-term (3 months postimplementation) and long-term (12 months postimplementation). Missing data were analyzed using Little's MCAR test⁴⁰ at each time period. Results indicated that data at 3 months postimplementation were missing 2.95%, with the Little test (χ²=29.12, df=28, p=0.406), and data at 12 months were missing 2.27% with the Little test (χ²=23.56, df=23, p=0.428). Tests indicate that the missing data were random, and no imputation was required.

The model analysis strategy concentrates on covariate influence over time. The analysis of repeated semicross-sectional data from this study poses the potential problem of heteroscedasticity (the violation of the assumption that all residuals are homoskedastic, or have the same variance). This violation would result in biased estimates. To assess the issue of heteroscedasticity in our repeated cross-sectional data, we conducted a two-level hierarchical model for the repeated outcomes.^{41 42} The model was written as a two-level hierarchical structure, where EHR acceptance is a function of various covariates.

The initial intercepts were modeled as random variables. We treated our two time periods as a repetition at level 1 (indicated by t) nested within nurses (indicated by i). Let z_t be a dummy variable of indicator values for each nurse (i), Z_1i=1 if t=3 months and 0 otherwise, Z_2i=1 if t=12 months and 0 otherwise.

A general multivariate hierarchical model was considered, with the model for these data written as

$${\text{Y}}_{\text{ti}}={\displaystyle \sum _{\text{t}=1}^2{\beta }_{0,\text{t}}{\text{Z}}_{\text{ti}}}+{\displaystyle \sum _{\text{t}=1}^2{\displaystyle \sum _{\text{h}=1}^{\text{H}}{\beta }_{\text{h,t}}{\text{Z}}_{\text{ti}}{\text{x}}_{\text{h,ti}}}+}{\displaystyle \sum _{\text{t}=1}^2{\mu }_{\text{t}}{\text{Z}}_{\text{ti}}+{\text{e}}_{\text{ti}}}$$ (1)

where x_h,ti indicates the covariates (eg, level of computer experience). To assess the issue of heteroscedasticity in our repeated cross-sectional data, we conducted a two-level hierarchical model for the repeated outcomes.

The nurses worked on one of the four ICUs at the two time periods, that is, 3 months and 12 months. Because the analysis initially assumes nurse independence, we assessed the degree of within-ICU dependency. The existence of a non-zero intraclass-ICU correlation coefficient would indicate non-independence in nursing response, which would result in artificially reducing the estimated SE, and bias the interpretation of the significance of a parameter. To assess a cut-off level, we estimated the design effect, which is based on the intra-ICU correlation (1+(average unit size−1)×intraclass correlation). Simulation studies⁴³ have shown that if the estimated design effect is less than 2.00, the bias in SEs due to within-dependency is minimal and does not require adjusting. Finally, parameter contrasts between 3 months and 12 months were conducted based on procedures suggested by Goldstein.⁴¹

Results

Description of the sample

Characteristics of the respondents are described in table 1. The respondents in the two rounds differed significantly only in their level of computer experience: nurses in the 12-month round had more computer experience. This difference is controlled for in the acceptance model by including years of computer experience as a covariate.

Table 1.

Respondent characteristics in the 3-month and 12-month surveys

	3 months post (n=121)	12 months post (n=161)
Unit
Adult ICU	28 (23%)	45 (28%)
Cardiac ICU	35 (29%)	46 (29%)
Pediatric ICU	22 (18%)	21 (13%)
Neonatal ICU	36 (30%)	49 (30%)
Age
≤34	49 (43%)	69 (44%)
35–44	28 (24%)	42 (27%)
45–54	34 (30%)	34 (22%)
55+	4 (4%)	12 (8%)
Experience with present employer
<1 year	11 (9%)	22 (14%)
1–5 years	48 (40%)	68 (42%)
6–15 years	22 (18%)	22 (14%)
>15 years	38 (32%)	49 (30%)
Experience in current ICU
<1 year	17 (14%)	28 (17%)
1–5 years	49 (41%)	68 (42%)
6–15 years	24 (20%)	27 (17%)
>15 years	28 (24%)	38 (24%)
Experience with EHR/CPOE in outpatient clinics of this health system
None or very little	95 (80%)	128 (80%)
A little	13 (11%)	15 (9%)
Moderate amount	7 (6%)	4 (2%)
Much	1 (1%)	6 (4%)
Very much	3 (2%)	7 (4%)
General computer experience*
≤2 years	40 (33%)	35 (22%)
3–9 years	65 (56%)	93 (58%)
≥10 years	13 (11%)	32 (20%)
No of implementation activities involved in
0	79 (70%)
1	23 (21%)
≥2	10 (9%)

^*Significant difference between the two rounds at p<0.05.

Descriptive analyses

Table 2 provides information about basic statistics, Cronbach-α scores and correlations for the study variables. The scales of perceptions of EHR implementation, EHR usability, and usefulness of the three functionalities have a high internal consistency, with Cronbach α scores ranging from 0.90 for usefulness of CPOE to 0.98 for inputs into decisions regarding EHR implementation (see table 2). A factor analysis performed on the usability and usefulness questions showed that the items for each scale load onto single factors that account for between 70% (usability) and 92% (eMAR usefulness) of the total variance.²⁹

Table 2.

Reliabilities (α) for scale variables, correlations (r), means and SD (3 months: n=121; 12 months: n=161)

	Mean (SD)	EHR usability	CPOE usefulness	eMAR usefulness	Nursing flowsheet usefulness	Information about EHR implementation	Inputs in decisions regarding EHR implementation
Reliability of scales		α=0.928	α=0.904	α=0.956	α=0.924	α=0.939	α=0.985
3 months post-EHR implementation
EHR acceptance	59.10 (25.91)	r=0.718	r=0.674	r=0.626	r=0.577	r=0.514	r=0.462
EHR usability	49.82 (20.88)		r=0.683	r=0.577	r=0.598	r=0.486	r=0.418
CPOE usefulness	48.08 (24.07)			r=0.622	r=0.524	r=0.496	r=0.398
eMAR usefulness	64.01 (25.91)				r=0.601	r=0.396	r=0.282
Nursing flowsheet usefulness	66.62 (23.81)					r=0.360	r=0.304
Information about EHR implementation	56.70 (24.68)						r=0.577
Inputs in decisions regarding EHR implementation	41.56 (28.73)
12 months post-EHR implementation
EHR acceptance	65.62 (23.90)	r=0.767	r=0.705	r=0.623	r=0.698	–	–
EHR usability	56.41 (20.55)		r=0.674	r=0.551	r=0.582	–	–
CPOE usefulness	56.34 (24.14)			r=0.562	r=0.510	–	–
eMAR usefulness	70.99 (22.32)				r=0.684	–	–
Nursing flowsheet usefulness	70.37 (22.50)

CPOE, computerized provider order entry; eMAR, electronic medication administration record; EHR, electronic health records.

All correlations are significant at the p=0.01 level.

Table 3 shows nurses' perceptions of (1) the information they received about the EHR implementation and (2) the significance of their inputs into implementation decision-making. Nurses' perceptions of the information tend to be slightly positive, as indicated by the mean scale value of 56.7 (from a potential range of 0 to 100). Their ratings of individual scale items are very similar for each measure, ranging from an average of 4.64 for the timeliness of the information to 4.16 for the accuracy of the information (on a scale of 1–7). In contrast, nurses' assessments of the significance of their inputs into decision-making were slightly negative (mean scale value of 41.6). Again, the ratings are similar across the six questions of the scale, ranging from an average of 3.63 for timeliness to 3.38 for meaningfulness. Seventy-one percent of ICU nurses surveyed at 3 months indicated that they never participated in any of the 11 implementation activities. The most frequently reported implementation activities in which ICU nurses participated are nursing feedback team (23%) and departmental meetings (14%).

Table 3.

Nurse perceptions of electronic health records (EHR) implementation (3-month data), n=119

	Mean (SD)
Information received about EHR implementation
Insufficient (1)–sufficient (7)	4.59 (1.55)
Incomplete (1)–complete (7)	4.54 (1.52)
Non-timely (1)–timely (7)	4.64 (1.55)
Vague (1)–precise (7)	4.16 (1.67)
Useless (1)–useful (7)	4.48 (1.61)
Scale of Information on EHR implementation	56.70 (24.68)
User inputs in decision-making on EHR implementation
Non-timely (1)–timely (7)	3.63 (1.74)
Insufficient (1)–sufficient (7)	3.47 (1.74)
Useless (1)–useful (7)	3.50 (1.79)
Meaningless (1)–meaningful (7)	3.38 (1.75)
Bad/poor (1)–good (7)	3.48 (1.72)
Non-productive (1)–productive (7)	3.51 (1.69)
Scale of Inputs in EHR implementation	41.56 (28.73)

The numbers in parentheses indicate the scores assigned to end-points for each question. Multi-item scales values were calculated by combining items, calculating the average response, and rescaling it to range from 1 (negative) to 100 (positive). Data presented in this table include 119 of the total 121 intensive care unit nurses who participated in the 3-month survey; two intensive care unit nurses did not answer these questions on implementation.

Overtime comparisons of nurses' assessments of the acceptance, usability, and usefulness of the EHR technology can be found in figure 1 and table 4. On average, nurses' perceptions were more positive at 12 months than at 3 months. Overall EHR acceptance was slightly positive at 3 months (mean 6.32 on a 10-point scale) and significantly higher at 12 months (mean 6.91). (Note that in figure 1, EHR acceptance was rescaled to range from 0 to 100 to permit comparison with the usability and usefulness scales.) The patterns for usability and usefulness are similar. At 3 months after implementation, nurses had nearly neutral assessments of EHR usability (mean scale value of 49.8 on a 100-point scale), while 1 year after implementation they had slightly positive perceptions (mean scale value of 56.4). The EHR usability items whose average scores significantly improved in this period were related to the straightforwardness of task performance, the helpfulness of screen messages, the degree to which users' needs are taken into consideration, and the ease of correcting mistakes. The usability item related to help messages on screen received the lowest score (mean=4.80 at 3 months) but significantly improved at 12 months. The EHR usability items related to learning to operate the system, exploring new features, and ability to correct one's mistakes received the highest scores (means=6.44, 6.31 and 6.44 respectively at 12 months).

Figure 1.

Comparisons of electronic health records (EHR) acceptance, usability and EHR functionality usefulness at 3 months and 12 months after EHR implementation. CPOE, computerized provider order entry; eMAR, electronic medication administration record.

Table 4.

Perceived acceptance, usability, and usefulness of the electronic health records (EHR) technology by intensive care unit nurses (mean (SD))

	3 months, n=121	12 months, n=160	p Value
EHR acceptance
Please check the box that best reflects your acceptance of the EHR: dislike very much and don't want to use (1)–like very much and eager to use (10)	6.32 (2.33)	6.91 (2.15)	0.031*
Value of EHR usability	49.82 (20.88)	56.41 (20.55)	0.009**
Learning to operate the system: difficult (1)–easy (10)	5.95 (2.17)	6.44 (2.20)	0.064
Exploring new features by trial and error: difficult (1)–easy (10)	5.89 (2.28)	6.31 (2.15)	0.116
Remembering names and use of commands: difficult (1)–easy (10)	5.53 (2.26)	5.90 (2.15)	0.167
Tasks can be performed in a straightforward manner: never (1)–always (10)	5.36 (2.24)	6.01 (2.16)	0.015*
Help messages on screen: unhelpful (1)–helpful (10)	4.80 (2.24)	5.60 (2.40)	0.005**
Experienced and inexperienced users' needs are taken into consideration: never (1)–always (10)	5.28 (2.36)	5.83 (2.22)	0.048*
Correcting your mistakes: difficult (1)–easy (10)	5.60 (2.39)	6.44 (2.25)	0.003**
Scale of CPOE usefulness	48.08 (24.07)	56.34 (24.14)	0.007**
CPOE: difficult (0)–easy (9)	4.38 (2.47)	5.19 (2.38)	0.009**
CPOE: frustrating (0)–satisfying (9)	3.88 (2.56)	4.80 (2.34)	0.004**
Order entry (CPOE) functions as I expect: never (0)–always (9)	4.70 (2.03)	5.15 (2.13)	0.091
Scale of eMAR usefulness	64.01 (25.91)	70.99 (22.32)	0.017*
eMAR: difficult (0)–easy (9)	5.91 (2.35)	6.65 (1.99)	0.005**
eMAR: frustrating (0)–satisfying (9)	5.54 (2.57)	6.14 (2.29)	0.042*
The electronic medication administration record (eMAR) functions as I expect: never (0)–always (9)	5.83 (2.37)	6.37 (2.03)	0.044*
Scale of nursing flowsheet usefulness	66.62 (23.81)	70.37 (22.50)	0.181
Nursing flowsheet: difficult (0)–easy (9)	6.25 (2.18)	6.53 (2.00)	0.263
Nursing flowsheet: frustrating (0)–satisfying (9)	5.73 (2.30)	6.10 (2.26)	0.177

The numbers in parentheses indicate the scores assigned to end-points for each question. Multi-item scales values were calculated by combining items, calculating the average response, and rescaling it to range from 1 (negative) to 100 (positive).

Significant differences between the two rounds at *p<0.05, **p<0.01.

CPOE, computerized provider order entry; eMAR, electronic medication administration record.

On average, nurses viewed the eMAR and CPOE functionalities as easier and more satisfying to use at 12 months after implementation. The nurses also responded more frequently that the eMAR functions in ways they expect at 12 months. Their perceptions of nursing flowsheet usefulness did not change, perhaps because many of the nurses had been using this system for over 2 years prior to EHR implementation. To test whether the relatively higher level of nurses' computer experience at 12 months was driving the findings of overtime differences in EHR acceptance, EHR usability, CPOE usefulness, and eMAR usefulness, we ran regression analyses (not shown but available from the authors) on variables transformed to correct for skewed distributions. We found that the amount of time since EHR implementation significantly predicted EHR acceptance, EHR usability, and the usefulness of CPOE and eMAR, even when the effect of computer experience was held constant in the model.

Multivariate hierarchical model

The results of the model predicting the level of EHR acceptance in each round can be found in table 5; a graphical representation of the model showing the relationship between EHR acceptance and EHR usability and usefulness is displayed in figure 2. The estimated dependency for ICU based on the design effect was 1.86, indicating no adjustment is required (ICC=0.03, average unit size of 29.75). At both 3 months and 12 months, years of computer experience, work unit, and the respondent's rating of EHR usability and of CPOE usefulness were significant predictors of EHR acceptance. At 3 months postimplementation (but not at 12 months), the rating of eMAR usefulness was also a significant predictor. At 12 months after EHR implementation (but not at 3 months), the rating of nursing flowsheet usefulness was a significant predictor. Time contrasts indicate that the effects of only two predictors, years of computer experience and perceptions of eMAR usefulness, changed significantly between 3 and 12 months. The effect of computer experience on EHR acceptance increased, while the effect of eMAR usefulness declined substantially. The variance estimates indicate that EHR acceptance had more variability at 3 months than at 12 months relative to the predicted variables, but was not significantly different, providing support for the assumption of homoscedasticity. Finally, the R² estimates indicate that more of the variance in EHR acceptance is explained by the 12-month equation.

Table 5.

Model predicting electronic health records (EHR) acceptance by intensive care unit (ICU) nurses, 3 months and 12 months after EHR implementation (3 months: n=121; 12 months: n=161)

	3 months	12 months	Time contrast (3 months–12 months)§
	Coefficient (SE)	Coefficient (SE)	Δ	χ2	p value	Δ95% CI
Fixed effects
Intercept	65.801*** (3.510)	67.207*** (2.276)	1.406	0.113	0.736	−6.791 to 9.603
Years of computer experience	0.534* (0.282)	3.579*** (1.000)	3.045	8.593	<0.001***	1.01 to 5.08
Cardiac ICU‡	−8.970* (4.651)	−0.763 (2.963)	8.207	2.215	0.136	−2.60 to 19.014
Pediatric ICU‡	−8.440* (5.004)	−8.443** (3.842)	−0.003	0.001	0.974	−12.36 to 12.35
Neonatal ICU‡	−4.373 (4.793)	−3.067 (3.3218)	1.306	0.050	0.823	−10.12 to 12.74
Key dependent variables
EHR usability	0.426*** (0.120)	0.400*** (0.102)	−0.026	0.027	0.869	−0.334 to 0.282
Perceived usefulness of computerized provider order entry	0.212* (0.108)	0.274*** (0.070)	0.062	0.231	0.630	−0.019 to 0.314
Perceived usefulness of electronic medication administration record	0.275*** (0.093)	0.003 (0.082)	−0.273	4.847	0.027**	−0.516 to −0.003
Perceived usefulness of nursing flowsheet	0.066 (0.094)	0.264*** (0.075)	0.198	2.690	0.100*	−0.039 to 0.435
Random effects†
Variance estimate	240.13*** (34.66)	166.15*** (20.07)	−73.98	3.412	0.064*	−152.46 to 4.50
R2	0.54	0.67

*p<0.10, **p<0.05, ***p<0.01.

^†These variables were grand-mean-centered.

^‡The comparator group is the Adult ICU.

^§The time contrasts are about comparisons of the average responses of nurses between 3 months and 12 months postimplementation.

Figure 2.

Model of electronic health records (EHR) acceptance. CPOE, computerized provider order entry; eMAR, electronic medication administration record.

Discussion

On average, ICU nurses' acceptance of the EHR technology was rather positive and improved over time. Their average perceptions of EHR usability and the usefulness of CPOE, eMAR, and nursing flowsheets also improved over time. Although the data do not permit longitudinal analyses to be performed, the results suggest that these average improvements in acceptance, usability, and usefulness may be related to the ‘learning curve’ effect. That is, over time (from 3 to 12 months), ICU nurses become more familiar with the EHR technology and its various functionalities because of their increasing use of and exposure to the technology. Nurses may also discover ways that the EHR technology can help them in performing their work, and ways to incorporate EHR use into their workflow or workarounds when using the EHR that improve their workflow efficiency. In addition, during this period of time, physicians may become more proficient at using the EHR, in particular the CPOE functionality, which may reduce the nurses' need to contact physicians about specific orders. Also, post-EHR implementation, the medical center continued to invest in EHR optimization activities. Nurses' inputs regarding improvements in the design and use of the EHR technology were elicited from a focus group of nursing-unit leaders that met weekly.

The improvement in ICU nurses' EHR acceptance found in this study is similar to the improvement in nurses' acceptance of Smart infusion pump technology found in a study of nurses in one academic medical center¹²: nurses' acceptance of the Smart infusion pump technology was positive (means varying from 6.53 to 7.20 on 10-point scales), and significantly increased between 6 weeks and 1 year after implementation. This improvement was accompanied by numerous increases in the perceived usability of the Smart infusion pump technology, similar to findings in this study.

Only about 30% of ICU nurses participated in at least one of 11 implementation activities, mostly in feedback teams and departmental meetings. This may explain why ICU nurses were more likely to report positive opinions about information received about EHR implementation than about their inputs into decision-making related to EHR implementation. Similar findings have been reported in studies of other technologies, such as Smart infusion pumps.¹² EHR implementers should be encouraged to involve nurses in EHR design and implementation activities; participation in those activities can help to improve nurses' perceptions of information received about the implementation and inputs in decisions about the implementation.²⁴ Nurses' participation in the implementation process can also help them to better understand the technology and its functionalities, and their integration in their daily work. Effective and efficient participatory methods (eg, semistructured feedback sessions during scheduled break time) need to be developed to facilitate and support nurses' involvement; this is particularly important for ICU nurses who may not be able to ‘step away’ from patient care.

ICU nurses' perception of overall EHR usability and CPOE usefulness were consistent predictors of EHR acceptance at 3 and 12 months postimplementation. The continued impact of EHR usability and CPOE usefulness on ICU nurses' EHR acceptance emphasizes the need for sustained attention to the design of EHR technology even after implementation.^{44 45} Because CPOE changes the work of physicians and mid-level providers, that is, the primary intended users of CPOE, changes in nursing workflows enabled by ordering providers' use of CPOE may influence nurses' perceptions of the EHR. One important change is that CPOE requires providers to specify many aspects of orders that were previously specified by nurses, pharmacists, and others. Therefore, nurses may feel less burdened by the need to interpret orders and spend time contacting physicians to clarify orders.

Another change is that providers can enter orders from other locations in the hospital and elsewhere, making verbal orders less necessary, which anecdotal evidence affirms. Physicians are the primary users of the CPOE functionality, and their work was significantly influenced by CPOE implementation.⁴⁶ Such changes in the work of physicians may indirectly affect nurses' perceptions of CPOE usefulness and their acceptance of EHR technology by making nurses' work more manageable, in both the short- and long term.

Nurses' perceptions of the usefulness of the eMAR and nursing flowsheet had varying impacts on EHR acceptance. At 3 months, nurses who perceived the eMAR as useful were more likely to accept the EHR technology and more eager to use it; however, this was not the case at 12 months. Implementation of the eMAR was a significant change for nurses, who previously used a paper MAR. Therefore, we would expect nurses who perceive the eMAR as useful at 3 months would be more likely to accept the EHR technology and more eager to use it. After a short period of use, nurses understand the benefits of the eMAR functionality, such as access to up-to-date information about medication administrations. This short-term effect of eMAR usefulness on EHR acceptance may disappear at 12 months because, after a transition period, ICU nurses are used to the eMAR, and its usefulness is no longer relevant for their acceptance of the technology; other EHR functionalities may become important over time and influence acceptance. (It is important to note that EHR usability remains a consistent predictor of EHR acceptance over time.) In contrast, nurses' perception of the usefulness of the nursing flowsheet influenced EHR acceptance at 12 months postimplementation but not at 3 months. The nursing flowsheet is a more dynamic functionality of EHR technology. After EHR implementation, iterative changes were made to care processes and the flowsheets that support those processes (eg, addition of flowsheet rows to be completed by nurses), potentially accounting for some of the importance of perceived usefulness of the nursing flowsheets as a determinant of EHR acceptance in the long term (12 months postimplementation). The average perceived usefulness of the nursing flowsheet did not improve between 3 and 12 months after EHR implementation; however, the usefulness ratings are fairly similar to that of the eMAR usefulness.

Study limitations include the sample, as ICU nurses participating in the study came from only one institution that invested significant resources in the design and implementation of the EHR technology. However, this design allowed a deep understanding of the implementation, the characteristics of the EHR technology, and their influence on ICU nurses. The nurses answering the 12-month survey had more computer experience than the nurses who participated in the 3-month post-EHR implementation survey. Given the different response rates at 3 and 12 months, it is possible that nurses who had more computer experience were more likely to participate in the 12-month survey (response rate: 72%) as compared to the 3-month survey (response rate: 51%). The effect of computer experience was entered as a covariate in the models predicting EHR acceptance; at both 3 and 12 months, years of computer experience were positively related to EHR acceptance. As shown in table 5, computer experience was a stronger predictor of EHR acceptance at 12 months than at 3 months. The impact of technology experience on acceptance is well documented.^{7 47} It is possible that our results are affected by the different survey response rates (51% at 3 months vs 72% at 12 months). However, the only demographic difference that we could identify between nurses who participated in the 3-month survey and nurses who participated in the 12-month survey was for computer experience, and we controlled for this difference in the models.

As the response rates were 51% for the 3-month survey and 72% for the 12-month survey, a selection bias is possible as participating nurses may have more positive perceptions of the EHR technology and its implementation. On the other hand, nurses who had more negative perceptions of the EHR technology and its implementation may have been more likely to participate in the survey in order to express their opinion. However, our results are consistent with other studies of nurses' acceptance of technology, in which increasing use and familiarity with the technology produces an improvement in perceptions of acceptance.¹²

More research using a longitudinal design is needed to further understand how EHR-related predictors of technology acceptance may change over time.⁷ This would help implementers of EHR technology identify key issues that need to be addressed in the short- versus long term. In addition, the EHR implementation process can be viewed as continuous,⁴⁴ involving activities such as optimization of the EHR or software upgrades. These changes over time may influence the usability and usefulness of the EHR and its functionalities, and therefore the level of EHR acceptance by nurses and other users.

Conclusion

As EHRs and other forms of health IT are increasingly being implemented in complex environments such as ICUs, healthcare organizations and designers of the technology need to consider end-user acceptance and the technology-related factors that influence acceptance. Our study shows that ICU nurses' acceptance and perceptions of EHR usability and usefulness improved over a period of 1 year after EHR implementation. EHR usability and the usefulness of CPOE were consistent predictors of EHR acceptance in the short-term (3 months post-EHR implementation) and the long-term (12 months post). The usefulness of the eMAR influenced EHR acceptance at 3 months, whereas the usefulness of nursing flowsheets was a predictor of EHR acceptance at 12 months. Even after 1 year of EHR use, the characteristics of the EHR technology (usability and usefulness) have a significant impact on acceptance and use of the technology by ICU nurses. Therefore, it is important for healthcare organizations to continue their efforts to optimize the design and use of EHR after the technology is implemented.