CLINICIAN PERCEPTIONS OF AN ELECTRONIC MEDICAL RECORD DURING THE FIRST YEAR OF IMPLEMENTATON IN EMERGENCY SERVICES

INTRODUCTION

Electronic medical records (EMRs) used in emergency departments (EDs) have great potential for improving quality of care by increasing access to complete patient records and by providing context specific decision support. ¹ EMRs may also prove to be important ED tools for public health purposes such as responding to bioterrorism threats and permitting early recognition of new epidemics ^{2, 3}. Notably, the IOMs Future of Medicine report listed adoption of robust information systems as a key recommendation for improvement and modernization of emergency health services⁴.

While the potential value of EMRs in emergency care is great, there are also barriers unique to emergency services settings including planned and unplanned system downtime in a 24 hour operation, challenges with entering information into a computer during a real-time emergency, and potential reductions in throughput in already overwhelmed EDs ⁵.

Installations have yielded mixed results. For example, an evaluation of the implementation of an emergency department expert charting system found that use of the system consistently improved care for occupational blood exposure patients while generating no consistent improvement for care of low back pain or pediatric fever cases. ⁶ Overall, physicians generally believed that the system improved clinical documentation but a number of concerns were expressed.

Understanding factors associated with clinician acceptance of and satisfaction with new clinical information technology (CIT), including EMRs, is critical for developing effective implementation plans that can support correct and consistent use, enhance user satisfaction, and maximize beneficial outcomes. Research has found that initial acceptance of CIT is associated with influential leadership, readily available technical assistance, integration with workflow, and user involvement in design and implementation. ^7-9

Most research on physician satisfaction with EMRs has studied initial satisfaction with the systems. This study expands upon prior user satisfaction research¹⁰ by measuring satisfaction longitudinally, over the first year of implementation, and by using conceptual models of technology acceptance to help explain satisfaction. Results will serve as pilot data for the development and testing of EMR implementation strategies across institutions.

METHODS

Study Setting

This study was conducted at Nationwide Children's Hospital (NCH), a free standing, 328-bed, tertiary care hospital affiliated with The Ohio State University. All elements of this study were approved by the Nationwide Children's Hospital Institutional Review Board. In 2006, the hospital began an institution wide roll-out of a commercial ambulatory EMR, starting with the ED. The ED roll-out was followed by sequential roll-outs in three off-site urgent care centers. Prior to the EHR implementation, all documentation and orders were handwritten. No significant clinical decision support was available. One computerized tracking system (MedHost) was in use.

The implemented NCH EMR includes documentation for physicians and nurses, laboratory and radiology reports, as well as physician computer order entry. Beyond documentation, the system offers decision support for weight-based dosing, formulary-based medication selection, and medication interactions. Physicians can customize the system for their own use by developing key-stroke codes to enter blocks of commonly used text.

On the “go live” date at each site, use of the system was mandated and paper processes were discontinued with the exception of documentation for critical care cases and trauma alerts.

All clinicians received at least six hours of training on the system. When the system was launched, technical support staff was stationed on the units to provide immediate support. For the first week, the technical support staff-to-clinician ratio was nearly one-to-one. In addition, expanded attending physician, resident, and nursing coverage was provided during the first month of implementation to ensure that quality of care and efficiency were maintained during the process.

Population

Physicians and nurses working in the ED or in an urgent care center at the time of roll-out were recruited for our study using an email solicitation. This included attending physicians, fellows, nurse practitioners, and other nurses. Participants were asked to complete an online survey at three points in time. The first wave of surveys occurred between 30 and 89 days after implementation. This was done to give most clinicians the opportunity to experience using the system before they received a survey. Follow-up surveys were sent 90 to 179 days after implementation (wave 2), and 180 to 270 days after implementation (wave 3). All surveys were conducted between November 2006 and July 2007 using an online survey with a unique individual identifier in order to link data longitudinally across waves of the survey.

Survey Components

User characteristics studied included demographics, perceptions of the new EMR, and general attitudes toward technology. Demographic variables included profession (nurse or physician), age, and gender. Perceptions of the EMR were measured using items adapted from the Unified Theory of Acceptance and Use of Technology (UTAUT)¹¹ which includes four core constructs: effort expectancy, or perceived ease of use, the degree to which the user finds the system easy to use (4 items e.g., “Learning to operate the EMR is easy for me”); performance expectancy, or perceived usefulness, the degree to which the user believes the system will have the desired performance attributes (3 items e.g. “Using the EMR enables me to complete tasks more quickly”); social influence, the degree to which the user believes that peers and leaders support the use of the system (3 items e.g., “People who influence my behavior think that I should be a committed EMR user”); and facilitating conditions, the degree to which institutional supports are available to assist with use (4 items e.g., “A specific person, or group, is available for assistance with system difficulties”) ¹¹. Computer literacy (7 items e.g., “I find computers easy to work with”), affective feelings toward computers (5 items e.g., Computers and everything related to them fascinate me), and computer self-efficacy (4 items e.g., “I am confident that I can learn computer skills”) were measured using scales from the Beckers and Schmidt Computer Anxiety Scale.¹²

Our primary outcomes of interest were user satisfaction and committed use. User satisfaction was measured with 3 items, each scored on a 5-point Likert scale, ranging from “1-strongly disagree” to “5-strongly agree.” The items were: 1) I am completely satisfied with the functionality of the system; 2) I am completely satisfied with the reliability of the system; and, 3) I would recommend the system to others. An overall satisfaction score was calculated by averaging the responses from these three items. All perception measures were rescaled on a 0 to 10 scale where 10 represents highest agreement.

Analysis

Our analysis included two sections: 1) correlates of initial satisfaction; and, 2) changes over time in perceptions of and satisfaction with the system. To study correlates of initial satisfaction, we selected the first survey response for each of the 71 clinicians surveyed. First, bivariate relationships were tested between each categorical independent variable and satisfaction using a Wilcoxon rank sum because of the non-normal distribution of responses. Categorical independent variables included age (<40 or ≥40), gender, and role (physician or nurse). For these categories, we also report the percentage of 4 and 5 responses for each of component satisfaction scales (reliability, function, and willingness to recommend) and evaluate demographic differences using the chi-squared test. Next, bivariate relationships with the continuous independent variables including effort expectancy, performance expectancy, social influence, facilitating conditions, computer literacy, affective feeling toward computers, and computer self-efficacy were testing using Spearman's correlation coefficient. Finally, a stepwise multivariate linear regression was used to identify factors most predictive of satisfaction.

The analysis of changes over time was performed only for clinicians who completed the survey in wave 1 and in wave 2 and/or 3. Changes in mean satisfaction and attitudinal scores between wave 1 and wave 2, and between wave 1 and wave 3 were tested using the Wilcoxon signed rank test.

We note that while our sample included information from implementation in the primary ED and in off-site urgent care centers, it is not possible to conduct site stratified analysis because some physicians and nurses may practice in more than one location. The approach to implementation was consistent across sites so we believe that any site effect would be minimal.

RESULTS

Seventy-one individuals completed at least one wave of the survey, representing approximately 50% of the ED/UC staff. Nurses made up 62% of respondents. The response rate was 60% for physicians and 44% for nurses. Table 1 describes the characteristics and initial satisfaction of the clinician respondents.

Table 1.

Sample demographics and overall satisfaction

Characteristic	N‡	Mean (median) Overall Satisfaction	%4-5 Reliability	%4-5 Function	%4-5 Recommend
Total	71	3.35 (3.67)	60.6	47.9	60.6
Gender
Female	58	3.35 (3.67)p=0.80	58.6	49.1	55.2†
Male	12	3.31 (3.33)	66.7	41.7	83.3
Age Category
<40	32	3.49 (3.67) p=0.26	65.6	61.3†	59.3
40+	39	3.24 (3.00)	56.4	38.9	61.5
Role Category
Physician	25	3.35 (3.33) p=0.77	60.0	44.0	68.0
Nurse	44	3.35 (3.35)	61.4	51.2	54.5

^†Difference marginally significant, p<0.10

^‡One person did not report gender and two people did not report role category. Cases with missing data are excluded only from the analysis in which their information is missing

Thirty-five responses were received in wave 1, 46 responses were received in wave 2, and 53 responses were received in wave 3. Of the 35 respondents in wave 1, 20 (57.1%) completed a wave 2 survey and 25 (71.4%) completed a wave 3 survey. Waves 2 and 3 also included clinicians who did not complete the baseline survey because they had not yet started using the system because of the roll out schedule or because they had not chosen to complete the survey the first time they received it but chose to participate when presented with a later opportunity.

User Satisfaction

Initial satisfaction with the system was moderate. Over 60 percent of respondents agreed or strongly agreed that they were completely satisfied with the system's reliability. However, few than half agreed or strongly agreed that they were completely satisfied with the system's functionality. Sixty percent agreed or strongly agreed that they would recommend the system to a colleague. The average response across the three satisfaction items, each measured on a 5-point Likert scale, was 3.35 (Table 1).

Correlates of Satisfaction

Using the first response from each of the 71 survey respondents, we assessed correlates of satisfaction. Overall satisfaction with the system was not significantly associated with demographics including gender, primary role (nurse vs. physician), or age (Table 1). As shown in table 2, satisfaction was, however, strongly associated with effort expectancy (r=0.57, p<0.01), performance expectancy (r=0.61, p<0.01), facilitating conditions (r=0.52, p<0.01), social influence (r=0.38, p<0.01), computer literacy (r=0.37, p=0.01), and affective feelings toward computers (0.26, p=0.03). In a multivariate linear regression model, built using stepwise selection, only facilitating conditions and performance expectancy remained significantly associated with satisfaction. This model explained nearly half of the variance in satisfaction (r²=0.45).

Table 2.

Spearman Correlations between initial satisfaction and perceptions of the EMR and general technology

Characteristic	Correlation with overall Satisfaction	P
Effort Expectancy	0.57	<0.001*
Performance Expectancy	0.61	<0.001*
Social Influence	0.38	0.001*
Facilitating Conditions	0.52	<0.001*
Computer Literacy	0.37	0.002*
Computer self-efficacy	0.20	0.10
Affective feelings toward computers	0.26	0.03*

Changes over time

We analyzed changes over time for the disaggregated components of our satisfaction score: satisfaction with functionality, satisfaction with reliability, and willingness to recommend, each measured on a 5-point Likert scale (Table 3). For each variable, we assessed within-subject change from baseline to wave 2 (90 to 179 days post implementation), and from baseline to wave 3 (180 to 270 days post implementation). No significant changes in satisfaction were found at wave 2 or wave 3. Satisfaction with functionality trended higher over the study period while satisfaction with reliability trended lower.

Table 3.

Intrapersonal change in satisfaction and system perceptions

Characteristic	ΔWave 2†	p	ΔWave3†	p
Overall Satisfaction	-0.02	0.93	-0.07	0.67
Satisfaction with Functionality	+0.10	0.67	+0.25	0.23
Satisfaction with Reliability	-0.29	0.33	-0.44	0.07
Willingness to Recommend	+0.14	0.38	+0.04	0.80
Effort Expectancy	-0.12	0.73	+0.50	0.13
Performance Expectancy	+0.56	0.17	+1.20	0.01*
Social Support	-0.24	0.60	+0.53	0.12
Facilitating Conditions	-0.42	0.31	+.075	0.86

^*Statistically significant (p<0.05)

^†Average within individual change from wave 1

Of the system perception and technology attitudes measured, only performance expectancy showed significant improvement at wave 3, with an increase from 8.5 to 9.9 (p<0.01). Effort expectancy, social support, and facilitating conditions all trended toward improvement by wave 3, but none reached statistical significance. No changes were significant at wave 2. No changes were seen in computer literacy, affective feelings, self efficacy over time.

DISCUSSION

This study found generally high levels of satisfaction with a newly implemented electronic medical record and found that some elements of satisfaction increased over time. The overall initial satisfaction level was 3.35 on a 5-point scale. Willingness to recommend and satisfaction with reliability both had agreement rates over 60 percent. User satisfaction with system's functionality had the lowest initial agreement rate but showed the greatest increase during the study period. Improvement in satisfaction with functionality over time, while not statistically significantly, was likely influenced by the combination of iterative minor improvements in the system post-implementation and increased user comfort associated with experience using the system.

Perceptions of the system including effort expectancy, performance expectancy, social influences, and facilitating conditions were constant over time suggesting that the users’ initial impressions have a lasting impact. This finding lends strength to the argument that it is important to develop and implement a roll-out plan that maximizes perceptions of the system in the first weeks of implementation. The NCH roll-out plan combined clinical input in system design, role-tailored training, and extensive technical support during the first month of implementation to achieve this goal.

Perception of facilitating conditions was the strongest predictor of initial satisfaction. This construct included whether the clinicians felt they had the resources and knowledge to use the system, believed the system was compatible with existing processes and systems, and believed technical support was available. This finding reinforces qualitative findings that training and support are essential to the successful implementation of a new clinical information technology ^13-15. Given the strong relationship between facilitating conditions and satisfaction, a less intensive implementation strategy than used by the study institution would likely have yielded notably lower satisfaction.

The other significant predictor of satisfaction was performance expectancy, including perceptions of the system's ability to improve performance and productivity. This finding reinforces the importance of engaging clinicians in designing a system that they believe will ultimately improve quality of care and efficiency, even if this involvement may create short-term disruptions or delays.

Interestingly, satisfaction with the system was not associated with clinician demographic including gender, role (nurse vs. physician), or age. This finding is contrary to concerns that older clinicians are less accepting of new technology and may be a result of our study setting, an academic medical center. In academic settings such as ours, the continual implementation of practice innovations is likely to reduce anxiety typically associated with each new technology. Further, clinicians uncomfortable with the regular introduction of new technologies may be less likely to choose an academic practice setting where such innovations are more common.

We note that the findings from this single site study may not be generalizable to other facilities considering implementing an EMR. Our facility had an inpatient EMR in place prior to the implementation of the ambulatory EMR. While most ED clinicians did not use the inpatient systems, experiences with the inpatient EMR may have biased respondents’ reported initial satisfaction with the ambulatory system for those who did. Also, characteristics of the local culture may affect acceptance of a new technology in ways that differ from other emergency departments. Our results may also be influenced by selection bias. Those who opted not to participate or not to complete all waves of this voluntary survey may differ from the participants. Voluntary respondents to satisfaction surveys are frequently drawn from those with overly positive or negative perceptions regarding the topic. Additionally, nurses who do not sit at a workstation may have been less likely to respond to an online survey. The potential direction of bias associated with self-selection in uncertain. Given that there was no significant difference between satisfaction in physicians and nurses, proportionate under-representation of nurses should not bias overall estimates. Despite limitations, our approach allowed us to investigate implementation at this single site in considerable depth and over time adding strength to our findings.

CONCLUSIONS

Our findings suggest that satisfaction with an ambulatory EMR system implemented in a pediatric emergency department and associated urgent care clinics is strongly associated with perceptions of the system's usefulness and with level of technical support. Satisfaction can increase over time with increased experience and iterative system improvement; however, many perceptions of the system at launch persisted through the first months of use. As a result, system implementation plans must make efforts to ensure positive early impressions of training, support, and performance to maximize the likelihood of obtaining high user satisfaction. Future study is needed to formally test if implementation strategies designed to maximize facilitating conditions and perceived usefulness are more effective than usual implementation schemes.