Impact of Electronic versus Paper-Based Recording before EHR Implementation on Health Care Professionals' Perceptions of EHR Use, Data Quality, and Data Reuse

Abstract

Background  The implementation of an electronic health record (EHR) with structured and standardized recording of patient data can improve data quality and reusability. Whether and how users perceive these advantages may depend on the preimplementation situation.

Objective  To determine whether the influence of implementing a structured and standardized EHR on perceived EHR use, data quality, and data reuse differed for users working with paper-based records versus a legacy EHR before implementation.

Methods  We used an electronic questionnaire to measure users' perception before implementation (2014), expected change, and perceived change after implementation (2016) on three themes. We included all health care professionals in two university hospitals in the Netherlands. Before jointly implementing the same structured and standardized EHR, one hospital used paper-based records and the other a legacy EHR. We compared perceptions before and after implementation for both centers. Additionally, we compared expected benefit with perceived benefit.

Results  We received 7,611 responses (4,537 before and 3,074 after implementation) of which 5,707 (75%) were from professionals reading and recording patient data. A total of 975 (13%) professionals responded to both before and after implementation questionnaires. In the formerly paper-based center staff perceived improvement in all themes after implementation. The legacy EHR center experienced deterioration of perceived EHR use and data reuse, and only one improvement in EHR use. In both centers, for half of the aspects at least 45% of responders experienced results worse than expected preimplementation.

Conclusion  Our results indicate that the preimplementation recording practice impacts the perceived effect of the implementation of a structured and standardized EHR. For almost half of the respondents the new EHR did not meet their expectations. Especially legacy EHR centers need to investigate the expectations as these might be different and less clear cut than those in paper-based centers. These expectations need to be addressed appropriately to achieve a successful implementation.

Background and Significance

Data recorded in electronic health records (EHRs) are primarily used to provide care to patients. However, more and more other uses of these data emerge such as quality audit, finance, and scientific research. To enable reuse of data, the meaning of data should be clear. A common approach to make the meaning of data explicit is structured and standardized data recording through EHRs. Studies have shown that structured and standardized EHRs may increase data quality and reuse possibilities but also may increase recording time, which negatively influences adoption and ease of data recording. 1

Previous research showed that health care professionals had high expectations regarding the capabilities of a new EHR but many expectations were not met. 2 Studies have shown that after the implementation of an EHR, physicians were less positive regarding their ability to create high-quality documentation, 3 their overall satisfaction, 4 and their satisfaction with ease of use. 4 Additionally, the interaction between physicians and patients might change when paper records are replaced with an EHR. 5 An inherent yet complicating factor is that the implementation of an EHR is a sociotechnical process. 6 Not only does the implemented system influence the health care professionals in their way of working, 7 the changing workflows and human–computer interaction also influence the experienced usefulness of the system and thereby the success of the implementation and adoption of the system. 8

In a previous study our research group validated a conceptual model, based on the technology acceptance model, which showed that attitude of EHR users toward structured and standardized recording influences their intention to adopt structured and standardized recording. 9 This means that to improve the adoption of structured and standardized recording practices, it is important to measure the attitudes and underlying perceptions of the EHR users regarding this recording practice. Three important aspects influencing this attitude are EHR use, data quality, and data reuse. 9 Changes in perceptions before and after an EHR implementation can indicate whether staff is satisfied with the new system and workflow, or whether adjustment of the EHR or workflow is required.

The extent of these effects might depend on the preimplementation situation of a hospital. Whether the organization comes from paper-based recording or a previous, less advanced, EHR might influence the implementation of an EHR and its effects on the staff. For example, research shows that practice managers migrating from paper-based records have different priorities in EHR implementations than those who previously used legacy EHRs. 10

Previous studies have investigated transitions from paper-based records to locally developed applications 11 and transitioning from a legacy EHR to a more advanced system. 12 However, little is known about the effect of the preimplementation situation on the postimplementation perceptions. In this study we have the unique opportunity to investigate this in a situation where two university hospitals, one using paper-based records and one using a legacy EHR, jointly implement a new structured and standardized EHR.

Objective

The aim of this study is to determine whether EHR use, data quality, and data reuse, as perceived by health care professionals, are affected differently by the implementation of a structured and standardized EHR between hospitals coming from paper-based records or from a legacy EHR. Additionally, we analyze to what extent postimplementation perceptions match preimplementation expectations.

Methods

This study was performed in two university hospitals located in Amsterdam, the Netherlands. Center A has 1,003 beds and center B has 733 beds. Each hospital has approximately 7,000 staff members. As university hospitals, both centers provide up to the highest level of complex specialized care, educate medical students, and perform research. A single team implemented the EHR in both centers, as these centers are in the process of merging to become one organization. Because the centers are in the same city the organizational and environmental conditions are comparable. The main difference between these two centers relevant for this study is the documentation practice before the implementation. Center A already used a legacy EHR that some departments combined it with a paper-based record, whereas center B predominantly used paper-based records supplemented with electronic information on laboratory and radiology results.

In the (predominantly) paper-based center, a digital system existed for the storage and consultation of radiology and laboratory results. Most patient data were recorded in paper-based records, unstructured, and in free text. This resulted in much freedom of expression for the health care professionals, but limited opportunities for reuse of data. When data of a patient had to be shared, the paper file had to be shared physically. This means that it took time to access patient data from colleagues, and that the files were not always available immediately. Moreover, reuse of data often led to manual copying of information. The legacy EHR center, on the other hand, used a best-of-breed solution in which the hospital information system consisted of a central patient record (for physicians, not nurses) with numerous linked ancillary systems, e.g., for radiological images and laboratory results. The data in these systems were partly structured and contained large amounts of free text. To a limited degree, this “forced” the personnel to record data in a structured manner, e.g., recording allergies in a structured list. This decreased the freedom of expression of the personnel, but increased possibilities for data reuse. Another advantage was that data were available more quickly and across the entire center.

The new integrated EHR was implemented to facilitate processes of collaboration and data exchange within and between the two centers. It was a best-of-suite solution providing a fully integrated EHR to both centers. In this new EHR, data were structured and standardized using problem-oriented recording with SOAP (subjective, objective, assessment, plan) documentation structure and national and international terminology systems. Although one center already used an EHR, neither had an integrated EHR yet. A major goal of implementing the new EHR in both centers was to record data once, in an integrated patient record, and enable their reuse for research, education, and improvement of quality of care. One of the focal points of the implementation team was therefore data quality. 13 This led to the configuration choice to record data as structured and standardized as possible, i.e., with minimal use of free text.

The implementation was performed and supported by a team of several types of specialists from different organizations. The team included personnel from the (global) vendor, experts on the technical aspects of EHR implementations, health informaticians, and experts on health care (including health care providers). Additionally, this team included health care providers that had practical experience with the system from other organizations where the same commercial EHR was already implemented. The vendor was not involved in our study.

To answer our research question, we measured user baseline perception before implementation, expected change and perceived change on the three themes of EHR use, data quality, and data reuse. We studied whether perceived change was affected by the baseline situation, i.e., coming from a paper-based center or a legacy EHR center. Comparing the expected change with the perceived change in our three themes shed light on the extent to which users' expectations were met.

Based on the literature 14 15 16 and discussions with experts in the fields of health care and questionnaire development, we developed two organization-wide questionnaires that contained, among other questions, 13 statements that focused on EHR use, data quality, and data reuse. For each statement the perception of the respondents was measured by the level of agreement on a 3-point rating scale (do not agree, neutral, and agree). In the questionnaire sent out before the EHR implementation, respondents also scored their level of expectation for the new system on the same statements (per statement the respondent could indicate whether it would become worse, stay the same, and become better). After the implementation the respondents indicated their current perceptions on the same 13 statements.

In April and May of 2014, the invitation for the first questionnaire was sent via email to all staff of the two centers, i.e., those using the EHR as well as those not using the EHR such as supporting staff. This was 1.5 years before the EHR implementation in center A and 2 years before the implementation in center B. A reminder was sent 20 days after the initial invitation. Additionally, the questionnaire was announced on the organizations' intranet. In total, the questionnaires were open for approximately 4 weeks each.

The second questionnaire was sent to the two centers separately to account for the different implementation dates of both centers. In center A the invitation was sent in May 2016 and in center B in November 2016, both approximately 7 months after the implementation. In both centers a response was possible for approximately 3 weeks with a reminder half-way through this period. Again, all staff members were invited to participate.

Upon completion of the questionnaire, linked email addresses were replaced with a hashed variable (using SHA-256 with a salt) rendering the answers of the respondents anonymous. The email addresses and the key for the hashing were removed before data analysis. The study design for both participating centers was submitted to the ethics committee of one of the centers and was exempt from review.

Statistics

Only responses from staff working with patient data that filled in the questionnaire before and after the implementation (double responders) were used in the further analysis. For these health care professionals we had three data points for each statement: perception and expectation before the implementation of the EHR, and the perception after the implementation. We distinguished three professions: physicians, nurses (both groups read and write in the patient record), and administrative staff (who predominantly only read from the patient record). All other personnel that use patient data (e.g., researchers, managers, and support staff) are combined in the “other” group.

To test for response bias, we compared the characteristics and answers of the respondents that responded to both lists (double responders) with those that only responded to either one of the questionnaires (single responders) using chi-square tests. We compared distributions of age (categories), profession, gender, and organizations. For the answers to the 13 statements we compared the perceptions before and after implementation, and the expectations.

To investigate whether the effect of the implementation on our 13 statements is different between our two centers, we created 13 ordinal logistic regression models. For each statement a model was defined where the outcome variable was the perception of the respondent on that specific aspect. Fixed effects included in the model were the before/after variable (a value of 0 indicating before implementation, and a value of 1 thereafter), the center (formerly paper-based or formerly legacy EHR), the interaction term of before/after and center, and profession. The latter was defined as three binary variables (nurses, administrative staff, and other) where the physicians were chosen as the reference group. Respondents could indicate more than one profession which is especially relevant for physicians with a large research (other) task. We developed two separate sets of models with each set having one of the centers as the reference center. This enabled us to investigate the significance of the implementation for each of the two centers separately since the significance of this variable is only valid for the reference center in the model. Inclusion of the interaction term in the model was tested with analysis of variance (ANOVA). In each model, the anonymous identifier (linking the two observations of each respondent) was added as a random effect. For each variable we report the coefficient and p -value.

To analyze whether the experienced effect was in line with the expectations of the staff, we first calculated the experienced effect by comparing the perceptions before and after the implementation (perception became worse, better, or remained the same). Next, we compared this with the expectations and calculated the percentage of respondents where the effect of the EHR was worse than expected, as expected, or better than expected. A respondent could be in the “as expected” group when the expectation was that the aspect would improve, and it did improve, but also when an aspect was expected to remain the same or to deteriorate and it did remain the same or deteriorate respectively. We calculated these percentages for six groups: the total respondents' group, the respondents from center A, those from center B, the physicians, the nurses, and the administrative staff.

All analyses were performed using the statistical environment R (version 3.4.1). 17 p -Values of <0.05 were regarded as significant (corrected for the number of tests with Bonferroni correction).

Results

In total, we received 7,611 responses to our questionnaire, 4,537 (response rate: 32%) before and 3,074 (response rate: 22%) after the implementation. Of the respondents, 5,707 (75%) indicated using patient data and 1,904 (25%) stated never using patient data, hence these latter respondents were excluded for further analyses. After matching respondents on the hashed anonymous identifier, we had a group of 975 care professionals that responded to both the pre- and post-EHR-implementation questionnaires.

The respondents' demographics are summarized in Table 1 . The double responders were significantly older and nurses were overrepresented in comparison to single responders. The perceptions and expectations before EHR implementation and perceptions after the implementation were not significantly different between the single and double responders.

Table 1. Demographics of respondents and result of chi-square test to check for differences between single and double responders.

	Double responders		Single responders		Result of chi-square
	n	(%)	n	(%)
Total	975	(100)	3,757	(100)	0.018
Center A (legacy EHR)	523	(54)	1,951	(52)
Center B (paper-based)	452	(46)	1,796	(48)
Function					0.000 a
Physicians	183	(19)	771	(21)
Nurses	349	(36)	1,016	(27)
Administrating staff	166	(17)	487	(13)
Other	277	(28)	1,488	(40)
Age					0.000 a
< 30	139	(14)	906	(25)
30–39	236	(24)	997	(28)
40–49	232	(24)	669	(19)
50–59	318	(33)	775	(21)
≥ 60	47	(5)	268	(7)
Gender					0.021
Male	219	(23)	898	(25)
Female	753	(77)	2,738	(75)

Abbreviation: EHR, electronic health record.

Note: p -Value after Bonferroni correction: 0.0125.

^aSignificant values.

Tables A–E ( Supplementary Material , available in the online version) detail the perception (A) and expectation (B) scores of the health care professionals on the 13 statements before the implementation of the EHR, the perceptions after the implementation (C), the change in perception on the 13 statements after the EHR implementation (D), and the change compared with the expectations of the professionals (E). Interpretation of these tables is included in the Supplementary Material (available in the online version).

The main results of our analysis are stated in Tables 2 and 3 , and Fig. 1 . Tables 2 and 3 show whether our respondents had a significantly higher (positive coefficient) or lower (negative coefficient) perception on our 13 statements after the implementation of the EHR (“implementation” column). It shows whether respondents in our two centers had a different baseline perception (“center” column) and whether the effect of the implementation was different between the two centers (“interaction term” column). They also indicate whether nurses, administrative personnel, or other staff had significantly different perceptions compared with physicians. Table 2 has center A (legacy EHR) as the reference center to analyze the significance of the implementation on the perceptions of the staff in that center. Table 3 shows the same with center B (paper-based) as the reference center. The addition of the interaction term significantly improved two models (statements 4 and 11). Therefore, to create comparable models, we included the interaction term to all models.

Table 2. Coefficients of the regression models on the associations between EHR implementation and perception on our 13 statements on EHR use, data quality, and data reuse (including p -value). Center A (legacy EHR) used as reference cente r ; interaction term is implementation combined with center B (paper-based) .

	Implementation	p -Value	Nurse	p -Value	Administrating staff	p -Value	Other staff	p -Value	Center B	p -Value	Interaction term	p -Value
EHR use
1. I have a clear overview of the patients' most important data (e.g., allergies)	−0.04	0.791	0.28	0.035	0.50	0.016	0.48	0.001	−0.11	0.446	0.12	0.545
2. The documentation method directly supports me in providing good care	− 0.49	0.000	0.55	0.000	1.11	0.000	0.97	0.000	0.27	0.079	0.19	0.360
3. The documentation method is time efficient	− 0.63	0.000	0.41	0.013	0.84	0.000	0.82	0.000	0.14	0.369	0.18	0.446
4. The patient data are stored and exchanged securely	−0.14	0.358	0.17	0.289	−0.03	0.884	0.02	0.912	− 1.08	0.000	1.34	0.000
5. The EHR provides all types of health care professionals a place to document patient data, in one integrated patient file	1.27	0.000	0.44	0.009	−0.15	0.573	0.28	0.136	0.11	0.526	−0.15	0.543
6. I document data in a way that colleagues can properly use those data in providing care	0.22	0.265	0.80	0.000	0.26	0.364	0.80	0.000	−0.56	0.004	0.68	0.016
Data quality
7. The patient data in the file are correct	−0.46	0.002	−0.04	0.811	−0.46	0.055	0.37	0.069	−0.25	0.166	0.41	0.062
8. The patient data in the file are up-to-date	0.17	0.225	−0.20	0.186	−0.37	0.071	0.14	0.398	−0.26	0.095	0.39	0.059
9. The patient data in the file are complete	−0.07	0.598	0.08	0.599	0.40	0.049	0.42	0.011	−0.14	0.381	0.38	0.063
Data reuse
10. I can easily create an overview of the patients' data for the patient or external referrers	−0.46	0.003	0.54	0.001	0.80	0.001	0.17	0.347	−0.33	0.052	0.63	0.006
11. The documentation method provides me with a way of properly communicating with colleagues and external referrers	− 0.61	0.000	0.25	0.093	0.55	0.028	0.29	0.096	− 0.68	0.000	1.05	0.000
12. The data I document on my patients provide me with an insight in the quality of the care I provide	− 0.49	0.001	0.95	0.000	1.44	0.000	1.43	0.000	0.24	0.151	0.05	0.817
13. The documentation method supports me or my colleagues in doing research	0.30	0.073	1.49	0.000	1.67	0.000	1.19	0.000	−0.05	0.775	0.27	0.263

Abbreviation: EHR, electronic health record.

Note: p -Value after Bonferroni correction: 0.0019; significant values are given in bold.

Table 3. Coefficients of the regression models on the associations between EHR implementation and perception on our 13 statements on EHR use, data quality, and data reuse (including p -value). Center B (paper-based) used as reference cente r ; interaction term is implementation combined with center A (legacy EHR) .

	Implementation	p -Value	Nurse	p -Value	Administrating staff	p -Value	Other staff	p -Value	Center A	p -Value	Interaction term	p -Value
EHR use
1. I have a clear overview of the patients' most important data (e.g., allergies)	0.09	0.564	0.28	0.035	0.50	0.016	0.48	0.001	0.11	0.446	−0.12	0.545
2. The documentation method directly supports me in providing good care	−0.30	0.053	0.55	0.000	1.11	0.000	0.97	0.000	−0.27	0.079	−0.19	0.360
3. The documentation method is time efficient	−0.45	0.008	0.41	0.013	0.84	0.000	0.82	0.000	−0.14	0.369	−0.18	0.446
4. The patient data are stored and exchanged securely	1.20	0.000	0.17	0.289	−0.03	0.884	0.02	0.912	1.08	0.000	− 1.34	0.000
5. The EHR provides all types of health care professionals a place to document patient data, in one integrated patient file	1.13	0.000	0.44	0.009	−0.15	0.573	0.28	0.136	−0.11	0.526	0.15	0.543
6. I document data in a way that colleagues can properly use those data in providing care	0.89	0.000	0.80	0.000	0.26	0.364	0.80	0.000	0.56	0.004	−0.68	0.016
Data quality
7. The patient data in the file are correct	−0.05	0.741	−0.04	0.811	−0.46	0.055	0.37	0.069	0.25	0.166	−0.41	0.062
8. The patient data in the file are up-to-date	0.56	0.000	−0.20	0.186	−0.37	0.071	0.14	0.398	0.26	0.095	−0.39	0.059
9. The patient data in the file are complete	0.31	0.041	0.08	0.599	0.40	0.049	0.42	0.011	0.14	0.381	−0.38	0.063
Data reuse
10. I can easily create an overview of the patients' data for the patient or external referrers	0.16	0.331	0.54	0.001	0.80	0.001	0.17	0.347	0.33	0.052	−0.63	0.006
11. The documentation method provides me with a way of properly communicating with colleagues and external referrers	0.45	0.007	0.25	0.093	0.55	0.028	0.29	0.096	0.68	0.000	− 1.05	0.000
12. The data I document on my patients provide me with an insight in the quality of the care I provide	−0.44	0.007	0.95	0.000	1.44	0.000	1.43	0.000	−0.24	0.151	−0.05	0.817
13. The documentation method supports me or my colleagues in doing research	0.58	0.001	1.49	0.000	1.67	0.000	1.19	0.000	0.05	0.775	−0.27	0.263

Abbreviation: EHR, electronic health record.

Note: p -Value after Bonferroni correction: 0.0019; significant values are given in bold.

Fig. 1.

Per statement whether the effect of the implementation was worse than expected ( red , lower part of bar), as expected ( gray , middle part of bar), or better than expected ( green , top part of bar). For each statement six groups are depicted: total group double responders (T), center A (CA), center B (CB), physicians (P), nurses (N), and administrative staff (A).

Taking statement 11 (peer communication) as an example, Table 2 shows a statistically significant negative coefficient in the implementation column. This means that after the implementation of the EHR, the perception on this statement of the staff of the legacy EHR center (the reference center of Table 2 ) was worse than at baseline. No significant coefficients in Table 2 on the different functions (columns nurse, administrating staff, and other) means that these groups did not have different perceptions as compared with the physicians. The significantly negative center B coefficient shows that, at baseline, the perception in the paper-based center was lower than in the legacy EHR center. Finally, the significantly positive interaction term indicates that the effect of the implementation was different between both centers, i.e., there is a larger effect in the paper-based center.

Perceived Change

For only two statements (statements 4 security and 11 peer communication) the effect of the implementation was significantly different between the two centers, i.e., the effect in center B was more positive. These were the only statements with significantly different baseline perceptions between the two centers, where respondents in center B (paper-based) were more negative.

On the theme “EHR use,” center A (legacy EHR) became significantly more negative on statements 2 (method supports care provision) and 3 (efficiency). Center B (paper-based) became significantly more positive on statements 4 (security) and 6 (documentation method). Both centers experienced an improvement on statement 5 (integration). This is the only statement for which respondents from both centers experienced a positive effect of the new EHR. We did not observe an effect of the EHR implementation on statement 1 (overview of data).

The only statement within the theme “data quality” with a significant change was statement 8 (data are up-to-date), i.e., the staff of center B (paper-based) was significantly more positive on this statement after the implementation than before. On the other statements, staff of neither center had different perceptions after the implementation of the EHR.

When considering data reuse, center A (legacy EHR) experienced a significantly more negative perception than before the implementation of the EHR on statements 11 (peer communication) and 12 (insight in care quality). In center B (paper-based) the staff was significantly more positive than before the implementation about statement 13 (research support). There were no differences in perception between before and after the implementation on statement 10 (creating an overview).

Overall nurses and administrative staff were significantly more positive on statement 5, and other staff on statement 6, when compared with physicians. Statements 5, 10, and 11 are more positively perceived in all of these groups as compared with physicians.

Expected Change versus Perceived Change

Fig. 1 shows for each statement whether the effect of the implementation was worse than expected, as expected, or better than expected for six subgroups. Both centers show similar patterns with the exception of statements 4 (security), 10 (creating an overview), and 11 (peer communication), whereas in center A (legacy EHR) 8 to 10% more respondents indicate results worse than expected. The figure shows that, in general, a large percentage of respondents (at least 45%) experienced results from the implementation that were less than expected for the statements 2 (method supports care provision), 3 (efficiency), 9 (data are complete), 10 (creating an overview), 11 (peer communication), and 12 (insight in care quality). Statement 7, on the correctness of the patient record, has the most responders perceiving a result that was better than expected. However, these percentages are still relatively low (12%).

Discussion

After the implementation of a structured and standardized EHR, the overall perception of users in the formerly paper-based center improved or remained stable for our three themes: EHR use, data quality, and data reuse. In the legacy EHR center, perceptions on EHR use and data reuse decreased or remained stable after the implementation, and perceptions on data quality remained stable. For only one aspect of EHR use did users in both centers experience an improvement: the EHR enabled all types of health care professionals to record data. Regardless of the prior recording practice, results show that nurses and administrative staff had a more positive perception than physicians on all three themes.

At baseline there were only two aspects (security and peer communication) where the respondents of the paper-based center were more negative than those of the legacy EHR center. These were the only aspects where the effect of the implementation was stronger (a larger positive change) in the paper-based center than in the legacy EHR center. This indicates that there was more room for improvement in the paper-based center on these two aspects. As mentioned before, in the previously paper-based center, working with paper-based files made it more difficult to share data with colleagues as the paper file had to be physically shared between physicians. Furthermore, the physical form of the patient file meant that the data were vulnerable to loss or unauthorized access. Implementing a new EHR provided solutions to these problems in the paper-based center more than in the legacy EHR center where the legacy EHR already provided partial solutions to these problems.

Our results show that, in general, respondents expected that the new EHR would perform at least as well as the old system. There was one exception: the statement “the documentation method is time efficient” for which 16 to 32% expected the new EHR to perform worse. Almost half (45% or more) of our respondents experienced results worse than expected from the implementation in all three of our themes. Especially the effects of the EHR implementation on data reuse were below expectation. Only a very small percentage (12% or less) of respondents indicated a better than expected perception on any of the themes after implementation. The degree to which the new system's functioning adheres to the expectations of the health care professionals can influence their attitude toward the system. A negative attitude might lead to a negative intention to act. 18 The latter, in turn, might pose a threat to the success of the implementation when users decide to not use or not correctly use the new EHR.

The statistically significant effects of the implementation in the paper-based center were all improvements. In contrast, the changes in the legacy EHR center were, apart from one, all deteriorations. Our results indicate that when implementing an EHR, the center's pre-implementation recording practice influences the effects of the implementation on the staff's perceptions. Apparently, users experience the largest improvement in the move from paper to electronic, and less in the move from a legacy EHR to an advanced structured and standardized EHR.

To improve health care providers' perceptions of the EHR, it is therefore important that a hospital implementing an EHR considers its preimplementation recording practice. When this is a legacy system, the implementation team needs to investigate what the expectations of the staff concerning the new EHR are. These might be different and less clear cut than those during an implementation coming from paper-based records. This is especially the case since these EHR users will compare the new system to their old system and might find many things that were more easily done in the old system. This is because they were used to their working processes or because the old system provided more freedom to work according to personal preferences. Additionally, paper-based users might be willing to expend more effort after the implementation as they experience larger improvements than legacy EHR users. Providing a test environment where users can work with the new system before implementation and experience the planned changes hands-on can help to collect the users' expectations.

When investigating expectations, it is essential to see whether these expectations are realistic, i.e., whether they concern functionalities or changes in work processes that are actually planned to be implemented. Realistic expectations need to be addressed by the EHR implementation to make sure that the staff stays motivated to use the EHR according to the new work processes. Unrealistic expectations need to be tempered. The high expectations we found indicate that staff expected the new EHR to outperform the old documentation system. However, for half of our statements at least 45% of our respondents indicated that the results were less than expected. For these aspects, it should be investigated whether this is based on real decline in functionality, which requires changes to the system, or a lack of training or understanding, which requires additional training of the staff. Additionally, it is important that staff understands what is actually influenced by the implementation of an EHR. For example, aspects on data exchange cannot improve just by implementing an EHR in one center. To improve on data exchange, the implementing centers will need to work together with other organizations with whom the data need to be exchanged.

Risko et al found that physicians can experience an initial decrease in efficiency. 19 The current study shows that the physicians of the legacy EHR center experienced a decrease in documentation efficiency after the implementation. Research shows that after an initial decline at short-term follow-up (6–12 months) an upward trend can be observed at long-term follow-up (12–24 months). 4 An additional questionnaire 2 years after the implementation could be used to check whether we can detect a similar effect extending our findings to a longer follow-up period. Furthermore, additional data on actual changes in completeness and correctness of the patient record could triangulate our current results.

Our results are in line with our previous time-and-motion study 20 that showed that physicians needed more time for data recording and had less time for direct patient contact. The present study indicates that the measured differences are also experienced by the physicians, and is therefore complementary to the time-and-motion study.

The main strength of this study is that, to our knowledge, this is the first study investigating the perceptions of user experiences before and after the introduction of a joint EHR combining two academic hospitals coming from two different baseline situations. Including two centers that implement the same system enables us to compare these two baseline situations (paper-based vs. legacy EHR). Our study design allows us to not only get the users' opinions on the new EHR, but also to see the perceived difference compared with the old situation and the expected change. Our results indicate that the preimplementation recording practice is an important factor to consider during the planning and execution of an EHR implementation.

Another strength of this study is the inclusion of all health care professionals working in two university hospitals. We included physicians, nurses, and other professionals from all available specialties and all organizational levels of the organizations. Research from Ratwani et al 21 identified gaps between literature and key stakeholders' perceptions on themes encountered during the implementation of EHRs. We attempted to narrow this gap by including the end-users of the EHR as the main judges of our aspects. Additionally, through our questionnaire we were able to match the answers of respondents who responded to both our questionnaires. Comparing the double and single responders we found no indications for selection bias. Therefore, and to ease interpretability of the results we performed our main analysis on the answers of double responders. Another strength is that we sent out the second questionnaire 7 months after the implementation of the EHR. This means that the respondents had time to familiarize themselves with the new system and allowed for resolution of initial configuration issues. By sending out our questionnaire relatively early (compared with follow-up periods of 2 years or more) the results of this study could be used to steer the first set of changes in the system to accommodate for the wishes and expectations of the end-users.

The main limitation of our study is that we only have data on perceptions and expectations and cannot match these to the actual changes in EHR functionality and data quality that were caused by the implementation of the new EHR. Another limitation is the inclusion of only one type of EHR. This means that our results might be confounded by the implemented EHR itself, something we cannot correct for in the current study. This also limits generalizability of our study, although we believe that our conclusions are generalizable to most modern EHRs with a focus on standardized and structured data recording. Finally, the response rate of the questionnaire may seem low. However, the questionnaire was sent to all employees, many of whom may not have responded as they have no involvement with the EHR. As there are no baseline figures on the number of EHR users, we cannot determine the response rate among the targeted employees. Based on estimates of the number of clinicians and nurses, we estimate the response among our target population is around 60%, which we consider good.

A possible confounding factor is the level of autonomy in Dutch health care. The health care providers in Dutch hospitals (especially in academic centers) enjoy relatively high levels of autonomy. This means that a successful EHR implementation requires the support of the employees. Simply forcing changes in a top-down manner might result in opposition and suboptimal use of the new system. In situations with less autonomy, enforcing policies might have more impact and personnel might have less personal expectations.

Conclusion

Our study shows the impact of prior recording practice on the perceived benefits of the implementation of a standardized and structured EHR on EHR use, data quality, and data reuse. The implementation had positive effects on these themes in a formerly paper-based center, and mostly negative or neutral effects on a center coming from a legacy EHR. For almost half of the respondents the new EHR did not meet their expectations. These results indicate that the preimplementation recording practice is an important factor to consider during the planning and execution of an EHR implementation. Especially legacy EHR centers need to investigate the expectations of the staff as these might be different and less clear cut than those in paper-based centers. Problems with expectations that were too high could be mitigated or possibly prevented by ensuring that expectations before the implementation are realistic and by providing more training for staff after the implementation.

Clinical Relevance Statement

Paper-based or legacy EHR centers experience different effects of the implementation of a structured and standardized EHR regarding EHR use, data quality, and data reuse. The effects were more positive and clearer when transitioning from paper-based files to an EHR. This indicates that the preimplementation recording practice is an important factor to consider during the planning and execution of an EHR implementation.

Multiple Choice Questions

1. What is the impact of the introduction of a new structured and standardized EHR on the perceptions of health care providers concerning EHR use, data quality, and data reuse?

   1. Perceived benefits of the new EHR do not differ between hospitals that transition from paper to an advanced EHR and hospitals that transition from a legacy EHR to an advanced EHR.

   2. Perceived benefits of the new EHR are smaller when hospitals transition from paper to an advanced EHR compared hospitals that transition from a legacy EHR to an advanced EHR.

   3. Perceived benefits of the new EHR are greater when hospitals transition from paper to an advanced EHR compared with hospitals that transition from a legacy EHR to an advanced EHR.

   4. Perceived benefits of the new EHR are mixed when hospitals transition from paper to an advanced EHR compared with hospitals that transition from a legacy EHR to an advanced EHR.

   Correct Answer: The correct answer is option c. Health care professionals experienced different effects on EHR use, data quality, and data reuse based on their preimplementation recording practice. The implementation had positive effects on these themes in a formerly paper-based center, and mostly negative or neutral effects on a center coming from a legacy EHR. This indicates that the preimplementation recording practice is an important factor to consider during the planning and execution of an EHR implementation.

2. What do health care providers expect from the introduction of a new structured and standardized EHR concerning EHR use, data quality, and data reuse and are those expectations met after the introduction?

   1. The new system is expected to be worse than the old one, and expectations were met

   2. The new system is expected to be worse than the old one, but expectations were not met

   3. The new system is expected to be at least as good as the old one, and expectations were met

   4. The new system is expected to be at least as good as the old one, but expectations were not met

   Correct Answer: The correct answer is option d. Health care providers expected that the new system would perform equally well as the old system on almost all aspects. Time efficiency was the only aspect where the end-users had doubts about the new situation. After the implementation it was shown that on at least half of the aspects the expectations were not met, and the new system was performing less than expected.