Use of Electronic Health Record Simulation to Understand the Accuracy of Intern Progress Notes

Abstract

Background With the widespread adoption of electronic health records (EHRs), there is a growing awareness of problems in EHR training for new users and subsequent problems with the quality of information present in EHR-generated progress notes. By standardizing the case, simulation allows for the discovery of EHR patterns of use as well as a modality to aid in EHR training.

Objective To develop a high-fidelity EHR training exercise for internal medicine interns to understand patterns of EHR utilization in the generation of daily progress notes.

Methods Three months after beginning their internship, 32 interns participated in an EHR simulation designed to assess patterns in note writing and generation. Each intern was given a simulated chart and instructed to create a daily progress note. Notes were graded for use of copy-paste, macros, and accuracy of presented data.

Results A total of 31 out of 32 interns (97%) completed the exercise. There was wide variance in use of macros to populate data, with multiple macro types used for the same data category. Three-quarters of notes contained either copy-paste elements or the elimination of active medical problems from the prior days' notes. This was associated with a significant number of quality issues, including failure to recognize a lack of deep vein thrombosis prophylaxis, medications stopped on admission, and issues in prior discharge summary.

Conclusions Interns displayed wide variation in the process of creating progress notes. Additional studies are being conducted to determine the impact EHR-based simulation has on standardization of note content.

Editor's Note: The online version of this article contains a table of 28 core competencies for intern electronic health record use.

Introduction

The electronic health record (EHR) has become the major source of clinical information and documentation in health care. Consequently, graduate medical education trainees spend an increasing amount of time with the EHR.^1,2 With increased EHR use, a growing number of problems related to the integrity and quality of information has been communicated, including the use of word processing “copy-paste” functionality and predefined macros to populate notes; wide variations in individual use of these functionalities have also been reported.^3,4 This affects users' ability to cognitively process the large volume of information within the record in an effective manner, especially when related to the recognition of issues that can cause patient harm.⁵

Consequently, a number of core competencies for EHR-based education of new learners has been proposed.^6,7 However, the ideal method to optimize EHR training remains to be established. At Oregon Health & Science University (OHSU), we have employed high-fidelity simulation to create training exercises that mimic real world clinical cases to train and assess EHR competencies for medical students and residents; participation in these exercises significantly improves effective EHR use.^8,9

In this article, we describe the creation of an EHR simulation exercise based on progress notes as part of an intern learning week, as well as the lessons learned with respect to intern EHR use patterns.

Methods

Three months after beginning residency, all OHSU internal medicine interns participated in a weeklong “Intensive/Boot Camp” designed to prepare them to handle common problems and procedures. During this week, interns were split into groups of 5 or 6 members, and then rotated through 6 predefined stations throughout the week.

OHSU health care employs EpicCare (Epic Systems, Madison, WI) as its enterprise EHR. All interns received 1.5 days of Epic training delivered by the OHSU Epic Training Team at the beginning of residency. Training includes instruction on real world task completion relevant to interns' clinical practice. One month after this training, all interns participated in a dedicated exercise to test their ability to perform a set of 28 defined EHR use–related competencies (provided as online supplemental material) with our simulation version of the EHR. These competencies are based on those derived for medical student education.⁶

For simulations, we utilized workstations identical to the stations found in the hospital, but we ran a simulation version of Epic, as previously described.¹⁰ Briefly, this version was cloned from the production environment and populated with our high-fidelity simulation case while maintaining all unique log-in and user customizations utilized in the clinical environment.

After a brief orientation, each intern was given 60 minutes to review the simulated record of a patient with pneumococcal pneumonia, and then instructed to review the chart and create a progress note. The case contained 5 days of clinical information, including vitals, medications, laboratory values, progress notes, and a prior discharge summary. The case was designed with specific patient safety issues that interns needed to identify to assess whether they could effectively gather and process information to create a progress note (box). After the exercise, each note was graded by a member of the study team and was assessed for use of data importation tools (macros), use of copy-paste (defined as reproduction of an entire section of the plan without modification), and recognition of safety issues.

Recognizing Trends

 Worsening hypertension  New drop in hematocrit

Navigating Incorrect Notes

 Lack of DVT prophylaxis order written

Use of Prior Discharge Summary

 Known hemolytic anemia  Recent d/c of amlodipine

Recognition of Non-Autopopulated Labs

 Low TSH  Culture sensitivities

Abbreviations: EHR, electronic health record; DVT, deep vein thrombosis; d/c, discontinuation; TSH, thyroid-stimulating hormone.

This study was approved by the OHSU Institutional Review Board.

Results

A total of 31 out of 32 interns (97%) completed the exercise. We noted wide variance in documentation strategies. For vitals, 97% (30 of 31) of interns utilized macros to import data into their notes. For laboratory data, 45% (14 of 31) of subjects manually entered values; the rest used macros. Among these, 65% (20 of 31) employed a macro which imported 3 days of data, while the remaining only imported 1 day of data. For medications, 68% (21 of 31) of notes contained no medications at all, and the remaining either used macros or manually entered selected medications (figure, panel A).

figure.

Composition and Quality of Intern-Generated Progress Notes During Electronic Health Record Simulation

Panel A: Use of Macros Versus Manual Generation of Data for Laboratory, Vitals, and Medication Section of Note

Panel B: Strategy Used to Populate Each Section of the Progress Note as Defined by Active Problem

Panel C: Recognition Rate for Individual Safety Items Within the Chart

With respect to the quality of written information, 48% (15 of 31) of notes contained copy-paste elements and 39% (12 of 31) eliminated items from the problem list of the prior day's note. All instances occurred in the assessment and plan section, with 19% (6 of 31) duplicating the prior day's assessment. The remaining instances were spread throughout the remaining problem areas, excluding the primary problem, which was pneumonia (figure, panel B). When examining the notes for identification of safety issues, we found that all interns missed at least 1 item in the case, with a mean recognition rate of 47%. While some issues, such as recognition of the presence of hypertension, were recognized at a high frequency, issues requiring the use of a prior discharge summary (history of hemolytic anemia), looking at the medication list (absence of an active order for heparin), or identifying abnormal lab values not included in the laboratory macro (low thyroid-stimulating hormone) were poorly recognized (figure, panel C). Only 45% (14 of 31) of learners recognized that the patient's organism was resistant to the prescribed antibiotics, which caused great concern.

Discussion

This study demonstrated the use of a simulation exercise to evaluate EHR use and note-writing skills. Despite up to 3 months of clinical exposure with our EHR, there was still wide variation in the processes used to generate a note. Variations existed both in the mode of documentation and the importation of data. While macros were used extensively, there was still wide variability in the amount and quality of data imported. This variation in the process of note creation may have significant implications with respect to patient safety and may be a factor that explains the inability of learners to recognize the embedded safety issues.

There was also wide variation in the content of the assessment and plan of the progress notes, with a large number of notes containing either copy-paste information or complete omission of active issues from the problem list. The use of copy-paste functionality also introduces the potential for additional errors, particularly due to data omission. The use of a simulated case with known safety items allows for determination of the significance of these errors. While present-day interns are more knowledgeable about technology than the prior generation, their use of the EHR was still associated with a significant number of errors, suggesting that being a “digital native” is not sufficient to facilitate optimum EHR use.

There are limitations to our study. We could not control for the amount of time each intern spent on the wards prior to the exercise, nor for their exposure to an EHR prior to residency. The lack of an actual patient to examine may have also influenced the process of note creation. Finally, our sample encompasses only interns, although training level had little impact on the recognition of safety issues in our prior study.¹⁰

Conclusion

Overall, the results highlight the inadequacies of standard EHR training in the setting of advanced EHR use for data acquisition and documentation. High-fidelity EHR simulation with standardized cases designed to test effective EHR use may help to delineate improved understanding of current practice and to improve clinical and diagnostic cognitive skills. Simulation may also help inform EHR redesign by reflecting accurate use patterns. The impact of this exercise on intern note creation and performance on future simulations will be the subject of future research.

Finally, the wide variability of documentation strategies and EHR use patterns seen in interns suggests that a balance must be struck between documentation habits that promote effective clinical data retrieval/notation and the fostering of patient safety.