Establishing Electronic Health Record Competency Testing for First-Year Residents

Abstract

Background

Despite widespread use of electronic health records (EHRs), it is unclear whether residents possess the EHR skills to perform required tasks.

Objective

We assessed first-year residents' skills on specific EHR tasks.

Methods

Incoming residents were required to participate in EHR training before starting clinical rotations. The training team developed an assessment tool for 19 EHR tasks. Senior residents used a structured template to assess interns.

Results

For ambulatory workflow skills, most participants (range, 70%–100%) demonstrated competence. The 3 skills for which interns needed the most assistance were (1) creating and routing a result note (17 of 68, 25%), (2) deleting a medication or changing a dose in reconciling medications (10 of 68, 15%), and (3) finding results for the past 90 days (10 of 68, 15%). For inpatient workflow skills, most interns (range, 63%–100%) demonstrated competence. The 3 skills in which interns needed the most assistance were (1) placing a referral order at discharge (23 of 68, 34%), (2) finding a temperature on a flow sheet and trending it over time (14 of 68, 21%), and (3) creating a discharge summary, having it reviewed, and forwarding it to the primary care physician (14 of 68, 21%).

Conclusions

Our results should help EHR training teams at other institutions to better understand the strengths and weakness of EHR training approaches and to target training on tasks with the greatest performance deficits as well as toward underperforming individuals or groups.

What was known

Use of electronic health records (EHRs) has grown substantially, and EHR skills are important to the quality and safety of care.

What is new

Assessment of interns in a range of specialties on 19 EHR tasks for inpatient and ambulatory care.

Limitations

Single-site study may limit generalizability; assessment instrument was not validated.

Bottom line

The assessment identified common problem areas and underperforming residents for further targeted intervention; the approach can be readily adopted or adapted at other institutions.

Editor's Note: The online version of this article contains the survey instrument ^{(55.5KB, doc)} used in the study.

Introduction

Use of the electronic health record (EHR) has become increasingly common.¹ The Institute of Medicine has encouraged implementation of an EHR as a means of improving quality of care and patient safety. In teaching hospitals with an EHR, residents perform electronic ordering, data retrieval, data entry, and clinical documentation tasks on inpatients and outpatients using the electronic record. To date, education-based studies of EHR use in academic medical centers have focused on the impact of the EHR on training medical students and residents and on its potential as an educational tool.^2–4 In contrast, there has been little published on the training of residents on the EHR skills necessary to provide safe and competent care or on the assessment of residents' EHR skills.⁵ To our knowledge, there have been no studies assessing residents' ability to perform functions within the EHR that are directly linked to patient safety, such as the ability to accurately reconcile a medication list. The purpose of this study was to assess residents' skills on specific ambulatory and inpatient EHR tasks.

Methods

The University of California Davis Medical Center (UCDMC) is a large, comprehensive, academic medical center. The institution has used Epic (Epic Systems Corporation, Verona, WI), a proprietary EHR system, since 2003.⁶ The UCDMC has been recognized by the Healthcare Information and Management Systems Society with its Stage 7 Award, a reflection of the effectiveness of the institution in implementation and use of information technology applications to support clinical care.⁷ The UCDMC serves as the primary training site for 59 Accreditation Council for Graduate Medical Education (ACGME)–accredited residency programs with more than 600 residents.

Training with the EHR at UCDMC is coordinated by an EHR training workgroup. The workgroup includes the EHR medical director, 4 associate EHR medical directors, the associate dean of graduate medical education, 4 primary care and 3 specialty physicians, a chief resident, and representatives from nursing leadership, clinical administration, and members of the training teams from information technology, health information management, and human resources. Each year, incoming residents are required to participate in EHR training during orientation. The format of this training has changed over time, and in 2012, EHR training became classroom-based and began to include multiple points where residents practiced specific skills. The 8-hour training program was divided into 4 areas with the following time commitments: core skills (90 minutes), ambulatory workflow (150 minutes), inpatient workflow (180 minutes), and wrap-up (60 minutes). Interns were provided with pocket guides from Epic that included instructions and screen shots for common functions.

In 2012, the EHR training team established a working group to identify basic competency skills for EHR use. The working group reviewed prior EHR course content, interviewed residents on their training needs, queried the Medical Records Committee for common EHR concerns and workflow complaints, interviewed prior EHR trainers for common problems seen in training and refresher courses, and via consensus, identified a group of specific EHR skills for evaluation. Ten skills were identified as core skills for ambulatory workflow, and 9 skills were identified as core skills for inpatient workflow.

We prepared a structured assessment tool for those 19 skills. All 68 postgraduate year (PGY)-1 residents (100%) from emergency medicine, family and community medicine, internal medicine, pediatrics, psychiatry, and surgery were required to complete an observed assessment of those skills. We identified senior residents from each of those programs who performed the testing on their department's interns. We gave those senior residents the assessment template and provided instructions on how to assess each specific skill. We informed the evaluating residents that this was a no-fault assessment, meaning that the results of the assessment would be presented in aggregate to the program directors and the EHR training team, and individuals' performances would not be part of their academic record. Senior residents were instructed that they should not prompt interns on how to perform the skill, and they could use 1 of their patients or the practice version of the EHR in the assessment. For each skill, senior residents were to indicate whether the PGY-1 resident performed the task independently, with assistance, or needed additional training. Performing the task “with assistance” was defined as the intern needing 1 simple prompt to complete the task. If the PGY-1 resident needed more than 1 prompt to complete the task, they were scored as “needing training.” All assessments were done 3 to 4 months after the training session. This was to ensure that all of the PGY-1 residents possessed the ability to see patients in both the ambulatory and inpatient settings.

Ethical approval for this study was obtained from the Institutional Review Board of the University of California, Davis.

Results

Sixty-eight PGY-1 residents were evaluated using the assessment tool, representing 100% of the interns in these 6 programs. For ambulatory workflow skills (table 1) most of the interns (range, 70%–100%) demonstrated competence in all 10 skills without assistance. The 3 ambulatory workflow skills in which interns needed the most assistance in descending order were (1) creating and routing a result note (17 of 68, 25%), (2) deleting a medication or changing a dose in reconciling medications (10 of 68, 15%), and (3) finding results for the past 90 days using Results Review (10 of 68, 15%).

TABLE 1.

Outcomes of Electronic Health Records Ambulatory Workflow Competency Skills for Postgraduate Year-1 Residents in Emergency Medicine, Family Medicine, Internal Medicine, Pediatrics, Psychiatry, and Surgery (N  =  68)

For inpatient workflow skills (table 2), most of the interns (range, 63%–100%) demonstrated competence in all 9 skills without assistance. The 3 skills in which interns needed the most assistance were (1) placing a referral order at discharge (23 of 68, 34%), (2) finding a temperature on a flow sheet and trending it over time (14 of 68, 21%), and (3) creating a discharge summary, having it reviewed by the supervising resident, and forwarding it to the primary care physician (14 of 68, 21%). Less than 5% (3 of 68) of the PGY-1 residents were identified as needing additional training for any ambulatory or inpatient workflow skills.

TABLE 2.

Outcomes of Electronic Health Records Inpatient Workflow Competency Skills Assessment for Postgraduate Year-1 Residents in Emergency Medicine, Family Medicine, Internal Medicine, Pediatrics, Psychiatry, and Surgery (N  =  68)

For 1 task, medication reconciliation, we were able to compare a subset of 13 PGY-1 interns' actual performance versus their performance on the skills assessment. The family and community medicine interns have a continuity clinic in which they see assigned patients. The family practice clinic monitors ongoing compliance with medication reconciliation for all of its providers by using the process measure “Marked as reviewed” status. A summary of their performance is presented in table 3. In aggregate, the PGY-1 residents' actual performance of medication reconciliation improved over time, rising from 57% (108 of 188) in July 2012 to 80% (141 of 177) in November 2012.

TABLE 3.

Medication Reconciliation Performance by Family & Community Medicine Postgraduate Year-1 Residents (N  =  13), July–November 2012

Discussion

We determined that it is possible to perform competency assessments of specific EHR skills for a cohort of PGY-1 residents who were recently trained on EHR use. Our results showed that most interns were able to complete most of those tasks without needing assistance, and only a few interns were deemed as being substantially deficient in those skills and required additional training. However, there were areas of concern on specific skills in the ambulatory and inpatient setting. In the subset of family and community medicine interns, we were able to demonstrate improvement in the specific task of medication reconciliation over the 5-month observation period.

Although there have been no studies assessing a resident's ability to perform specific EHR functions, 1 study⁵ assessed residents' “computer skills.” Jwayyed and colleagues⁵ assessed the personal computer skills of emergency medicine residents, focusing on 23 tasks. It showed a similar variation in skills among the tested residents and concern for competence in specific tasks. For example, 31% (38 of 122) of the residents could not adequately perform a medical literature search.

Our study has several limitations. The assessment tool was self-developed and has not been validated. We used senior residents within each department to perform the assessment on their own interns, and that may have introduced bias. One prompt from the evaluator may not be clinically relevant for some of the assessed skills. We did not assess the incoming PGY-1 residents' skills with EHR use because prior experience with the EHR may have influenced the outcomes of the assessment. The residents' personal experience with EHR use, especially regarding how much ambulatory or inpatient care they were providing during the initial 3 months after EHR training, may have varied and affected performance on the test. Finally, EHR use may differ across specialties, affecting performance on specific tests.

The recent inauguration of the ACGME's Next Accreditation System will change how programs assess resident competence, with all residency programs expected to use educational milestones as the means to document progress.⁸ Our study found that not all residents achieved competence in specific and important EHR tasks, which might affect the quality and safety of patient care. For example, a resident who does not perform medication reconciliation accurately may contribute to a subsequent medication error. Further research in the assessment of EHR-specific skills is warranted, along with measures to ensure ongoing competence and the potential impact on patient safety.

Conclusion

Our testing intervention should help other institutions' training staff to better understand the strengths and weakness of EHR training programs, to inform interventions to enhance EHR use and improve patient safety, and to target additional training for underperforming individuals or groups. Physician-to-physician program-specific training may serve as an important adjunct to optimize use of clinically relevant EHR tools.