Model Guided Design and Development Process for an Electronic Health Record Training Program

Abstract

Effective user training is important to ensure electronic health record (EHR) implementation success. Though many previous studies report best practice principles and success and failure stories, current EHR training is largely empirically-based and often lacks theoretical guidance. In addition, the process of training development is underemphasized and underreported. A white paper by the American Medical Informatics Association called for models of user training for clinical information system implementation; existing instructional development models from learning theory provide a basis to meet this call. We describe in this paper our experiences and lessons learned as we adapted several instructional development models to guide our development of EHR user training. Specifically, we focus on two key aspects of this training development: training content and training process.

Introduction

Effective user training has been recognized as an important factor to improve health information technology (IT) implementation success, but development of a successful training program remains a challenging task^1,2. Part of the reason is that health care organizations lack industry-wide best practices in electronic health record (EHR) training, so often have to learn from their organization’s own training experiences, with can be costly and inefficient³. While current literature often introduces some design principles, there is still a gap in guidance on how to translate those principles into varied clinical practices or settings⁴. This literature may report on how organizations conducted the training and general principles, but the detailed process of developing training programs is often underemphasized and underreported. The literature may also include lessons from costly “do-overs” after the training program go-lives^3,5. Ideally, by following an instructional development model, those issues might be detected before the go-live, which could save money and time for clinicians and health care organizations.

American Medical Informatics Association (AMIA) board members published a white paper entitled “Core Content for the Subspecialty of Clinical Informatics”, which identifies core educational content for clinical informatics⁶. With continued national mandates for health IT adoption and use, the training needs for information systems in clinical settings, most notably EHRs, are expected to grow⁷. Among the numerous knowledge and skill content areas addressed by the AMIA board members, knowledge under “clinical information system implementation” includes “models of user training and support processes that can meet clinician needs” as one of three sub-topics⁶.

Instructional development is a well-established domain, with over 100 validated instructional development models⁸, and some of them may be particularly applicable to EHR training. For example, the Instructional Systems Development (ISD) model is a generic model developed in the 1950s to meet the needs of military and commercial organizations⁹. In ISD, training consists of five basic stages: analysis (A), design (D), development (D), implementation (I), and evaluation (E); referred to as the ADDIE model⁹. The ADDIE model is comprehensive in content and systematic in procedure, and covers almost all necessary components needed to carry out an effective training program. To meet a variety of specific training needs, other models build on the ISD model, and improve some of these stages. For example, the development stage may take a long period of time, and may fail to meet the pace of technology updates, e.g., EHR. The rapid prototyping model tries to facilitate training development by performing several elements simultaneously, condensing the generic ADDIE model into a four-level process, including: a) performing a needs analysis, b) constructing a prototype, c) utilizing the prototype to perform research, d) implementing the final system¹⁰.

The ISD model also fails to embrace the fact that reality can be different from what was planned, and may change over time, which is typical in complicated health care settings. Therefore, the R2D2 model (Recursive, reflective, design and development) provides the ability to update the training plan over time¹¹. Reflection involves critically considering work to date, and revising training plans and materials based on observation and other feedback. The recursive nature of the process means making the same decisions many times throughout the design and development process, so initial decisions are not necessarily the “final ones”. The implication of this model is that a training design team should actively reflect on and revise current training to meet ever-changing situations.

The current accounts for successes or failures in current EHR training often focus on the final format of training, and it is rare to see detailed descriptions of the training development processes. In this paper, we try to bridge that gap by discussing our experiences developing an EHR training program. The primary focus of this paper is to detail the systematic process of developing the training and lessons learned through our iterative development process. We consider training development as a problem that should take a sociotechnical system perspective, and adopted an iterative approach similar to rapid prototyping model and R2D2. We also conceptualize a two-phased EHR training development focused on training content and training process. We discuss what we learned during the design and development process that led to a successful training session.

Methods

Instructional models typically suggest that before training program development, one needs to conduct a thorough analysis to define training needs⁹’¹⁰’¹¹. We first present how to conduct this type of detailed analysis from a sociotechnical systems perspective⁶, and then detail how we designed and developed training content and processes based on these analysis results. Figure 1 outlines the conceptual model of training development we adapted from ISD, R2D2, and rapid prototyping for EHR training program development. We use the model to guide the design and development of EHR training for nursing students at a large public university, and we address how we conducted each stage of this model in this section, with focus on the development of content and process. We chose to use the clinical process of obtaining a medication history prior to the design phase, because it is an error-prone process involving comprehensive EHR-related competencies (e.g., information retrieval, documentation, decision making), and it has been used in comprehensive practice sessions in previous EHR implementation training in other health organizations⁵’¹².

Figure 1.

Conceptual model of training design and development process

The authors of the AMIA white paper on “Core Content for the Subspecialty of Clinical Informatics” adopt a holistic sociotechnical systems perspective, depicting clinical informatics at the intersection of three domains: clinical care, information technology, and health systems⁶. In order to develop a training program that fits the need of clinicians and reflects the complex nature of healthcare settings, it is necessary to consider EHR training as a sociotechnical systems problem. This view is particularly important during the analysis stage of an EHR training needs assessment, because these analysis results will influence decisions related to training design and delivery.

The training analysis phase should define training goals, and gain an in-depth understanding about the users and their required tasks, including their typical clinical practices, informatics needs, responsibilities, and current competencies. This information can be used to determine what resources, including EHR functions, physical environment, and infrastructure will best serve users’ needs.

Analysis of training goals

The goal of our training program is to help senior nursing students use an EHR efficiently and accurately while obtaining a patient’s medication history. More specifically, after training, nursing students should be able to:

• Use functions related to medication history taking to navigate, search for information, retrieve information, and document new information into EHR

• Reconcile any potential discrepancies in patient information in the system, such as omissions and out-of-date medications

Analysis of stakeholders

We involved a multi-disciplinary team in the training design process, including individuals with nursing, computer science, and human factors backgrounds. In addition, the targeted trainee group participated extensively in the development process.

Analysis of trainees’ characteristics

Training analysis must address the trainee’s current competencies in completing the clinical process of interest, and current EHR competencies, in order to define their training needs. If the trainee is insufficiently prepared to complete the clinical care process, training must first address that competency prior to conducting EHR-specific training. Studies reported best practice principles in EHR implementation training that align with this approach, which identified a need to “assess users’ skills and training needs” and “match training to users’ needs” as important best practices⁴. The analysis must also assess any technical competency prerequisites for using the EHR (e.g., using a keyboard and mouse, opening software).

In terms of competencies, the nursing students in our program had received classroom training in pharmacology and obtaining a medication history, had some clinical experience with EHRs through their clinical experiences (though not the EHRs used during training), and were competent with basic operations with computers (e.g., using a mouse and keyboard, and opening software).

Analysis of clinical tasks

To determine what tasks to train users on, an analysis should start with a clinical care process of interest, rather than the technical capabilities or functionalities of the EHR. The analysis should also determine the trainee’s roles and responsibilities within the process. In our training analysis, we first analyzed the clinical process carried out by nurses when obtaining a medication history. We built on an existing clinical process flowchart defining required clinical tasks¹².

Starting training from the perspective of a familiar clinical process is consistent with adult learning theory, which states that adults are more motivated to learn things relevant to their needs and experience¹³. This analysis can help avoid overloading the trainee with irrelevant information and EHR functionalities, which is a typical cause to ineffective training⁵.

Analysis of EHR technical functions

The training analysis should determine what technical aspects of the EHR (e.g., functions) are used during the clinical process of interest. After extensive review of the literature and analysis of EHRs, we identified what EHR functions were required to complete the clinical process, including both overhead functions (e.g., logging into the system) and domain functions (e.g., reviewing the current medication list in the EHR) as described by Zhang et al. ¹⁴. Specifically, trainees need at least six EHR functions to complete the clinical task, including:

1. Login to the system

2. Search for and retrieve a specific patient’s chart

3. Retrieve basic patient information, such as allergy, identity, vitals

4. Retrieve and document a patient’s medication list

5. Retrieve and navigate through a patient’s problem list

6. Retrieve and navigate through a patient’s past history, such as social, medication, surgery

Analysis of task-EHR integration/ workflow

Sociotechnical systems models suggest that we need to analyze the often-unexpected impact of an EHR on existing system elements, such as workflow¹⁵. For example, EHRs can impose high cognitive load for clinicians, making clinical tasks error-prone. Therefore we clearly mapped out the links between the clinical care process and the EHR use, including what tasks should be completed with the EHR, when in the process the EHR will be used, and how the technology will impact the trainee’s workflow for the clinical process of interest. The results from this analysis are used in a clinical task-EHR integration/work training section. As an EHR will likely impact the clinical care process, the analysis helps mitigate negative impacts of these changes, and explicitly address any redesigned clinical workflows in the training¹⁶.

Analysis of resources and constraints

Based on nursing students’ typical schedule, an ideal training session should be no more than 90 minutes to achieve best attendance rate and keep trainees attention. The physical environment is a Class Lab in nursing school, with a projector and white board. The training utilized laptop PCs with Windows Operating Systems with each trainee using their individual computer. The EHR is web based, which is accessible using a web browser.

Design and development

Based on the analysis results, we needed to make two main decisions: what to train (content) and how to train (process). We first decided necessary sections in the training program, which include clinical process review, pre-training evaluation of EHR competency, EHR use training, workflow integration training, and post-training evaluation. Then we worked on content and process for each section. Before we involved students in the study, we obtained approval from the Institutional Review Board.

Following the idea from the instruction development models, the general strategy to develop training content was to prototype materials for each training section, then test the materials with real trainees as they used the materials and processes. We worked to understand their perceptions, and reveal any confusions and ambiguity in the training materials. We worked with six different trainees individually on materials and processes, until we could identify no issues to be addressed and we were ready to roll out the training.

With the first three trainees, we focused on the content of the training. We asked each trainee to go through the training materials in order, observed them learning without imposing any time limit, and documented their time and questions. We asked them to think out loud so we could understand points of confusion, and tried to clarify verbally until they understood. With the second three trainees, we focused on the process of the training. We followed specific training protocols, with time limits for each section. We observed and documented issues as trainees went through the training, and communicated with them about the issues and their concerns at the end of the training.

We summarize major issues identified with each participant, and our corresponding changes in Table 1.

Table 1.

Training program development iterations (Iteration 1-3 is content oriented, 4-6 is process oriented)

Participant	Major Issues	Training Design Changes
1	a. The trainee was unfamiliar with some EHR technical terms when the terms are not commonly used in clinical care. b. The trainee lost attention when a video tutorial introduced too much information at a time c. The trainee remained confused about what she needed to do with EHR with different medication history sources after clinical process review d. The trainee did not understand why the EHR-workflow tutorial was related to “how they conduct clinical care”, and did not understand the task-EHR flowchart mapping e. The trainee did not feel comfortable in the physical environment, feeling too hot	a. Simplify or define terms to fit clinicians’ language. For example, “search for information” sounds more familiar than “information retrieval”, and “enter data/notes” may be more familiar than “documentation”. b. Instead of introducing all EHR functions at once, introduce two functions at a time, followed by a hands-on practice c. Add a review question about reconciling with multiple medication history sources and using EHR, and provide answers d. Redevelop task- EHR integration tutorial, making it more relevant to clinical care, and present the integrated workflow chart in resemblance to clinical process diagram e. Conduct training in an air-conditioned environment
2	a. The trainee did not get some key details of instructions printed on paper b.The trainee did not know what to do during hand-on exercise besides repeating what was demonstrated in the video	a. Add a training instruction transcript for all sections, and verbally communicate what to do before each section b.Add specific goals and assignments to the hands-on exercise, which covers the contents just demonstrated
3	a. The trainee had a problem organizing and finding files	a. Distribute files only when they are needed for each section, and ask trainees to put away in a file folder in checklist
	b. The trainee frequently asked “what’s next”	b. Add a file called progress checklist to list all training sections, tasks and time in order, so a trainee can have expectations and more control
4	a. The trainee was quicker to complete tasks than the time set for the training, and had to wait for next sections to begin b. The trainee’s cell phone rang during the training	a. In the instruction/protocol, the time set for each section is changed to the maximum time allowance, making it more self-paced; a smaller learning group (1 to 3 trainees per training session) can better accommodate different learning capabilities. b. Add a welcome PowerPoint slide projected to whiteboard with silenced cell phone reminder
5	The trainee seemed to miss some key information about task-EHR integration tutorial printed on paper	Provide video version tutorial in addition to paper version
6	The trainee went through the materials and processes very smoothly. Ready for go-live.	None.

After 6 iterations of development, we finalized the training program as described in Table 2.

Table 2.

Training program outline

Section	Brief Description of Process	Duration (min)
Check in	Trainees are seated and learn the goals and process of the training	5
Clinical process review	Trainees review the medication history taking process, answer review question, and get feedback via presentation of the solution	5
EHR Case 1	Trainees login to the EHR and complete patient Case 1 with no training, as a pre-test.	12
System use tutorial	Trainees watch a video demonstration of the EHR functions, and have hands-on practice of each function.	15
EHR and clinical process integration tutorial	Trainees watch a video demonstrating how to integrate the EHR functions into the clinical process, have hands-on practice, and get feedback	13
EHR Case 2	Trainees complete a more complicated patient case using the EHR	12

Implementation and Evaluation

We implemented the final version of training program in the nursing school Class Lab for 9 groups of trainees (n = 18 in total); each group had 1 to 3 trainees based on their availability. After training, we evaluated key aspects of the program by analyzing their performance data, and conducted interviews to gather their opinions about the training. Every session progressed as expected. Most trainees managed to complete the training slightly faster than scheduled.

Preliminary analysis of their performance showed the training was effective as measured by trainees’ prepost training performance comparison: 50% of trainees were able to correctly document the intended medication in case 1 (pre-training) compared with 100% of trainees able to correctly document the intended medications in the more complex case 2 (evaluation). Trainees were able to do it much faster too: for those able to complete case 1, it took them on average 532 seconds (95% CI = 455-609); Trainees took on average 430 seconds (95% CI = 373-487) to complete the more complex case 2. The reduced variance in duration is remarkable, because it is considered as one important indicator of an effective training, even with more trainees completing the task (n = 9 vs. n = 18). The results of their performance are summarized in Table 3.

Table 3.

Performance metrics

	Pre Training Session (Case 1)	After Training Session (Case 2)
# of Trainees who completed the case	9 out of 18	18 out of 18
Duration of completion (in seconds with 95% confidence interval)	532 (77)	430 (57)

In addition to the quantitative measures, we conducted interview with trainees to gather their opinions about the program. We asked them to comment in terms of usefulness, clarity and any other aspects related to the training quality, and we recorded their comments. Their feedback was highly positive, and some frequent quotes include:

• “It (the training process) is smooth and well-planned.”

• “They (the materials) are really clear and well explained.”

Discussion

Instructional models, such as ADDIE or R2D2, can be helpful to guide a successful EHR training program development process. They help identify and remedy potential issues before the training program execution, to avoid potential costly failures and do-overs. It is a best practice to involve targeted trainees in the process, and understand their work, needs, characteristics, and current competencies, and update the training materials and processes to best serve their needs. However, the training design and development process can be frustrating during early stages; for example, during our early development phase, a trainee involved in testing told us, “do not teach us how to use the (cumbersome) system, go fix it.” While an ideal and user friendly system is the ultimate goal, a good training program is necessary for users to take full advantage of any EHR system.

In order to make progress toward the correct direction during design and development process, different models all suggest conducting extensive analysis to map out the specifics of the planned training, such as trainees, resources, training goals. Based on the analysis results, we can go through iterations of prototyping, testing, redesign and development until we are confident the program is ready to be implemented. It is difficult to have a perfect training program through just one round of development, and we can always learn new things based on different feedback. The trainees who involved in the development process could also serve as super user instructors to other trainees.

The development of an EHR training program should focus on content and process, which are both key factors to ensure success. The content and process are interactive factors, as different content may be best trained using different methods or durations. Our experience was to emphasize more on content clarity and learnability during early stage, and focus more on process toward later stage to avoid trainees’ loss of attention and fatigue, such as self-pace learning, video demonstration vs. paper based content presentation. The model presented in this paper provides guidelines and principles for the design of content and process, and can be applied to various settings and tasks. However, different training programs may vary in detailed content and process based on training needs and resources, and everything should be vigorously tested before implementation.