A Unified Approach to Clinical Informatics Education for Undergraduate and Graduate Medical Education

Abstract

Vanderbilt University Medical Center has adopted a unified approach to undergraduate and graduate clinical informatics education. Twenty-three learners have completed the course which is designed around four key activities: 1) didactic sessions 2) informatics history and physical where learners observe clinical areas, document workflows, identify a problem to solve and propose an informatics-informed solution 3) informatics clinic where learners are side-by-side with practicing clinical informaticians and 4) interactive learning activities where student groups work through case-based informatics problems with an informatics preceptor. These experiences are coupled with opportunities for asynchronous projects, reflections, and weekly assessments. The curriculum learning objectives are modeled after the clinical informatics fellowship curriculum. Feedback suggests the course is achieving the planned goals. It is a feasible model for other institutions and addresses knowledge gaps in clinical informatics for undergraduate and graduate medical education learners.

Introduction

From the delivery of direct patient care to research and discovery, the role of informatics in the healthcare system continues to grow. With this growth, the role of the physician informaticist and informatics education for physicians has evolved over time. Historically, physicians with interest in informatics who desired formal training pursued graduate degrees in biomedical informatics to develop their skills and proficiency in informatics concepts and methods. Many of these programs are funded by the National Library of Medicine and are well-suited for the physician preparing for a research-focused career¹. However, the rapid adoption of electronic health records (EHR) resulting from the Health Information Technology for Economic and Clinical Health (HITECH) Act introduced technology into the healthcare setting and enabled the adoption of more advanced clinical information systems to support the delivery of patient care². With this evolution, physician interest in the application of informatics tools and methods to healthcare delivery increased, and physicians found themselves working in an applied clinical informatics role with less emphasis on research. In the early 2000s, Clinical Informatics leaders from the American Medical Informatics Association (AMIA) recognized the growing need for Clinical Informatics expertise and the necessity to standardize training and certification of Clinical informaticists. Working with the American Board of Medical Specialties (ABMS), the American Board of Preventive Medicine (ABPM) and the Accreditation Council for Graduate Medical Education (ACGME), they described the requirements for Clinical Informatics subspecialty training and certification and created a pathway for formal fellowship training^3,4. They recognized the critical role of Clinical Informatics leaders in the healthcare setting was as much about strategy and leadership as it was the information technology itself and captured this in the Core Content for the specialty⁵. Similar to traditional clinical fellowships, Clinical Informatics fellowships emphasize the practical application and supervised practice of fellows while also ensuring understanding of foundational informatics concepts and methods. The first board certified Clinical Informatics physicians became diplomates in December 2013, and the number of accredited fellowships has grown rapidly⁶.

With the growth and standardization of formal informatics training for physicians at the fellowship level, some in the informatics community have called for the earlier inclusion of informatics training in the undergraduate and graduate medical education curricula^5,7,8. Others have proposed standardized goals and learning objectives for learners in the earlier stages of their medical education^9,10. In recent years, the number of institutions with established programs earlier in medical training has increased. However, training opportunities for medical students and housestaff are less standardized. Several undergraduate medical education (UME)-based programs focus on developing research skills^11–13. Others offer experiential learning through shadowing and self-directed learning through assigned readings and journal clubs^14–16. One program redesigned their elective to increase the clinical applicability while still incorporating important aspects of the research experience and creating a more flexible rotation experience for fourth-year students¹⁷. This program supports integrating their Clinical Informatics fellows into the rotation to create a structure similar to traditional clinical rotations. However, much of the experience is delivered in an asynchronous format to support fourth-year students needing to set aside time for interviews.

From a graduate medical education (GME) perspective, curricula are also variable^18–24. Time commitments for these experiences range from two weeks to two years. Accordingly, the content varies from experiential shadowing experiences and self-directed learning to hands-on project work and electronic health record (EHR) certification.

Here, the authors describe Vanderbilt University Medical Center’s approach to clinical informatics education for undergraduate and graduate medical education. This program has three unique aspects. First, the curriculum is designed to survey the field of clinical informatics, focusing on content areas covered in the clinical informatics fellowship while also arming students with practical skills they can carry with them through the rest of their career, regardless of final specialty and practice setting. Second, the curriculum is delivered through familiar learning modalities which mirror those found in a traditional third and fourth-year UME clinical curriculum. Last, we intentionally integrate undergraduate and graduate medical education learners into the same experience. This approach allows us to cover more clinical informatics content while also exposing learners to other career opportunities in informatics.

Methods

In 2015-2016, the Vanderbilt University School of Medicine curriculum underwent a transformational redesign²⁵. As part of the curriculum redesign, the School of Medicine implemented “Integrated Science Courses” to integrate the teaching of foundational science with clinical application during the third and fourth years of medical school. The Clinical Informatics Integrated Science Course was developed in 2020 and piloted for three students in May 2021. This four-week elective came on the heels of a new GME elective in Clinical Informatics developed in 2019. Historically, housestaff involvement in informatics took the form of personalized one-off rotations with individual faculty members often focused on a longitudinal research experience. We shifted the experience to develop a more standard way for residents and fellows to interact with practicing Clinical Informaticists with emphasis placed on day-to-day application of informatics concepts in clinical practice and in the healthcare system. The learning modalities used for the GME elective became the foundation upon which we further developed the existing approach, which includes learners from all levels of the medical education path.

The course is currently offered twice per year at Vanderbilt University Medical Center. Each session is four weeks, contains multiple learning modalities and hosts a mixture of UME and GME learners.

As practicing physicians, the authors wanted to demonstrate that many of the same skills required to care for patients, particularly related to critical thinking and leadership, are necessary in the study and practice of clinical informatics. The course experience closely mirrors a traditional clinical rotation, highlighting the correlation with patient care. Just as the patient is at the center of clinical care, in clinical informatics, learners treat the healthcare system as their patient. The curriculum is designed around four primary learning modalities shown in Figure 1. There are also multiple modes of assessment reflecting a more comprehensive evaluation of a student’s performance.

Figure 1.

Overview of the Clinical Informatics UME and GME Course Structure and learning modalities.

Students come to the course with diverse background, knowledge, and experience in informatics. We have had students ranging from those for whom this was their first interaction with the field to those who had completed a PhD in Biomedical Informatics and from all different clinical specialties. As a result, didactics teach foundational knowledge to create a common knowledge base between students. Didactics are more commonly included in the earlier weeks of the course to enable students to apply that knowledge in clinical informatics practice during other portions of the course.

We considered American Health Information Management Association (AHIMA)²⁶ and American Medical Informatics Association (AMIA)^5,27 and the Clinical Informatics GME training curricula but most closely aligned our course objectives with those of the Clinical Informatics GME training. However, our goal was to develop a course that surveys the field of Clinical Informatics providing broad exposure to the field and preparing those who were interested in more in-depth study.

During Informatics Clinic, students spend time with practicing Clinical Informaticists to understand how the foundational knowledge they learned during didactics applies to the day-to-day practice of Clinical Informatics. Informatics Clinics are the least used learning modality and are more heavily concentrated in the last two weeks of the course. Informatics Clinics must be carefully selected to ensure the students have the foundational knowledge and context to enable their integration of the informatics concepts into practice. Even if GME learners have not had previous informatics training, they bring to the rotation a more sophisticated view of the healthcare system and, therefore, can often participate higher-level meetings and discussions that would be less valuable for UME learners.

The Informatics History and Physical component of the course allows students to apply the same critical thinking skills they use in patient care to a problem facing the healthcare system that could be improved with informatics techniques. Learners spend four hours each week in a clinical setting observing workflow using qualitative methods. Students may work with physicians, advanced practice providers, pharmacists, nurses, medical assistants, or any other member of the health care team. They may observe in an ambulatory, inpatient, perioperative or procedural area. UME learners are assigned to shadow a member of the healthcare team in a designated clinical area. GME learners shadow a member of the healthcare team from their specialty. Learners evaluate the healthcare system for areas of improvement, consider options to intervene, discuss how changes may impact other parts of the healthcare system, and how to measure the success of interventions. This process is intended to mimic patient diagnosis, treatment, and management. Learners present their History and Physical results to their peers and Clinical Informatics leaders where they are evaluated on the following criteria: (1) accurate representation of the problem they’re trying to solve; (2) justification to support why this is an important problem to address; (3) completeness a stakeholder assessment; (4) the completeness of their proposal to optimize technology, processes or the way people use them; (5) a mock-up or low-fidelity protype of their solution; (6) quality of the presentation and organization of the slides. Clinical leaders from respective areas are invited to attend. Compelling proposals have been shared with our operational IT leaders and implemented. This allows Course Directors to evaluate history-taking, critical thinking, and presentation skills as occurs during rounds on a traditional clinical rotation.

During Interactive Learning Activities (ILA), we explore informatics in more detail through problem or case-based learning. These half-day sessions typically require pre-reading and start with a brief lecture to ensure common baseline understanding of key topics among all learners. However, the majority of the time is spent with learners putting their new skills into practice through real-world scenarios precepted by the Course Directors. Topic covered in an ILA include: (1) Workflow analysis and Organizational Anthropology; (2) Clinical Decision Support; (3) Privacy and Security; (4) Secondary Use of EHR Data; (5) Decision Science; (6) Genomics; (7) Careers and Leadership. This modality serves to demonstrate the practical application of topics discussed in other areas of the course. Additionally, it serves as a method for the Course Directors to directly observe learners applying clinical informatics concepts to practice and problems much in the same way leaners are evaluated on their critical thinking, examination, diagnostic and management skills in a traditional clinical setting.

All course activities expose learners to the interprofessional nature of clinical informatics. Students attend Informatics Clinic with and are given lectures by nurse and pharmacy informaticists as well as informatics research faculty. In informatics H&P, students are often paired with non-physician team members in order to understand the multidisciplinary role of the informaticist.

Each week of the course, learners are assigned a project. They can work on these asynchronously throughout the week. Similar to the way procedures in the clinical setting reinforce the use of medical tools, projects are hands-on assignments to help students gain a better understanding of common informatics tools and resources they may find useful in their future careers. Project topics include: (1) leveraging tools in the EHR for efficient documentation; (2) creating a data collection instrument to support a real-world quality improvement, research or practice initiative; (3) leveraging data and reporting tools to answer important clinical questions; (4) applying design elements and concepts to enhance user interface and user experience.

All learners complete a journaling assignment on a topic of their choice discussed that week. This serves to provide a written record of their thoughts and feelings about that topic which we consider analogous to writing a clinic or progress note. It is an opportunity for learners to synthesize ideas, demonstrate integration of related concepts and show the depth of their understanding of that topic.

Additionally, UME learners complete a one-hour, weekly, open book quiz containing 20-30 questions which covers content from any course activity.

UME learners are given a numeric evaluation for each of the assessment activities, as well as a milestone’s-based assessment based the School of Medicine curriculum. GME Learners participate in all activities, but they do not receive a grade for their elective.

UME learners are invited to participate in an optional, 9-question self-assessment of clinical informatics knowledge prior to and following the session. Questions are listed in table 1, below. Student responses are based on a 5-point Likert scale from strongly disagree (1) to strongly agree (5).

Table 1.

Pre- and post-course survey

I understand the role of the Electronic Health Record (EHR) in the delivery of safe, effective, efficient, patient-centered healthcare; I understand the other roles of the EHR in the healthcare system.

I am able to define and understand the roles clinical decision support (CDS) plays in delivering patient care including quality, safety and legal issues.

I understand patients’ rights with respect to their health information and how health information may be used and shared with others. I am able to describe the processes, systems and regulations in place to protect those rights.

I understand how healthcare organizations are required to handle health information. I am able to describe the physical, technical, and administrative safeguards that must be in place to keep health information secure.

I am able to define the 3 domains of genomic testing that directly support patient care. I can describe the pipeline of data flowing from genomic sequencing to a final report that is ready for a patient and a provider.

I understand why we need data standards and can describe the elements of the healthcare system which use data standards in the delivery of routine care.

I understand the roles that people, process and technology play in the successful implementation and adoption of health information technology.

I am able to outline the elements contributing to a positive user experience.

I understand available training opportunities to prepare me for a career in biomedical informatics.

Students are also asked to rate on a scale of 0 (lowest) to 10 (highest) their interest in pursuing clinical informatics as a career before and after taking the course.

Two planned focus group sessions were held during each course offering, one at the course mid-point and one on the final day of the rotation. Learners are asked to share their general perceptions of the course to identify topics that warrant further exploration. Additionally, they are asked to use the keep-stop-start framework to identify strengths and weaknesses of the elective. Qualitative data were analyzed for themes to identify strengths and opportunities for the course.

Results

Since the inception of the course, we have had fifteen UME and eight GME learners, with course enrollment increasing with each of the three offerings. It is currently offered twice yearly and is open to third and fourth year medical students and all housestaff.

Of the fifteen UME learners who have completed the course, nine completed both of the optional pre-and post-course questionnaires. Results of the pre- and post-course responses can be seen in figure 2 below.

Figure 2.

Pre-course and post-course self-assessment survey results (full prompts in Table 1), n=9.

Mean pre-test assessments and post-test assessments were 3.8 and 4.8 for understanding the fundamental role of the EHR; 2.9 and 4.9 for understanding the role of clinical decision support in patient care; 3.4 and 4.6 for protecting patient privacy practices; 2.7 and 4.6 for data security practices; 1.9 and 4.6 for data standards; 3.1 and 4.8 for organizational anthropology; 3.0 and 4.4 for human factors and design; and 3.1 and 4.6 for professional development. The mean level of clinical informatics interest was 5.7 prior to the course and 7.4 following the course on a 10-point scale. The average pre-course assessment across all nine objectives was 2.9 with an average post-course assessment of 4.7. Students entered the course most comfortable with understanding the fundamental role of the EHR (3.8). Students entered the course least comfortable with bioinformatics (1.9). Students left the course most comfortable with understanding clinical decision support’s role in patient care (4.9). The bioinformatics learning objectives saw the greatest increase in comfort (1.9 to 4.6).

Several common themes arose across the six focus group sessions. Focus groups consisted of medical students from each course section meeting with course directors. A total of fifteen students participated in the focus groups, three from the first section, 6 from each of the second and third sections. We separated the themes into a keep, stop, start framework summarized in Figure 3 below.

Figure 3.

Common course feedback themes across six student focus groups.

Discussion

When developing this course, we felt that existing curricula, including previous versions of our own, did not meet the needs of UME and GME learners. It is easy for the uninitiated to come into the course thinking that clinical informatics is any one of the foundational sciences or buzzwords they have heard recently—computer science, artificial intelligence, clinical decision support. As a result, one of our first goals was to ensure that learners leave with a clearer understanding of what clinical informatics is. We developed a curriculum that would survey the field of informatics to provide practical skills they can apply in their future clinical practice and prepare them for future study in the field. We also wanted an experience that emulated a traditional clinical rotation in order to demonstrate how many of the same skills they use as a physician apply to the practice of clinical informatics. Lastly, we wanted to create a structure that allowed learners at different points in their training to learn together. Even when GME learners did not have prior informatics exposure, they brought a more sophisticated understanding of the healthcare system. While we did not have a structured assessment of how peer teaching contributed to this course, integration of UME and GME learners has been a welcome addition to the course based on feedback from student focus groups.

While our optional surveys are not a validated instruments, they were designed to measure the effectiveness of the course in meeting our stated learning objectives. Overall, the trend is toward understanding the stated learning objectives, which is a primary goal for the course. The students were more tightly clustered entering the course on some topics such as the role of the EHR (Figure 2, panel A) while other topics such as data standards, human factors engineering, and organizational anthropology show a broad distribution (Figure 2, panels F-H). This likely reflects two separate issues. First, students enter informatics courses with a wide variety of backgrounds. Second, many informatics terms may be misattributed. A common misattribution students make in the course, for instance, is confusing the concepts of privacy and security.

Focus groups provided invaluable feedback. We identified three themes in the “keep” category of responses. First was “concepts unique to this course.” While there are topics that many participants expected to learn about when they signed up for an informatics elective, such as discussions about the EHR, clinical decision support or machine learning, several students commented that they had no prior exposure to topics like organizational anthropology or formal discussions about leadership and conflict management. Students relayed that these topics changed their perceptions of the practice environment and the day-to-day operations of a healthcare system. The second theme in that category was “engaging learning modalities.” Multiple students remarked that didactics used in this course did not feel like traditional medical education didactics. For most of our didactics, we teach, or encourage guests lectures to teach, informatics through real-world cases, projects, implementations or other initiatives led by the speaker. This “Ted Talk” style was endorsed as an engaging and highly effective way to deliver content. In response to this feedback, we increased the number of didactics offered during the course. In this same category, students appreciated the hands-on, real-world application and scenarios used for the Interactive Learning Activities. Many of the cases in these settings are actual or closely based on actual scenarios encountered by the Course Directors in their Clinical Informatics practice. The last theme in this category was “unique experiences.” Since the healthcare system is our patient, we try as much as possible to include learners in meetings with clinical leadership where decisions that impact the healthcare system are made. Learners expressed that this gave them a behind-the-scenes view of how a medical center is run, a perspective they had never seen before.

In the “stop” category of themes, students identified a couple of ideas related to “unpredictability.” First was the lack of predictability in scheduling Informatics Clinics. There is not a predictable schedule of high-quality Informatics Clinic experiences from week to week and they would often be added relatively last minute to the students’ schedule. This unpredictability created a challenge for them when they needed to schedule other meetings for their longitudinal research work. Additionally, when operational informatics meetings came with little notice, it was difficult to ensure learners had the foundational knowledge and context to have a valuable experience. After receiving this feedback during the mid-course focus group of our first course offering, we became much more selective about learner participation in Informatics Clinic. We reorganized to make sure we consider the timing of related lectures and to ensure there can be adequate time for pre-meeting discussion for contextual awareness.

The only theme identified in the “start” category is “personalization.” This came out of two primary discussion points. First, multiple medical students requested to spend their Informatics H&P time in the clinical area coinciding with their specialty of interest as a way to demonstrate their interest in the specialty. While we conceptually support the idea, it is logistically impractical as the planning for student observation experiences in these areas takes place months ahead of when student enrollment in the course is complete. As a compromise, we offer to provide feedback for any student wishing to conduct an extra Informatics H&P observation and presentation and to invite clinical leadership from that specialty to the presentation. Second, is a request we’ve heard from some leaners to incorporate more hands-on data science activities. Interestingly, this contradicts feedback we received from a Focus Group of UME learners who reviewed the curriculum during its development. The group suggested that students would hesitate to participate in a course with data science or coding requirements. We feel this reflects the diverse expectations that learners have coming into an introductory clinical informatics course. We are currently considering adding programming as an optional element of the course.

Conclusion

In conclusion, Vanderbilt University Medical Center has adopted a unified approach to undergraduate and graduate clinical informatics education. Initial quantitative and qualitative feedback suggests the course is achieving the planned learning objectives. We feel this is a feasible model for other institutions. It addresses knowledge gaps in clinical informatics and creates a structure similar to traditional clinical rotations which include medical students, residents, and fellows.