Electronic Health Record Use in Internal Medicine Clerkships and Sub-internships for Medical Students Graduating from 2012 to 2016

Abstract

Background

As electronic health records (EHRs) became broadly available in medical practice, effective use of EHRs by medical students emerged as an essential aspect of medical education. While new federal clinical documentation guidelines have the potential to encourage greater medical student EHR use and enhance student learning experiences with respect to EHRs, little is known nationally about how students have engaged with EHRs in the past.

Objective

This study examines medical student accounts of EHR use during their internal medicine (IM) clerkships and sub-internships during a 5-year time period prior to the new clinical documentation guidelines.

Design

An online survey about EHR use was administered to medical students immediately after they completed USMLE Step 2 CK.

Participants

The sample included 16,602 medical students planning to graduate from US medical schools from 2012 to 2016.

Main Measures

Descriptive statistics were computed to determine the average percentage of students engaged in various health record activities during their IM educational experiences by graduation year.

Key Results

The vast majority (99%) of medical students used EHRs during IM clerkships or sub-internships. Most students reported that they entered information into EHRs during the inpatient component of the IM clerkship (84%), outpatient component of the IM clerkship (70%), and the IM sub-internship (92%). Yet, 43% of the students who graduated in 2016 never entered admission orders and 35% of them never entered post-admission orders.

Conclusions

Medical school graduates ought to be able to effectively document clinical encounters and enter orders into EHR systems. Although most students used and entered information into EHRs during their IM clinical training, many students appear to have received inadequate opportunities to enter notes or orders, in particular. Implications for graduate medical education preparedness are considered. Future research should address similar questions using comparable national data collected after the recent guideline changes.

INTRODUCTION

Over the last few decades, electronic health records (EHRs) have become integral to doctor-patient interactions and an important tool for collecting and storing medical information. Effective use of an electronic health record (EHR) is an important capability that medical students ought to learn and practice before entering a residency training program.^{1, 2} As reflected in the Association of American Medical Colleges (AAMC) core entrustable professional activities (EPAs) focused on documentation of clinical encounters,^{3, 4} medical students completing IM clerkships and sub-internships likely will need to fully access and utilize EHRs in order to be well-prepared for their roles as IM residents. Indeed, the Alliance for Clinical Education (ACE) specified in a policy statement that effective EHR use is an important medical school educational competency.¹

Until recently, guidelines from the Centers for Medicare & Medicaid Services (CMS) prevented certain aspects of student-documented health records from being used for billing purposes.⁵ As a result, teaching physicians would often reenter their students’ work, thus increasing their already taxing workload and limiting opportunities to use EHRs as a teaching tool. In general, little, if any, incentive existed for health systems to encourage medical student use of EHRs. New CMS clinical documentation guidelines now permit the previously unusable parts of student entries to become part of the official medical record after verification by teaching physicians.⁶ Medical educators and compliance officers have been encouraged to capitalize on this change to maximize student learning through revision of local student documentation policies.⁷ These new guidelines and possible subsequent healthcare system policy changes have the potential to alleviate unnecessary administrative burdens and enhance the scope of EHR-related educational opportunities provided to medical students during their clinical training.

Like attending physicians⁸ and residents,^{9, 10} medical students report using EHRs for many hours a day, more in the IM clerkship than in other clerkships.^{11, 12} Surveys conducted by the Clerkship Directors in Internal Medicine (CDIM) asked IM clerkship directors to report on their students’ use of EHRs during IM clerkships in 2016.¹³ Results indicated that 78.5% and 93.6% of students entered admission history and physical examination notes and inpatient progress notes during the IM clerkship, respectively.¹³ Analogous figures reported for the IM sub-internship were 88.2% and 90.3%.¹³ Reported entry of outpatient progress notes was 59.6% for the IM clerkship. Orders to be cosigned were entered by 56.5% of students during the IM clerkship and 85.9% during the IM sub-internship.¹³

These results are helpful in understanding student EHR use from the perspective of IM clerkship directors, but national student surveys regarding their own experiences are limited. Such studies are important because student experiences in varied clinical environments may yield different results for individual students, even within a single school, and clerkship directors’ impressions may differ from students’ perspectives. Using a national sample of medical students, the purpose of this study is to fill this gap by examining student accounts of their experiences using EHRs within IM clerkships and sub-internships. It focuses on a 5-year time period occurring prior to the publication of the new CMS clinical documentation guidelines.

METHOD

Immediately after completing the Step 2 Clinical Knowledge (CK) component of the United States Medical Licensing Examination (USMLE), students are asked to complete a web-based survey about their educational and examination experiences. Different forms of the survey are randomly assigned to students, with each form containing both common and unique questions. From August 2011 to July 2016, one form included questions related to student use of EHRs in IM inpatient and outpatient clerkship settings and in IM sub-internships. One-quarter of Step 2 CK examinees were randomly assigned to the EHR survey from August 2011 to July 2014 and one-half of the examinees were randomly assigned to it from August 2014 to July 2016.

The EHR survey asked medical students to indicate if they personally used an official EHR for review or entry of patient information on a routine basis during their IM clerkship (inpatient and outpatient components) and sub-internship. Separately for each of the three clinical experiences, students who indicated that they used an EHR were then asked whether they entered information into the record on a routine basis that became part of the official EHR. The phrasing of the survey questions made it clear that it was the expectation that student entries would be cosigned by a supervisor and that the student should indicate that they did not enter notes or orders if they only used the record to review patient information. Students who indicated that they did enter notes or orders were then asked what specific types of notes and orders they entered during the inpatient component of the IM clerkship and the IM sub-internship. The specific types of notes and orders included (1) admission history and physical examination notes, (2) progress notes, (3) admission orders, and (4) post-admission orders. The survey did not inquire as to the type of note or order for the outpatient component of the IM clerkship as admitting history and physical examination notes and admitting orders are less common in the ambulatory environment.

Study data included 16,602 medical students with complete, valid responses to the EHR survey questions.¹ These students were selected from an initial dataset that included 83,301 medical students who attended 142 US-based LCME-accredited medical school campuses and planned to graduate medical school between 2012 and 2016.

Dichotomous variables were created to represent each of the six EHR-related activities within and across the IM educational experiences. Derived from responses to the “used” and “entered” questions, another dichotomous variable was computed to represent read-only access; those who had “read-only access” used a record during the clinical experience, but did not enter any information into it. Two additional variables were created from responses to the survey questions about the entry of specific types of notes and orders and represented whether a student entered notes of any type and whether a student entered orders of any type. This process resulted in the following nine variables: (1) used a record, (2) entered into a record, (3) had read-only access, (4) entered admission history and physical examination notes, (5) entered progress notes, (6) entered notes of any type, (7) entered admission orders, (8) entered post-admission orders, and (9) entered orders of any type. The first three variables were created separately for the inpatient and outpatient components of the IM clerkship and for the IM sub-internship; the remaining six variables were created for the IM inpatient clerkship component and IM sub-internship clinical experiences only. This resulted in 21 variables for each medical student (3 activities times 3 clinical experiences, plus 6 activities times 2 clinical experiences). Nine more variables were then created to reflect student participation in the nine EHR-related activities during any applicable IM clinical experience. Thus, the final data set used for analysis had 30 dichotomous variables for each medical student, with a value of 0 if the student did not engage in the activity and a value of 1 if the student did.

Means for each of the 30 variables were calculated by graduation year. Differences in means for graduation years 2016 and 2012 were computed and Chi-Square analyses were used to test whether or not differences were statistically significant. All analyses were performed using IBM SPSS Statistics Version 23.0 (IBM Corp., Armonk, NY). This study was reviewed by the American Institutes for Research Institution Review Board and qualified for exempt status because it involved very minimal or no risk to study subjects.

RESULTS

Of the 83,301 students who took Step 2 CK during the study period, 25% percent of 55,454 (13,864) and 50% of 27,847 (13,924) were randomly assigned to the EHR survey. Thus, a total of 27,788 (13,864 plus 13,924) students received the EHR survey form. Of those students, valid responses were obtained from 16,602 of them. Therefore, the overall survey response rate was 60% (16,602/27,788). The study sample was generally reflective of the full dataset of students who took Step 2 CK during the study period. Students in the study sample had a mean Step 2 CK score of 243 (standard deviation [SD] = 18), and students in the full dataset had a comparable mean score of 240 (SD = 18). Also, all 142 US-based LCME-accredited medical school campuses associated with the 83,301 students who took Step 2 CK during the study period were retained in the final sample of 16,602 students.

Table 1 presents the percentage of medical students in each graduation year that used, had read-only access to, and entered note or order information into EHRs during their IM inpatient clerkship experience, outpatient clerkship experience, sub-internship experience, and across any of these clinical experiences. In each graduation year, almost all students used a record and entered information into a record during at least one of their clinical experiences, with increases over time.

Table 1.

Percentage of Medical Students Who Used, Had Read-Only Access to, and Entered Note or Order Information into an Electronic Health Record (EHR) During Their Internal Medicine Inpatient Clerkship, Outpatient Clerkship, and Sub-internship Experiences, and During Any Clinical Experience by Graduation Year

Internal medicine clinical experience	EHR activity†	Percentage of medical students by graduation year					Percentage change 2016–2012	P value
		2012	2013	2014	2015	2016
Core clerkship inpatient component		N = 1813	N = 2625	N = 2613	N = 4047	N = 5402
	Used a record	84	89	92	94	96	12	< .001
	Read-only access	14	15	13	13	13	− 1	.16
	Entered into record	71	73	79	82	84	13	< .001
Core clerkship outpatient component		N = 1572	N = 2211	N = 2197	N = 3379	N = 4445
	Used a record	77	85	89	92	94	16	< .001
	Read-only access	20	24	21	23	23	3	.01
	Entered into record	57	61	68	69	70	13	< .001
Sub-internship		N = 616	N = 609	N = 552	N = 973	N = 956
	Used a record	86	90	96	97	97	12	< .001
	Read-only access	7	8	6	5	5	− 2	.05
	Entered into record	78	83	89	91	92	14	< .001
Any clinical experience*		N = 616	N = 609	N = 552	N = 973	N = 956
	Used a record	92	96	99	98	99	7	< .001
	Read-only access	6	7	6	4	4	− 2	.04
	Entered into record	86	89	93	94	95	9	< .001

*Includes students who completed a sub-internship and had valid responses for the EHR activity in one or more of their internal medicine clinical experiences

†“Read-only access” and “entered into record” are mutually exclusive subsets of “used a record”

When examined by type of clinical experience, within each year, fewer students used an EHR during their outpatient clerkship experience, compared with their inpatient clerkship and sub-internship experiences, although the magnitude of these differences decreased over time.

With respect to the opportunity to enter information into a record, within each graduation year, the percentage of students with read-only access was highest for students’ outpatient clerkship experiences and lowest for their sub-internship experiences, with comparable increases over time within each experience type. Figure 1 visually depicts the results shown in Table 1. Panel a plots the percentage of students who used and had read-only access to an EHR in any of their clinical experiences by graduation year and panel b splits this information by type of clinical experience and shows that

Fig. 1.

Percentage of students who used and had read-only access to electronic health records by student medical school graduation year. Panel a depicts percentages for any clinical experience, while panel b depicts percentages by inpatient clerkship, outpatient clerkship, and sub-internship experiences.

Table 2 provides the percentage of students who entered different types of notes and orders into EHRs during their inpatient clerkship experience, sub-internship experience, and any clinical experience by graduation year. Within each graduation year, most students entered notes in at least one of their clinical experiences. When examined by clinical experience type, percentages were somewhat higher for sub-internships than for inpatient clerkship experiences. The percentage of students participating in note-related activities in any of their clinical experiences increased over time. For inpatient clerkship experiences, there were greater gains in note-writing activities within sub-internship experiences compared to inpatient clerkship ones.

Table 2.

Percentage of Medical Students Who Entered Various Note and Order Types into an Electronic Health Record (EHR) During Their Internal Medicine Inpatient Clerkship and Sub-internship Experiences and During Any Clinical Experience by Graduation Year

Internal medicine clinical experience	EHR activity	Percentage of medical students by graduation year					Percentage change 2016–2012
		2012	2013	2014	2015	2016
Core clerkship inpatient component		N = 1742	N = 2533	N = 2499	N = 3846	N = 5082
	Entered notes of either type	67	69	75	78	81	14*
	Notes, admission H&P	60	62	69	69	72	12*
	Notes, progress	63	65	71	74	76	13*
	Entered orders of either type	34	32	29	29	29	− 5*
	Orders, admission	26	23	19	18	17	− 10*
	Orders, post-admission	29	26	23	23	24	− 4*
Sub-internship		N = 336	N = 588	N = 510	N = 908	N = 892
	Entered notes of either type	57	77	84	87	88	31*
	Notes, admission H&P	54	74	79	80	82	28*
	Notes, progress	53	73	80	81	83	31*
	Entered orders of either type	44	64	68	70	69	24*
	Orders, admission	40	56	59	56	55	14*
	Orders, post-admission	40	58	61	62	62	22*
Any clinical experience†		N = 336	N = 588	N = 510	N = 908	N = 892
	Entered notes of either type	68	83	88	90	92	24*
	Notes, admission H&P	65	79	84	86	86	21*
	Notes, progress	64	79	85	85	88	24*
	Entered orders of either type	52	68	72	73	72	19*
	Orders, admission	47	60	61	59	57	11*
	Orders, post-admission	47	62	64	65	65	17*

*P < .001

†Includes students who completed a sub-internship and had valid responses for the EHR activity in one or more of their internal medicine clinical experiences

Compared to note-related activities, the percentage of students who entered orders was lower within graduation year for each clinical experience type. Most striking are the low percentages of students entering orders during their inpatient clerkship experiences. When examined by experience type, however, the picture becomes more complicated, with small declines in order-related activities within inpatient clerkship experiences and moderate gains within sub-internship experiences. Figure 2 visually depicts the results shown in Table 2. Panel a shows the percentage of students who entered notes and orders during at least one clinical experience and panel b shows these percentages separately for inpatient clerkship and sub-internship experiences.

Fig. 2.

Percentage of students who entered notes of any type and orders of any type into electronic health records by student medical school graduation year. Panel a depicts percentages for any clinical experience, while panel b depicts percentages by inpatient clerkship and sub-internship experiences.

Figure 3 shows the percentage change for entry of specific types of notes and orders over time for inpatient clerkship experiences, sub-internship experiences, and any clinical experiences. The figure illustrates that the largest percentage increase was for students writing progress notes during their sub-internship experiences. The figure also shows the small decreases in order entry within inpatient clerkships experiences.

Fig. 3.

Percentage changes of students who entered specific types of notes and orders between medical school graduation years 2012 and 2016 for internal medicine inpatient clerkships, sub-internships, and for all clinical experiences.

DISCUSSION

Competence in the underlying knowledge, skills, and abilities required for safe and effective use of EHR systems has been highlighted as an important aspect of contemporary medical education by the medical education community. Two of 13 EPAs developed by the AAMC focus on clinical documentation,^{3, 4} and IM program directors participating in the milestone project have highlighted the importance of competency in EHR use.² Yet, medical student use of EHR systems remains a complex issue due to several interconnected factors.

Issues surrounding patient safety, faculty workload and time constraints, federal guidelines and billing, healthcare system policies, and software compatibility all represent challenges connected to providing medical students with appropriate opportunities to develop clinical documentation skills using EHRs. It may be that certain aspects of EHR use are more important to develop at different times and that a more targeted use of EHRs will help alleviate some of the trials associated with preparing students for residency training and, in turn, later independent practice.

Intended as an exploratory effort to better understand medical student engagement with EHRs, this descriptive study indicates that the overwhelming majority of students report use of an EHR during IM clinical education. Findings also suggest, however, that some students may receive inadequate experience entering information into EHRs, particularly with entering orders in inpatient settings. With respect to trends over time, while student use of and entry of information into EHRs increased over the study period, student entry of orders into EHRs during inpatient clerkship experiences actually decreased during the same period. This is most pronounced for entry of admission orders and could be due to the increased availability of standardized hospital admission orders or order sets, which make it faster and easier for supervising residents and attending physicians to enter orders themselves.

These findings could indicate a lack of preparedness for effectively documenting, synthesizing, and interpreting patient information in the medical record for some students entering residency. Indeed, one recent paper notes that residency program directors view new residents as able to effectively collect a history, perform a physical examination, and document clinical findings, but that they struggle with developing, explaining, and justifying a differential diagnosis in a patient record.¹⁴ Similar work also suggests that students, particularly those from IM residency programs,¹⁵ may enter residency training without essential clinical documentation skills.¹⁶ Demonstrating association between student engagement with EHRs and subsequent performance in residency is beyond the scope of this study. When data are available, future research should empirically examine the relationships between medical student levels of EHR utilization and the quality of that engagement during IM clinical experiences with their subsequent performance as IM residents.

Compared to the student accounts of their EHR use reported in this study, IM clerkship directors have reported much greater student experience with order entry, higher levels of student entry of all hospital-based notes, and lower levels of student entry of outpatient progress notes.¹³ These discrepancies may be accounted for by the different samples examined, or by variation in inpatient and outpatient clerkship director responses coupled with consistencies in student responses. The differences may also highlight incongruences in perceptions between clerkship directors and students, especially with respect to what constitutes note and order entry. Exploring how students and clerkship directors conceptualize student use of EHRs during their IM clinical education could be a fruitful area for future exploration.

The new CMS guidelines allow teaching physicians to verify medical student clinical documentation in EHRs related to evaluation and management code services. It is possible that these recent changes will promote hospital policies that provide students with more opportunities to interface with EHRs. Such shifts in educational practice could better prepare medical students to more effectively use EHRs, a modern skillset needed on the first day of residency training. Another important benefit is that sufficient EHR training for medical students may aid in reducing physician burnout, an increasingly significant issue facing the medical profession today, by improving efficiency and reducing time interfacing with EHR systems.^{17, 18}

Results of this study can be used to help interpret national patterns of student engagement with EHRs before the implementation of health system policy changes that will likely promote greater medical student engagement with EHR during clinical education in IM. The current findings could be compared to future trends in student interactions with EHRs and used to evaluate the impact of the new CMS guidelines and associated EHR educational policies. This may help identify persistent gaps in student interactions with EHR systems and define learning objectives for educational experiences focused on certain aspects of EHR use moving forward.

This study has several limitations. First, not all students provided survey responses for all questions. This response drop-off primarily occurred as students progressed through the survey and was likely due to time constraints associated with survey completion, rather than substantive systematic differences among respondents related to the content of the survey questions. Second, students who completed an IM sub-internship after taking Step 2 CK and students who never completed an IM sub-internship were not included in the sub-internship sample. Findings may have differed if data for these students were available, although it seems unlikely that policies related to EHR use would change over the academic year in which students completed sub-internships. In addition, the sub-internship percentages were calculated based on only the students who completed a sub-internship. Lastly, findings reveal student accounts of their EHR experiences, but do not speak to how well students used EHRs. It is possible that students who entered orders or notes into the medical record did not do so in meaningful ways.

Still, the results of the present study provide a descriptive framework for thinking about student use of EHRs nationally during the time period when EHR systems were introduced and federal billing guidelines indirectly presented unique challenges for creating appropriate clinical documentation learning opportunities with EHRs for medical students. Given the complex issues surrounding medical student use of EHRs and the need for medical students to learn and practice how best to use EHRs for high-quality patient care,^{19, 20} the new federal guidelines provide some promise for the ability to provide students with additional, and perhaps more defined and targeted, EHR educational experiences.

Compliance with Ethical Standards

This study was reviewed by the American Institutes for Research Institution Review Board and qualified for exempt status because it involved very minimal or no risk to study subjects.

Conflict of Interest

The authors declare that they do not have a conflict of interest.