Skip to main content
HHS Author Manuscripts logoLink to HHS Author Manuscripts
. Author manuscript; available in PMC: 2017 Jul 31.
Published in final edited form as: Int J Med Inform. 2016 Mar 2;90:1–11. doi: 10.1016/j.ijmedinf.2016.02.011

A comparative observational study of inpatient clinical note-entry and reading/retrieval styles adopted by physicians

Rubina F Rizvi a, Kathleen A Harder a,b, Gretchen M Hultman a, Terrence J Adam a,c, Michael Kim d, Serguei VS Pakhomov a,c, Genevieve B Melton a,e
PMCID: PMC5534410  NIHMSID: NIHMS879121  PMID: 27103191

Abstract

Objective

The objective of this study is to understand physicians’ usage of inpatient notes by (i) ascertaining different clinical note-entry and reading/retrieval styles in two different and widely used Electronic Health Record (EHR) systems, (ii) extrapolating potential factors leading to adoption of various note-entry and reading/retrieval styles and (iii) determining the amount of time to task associated with documenting different types of clinical notes.

Methods

In order to answer “what” and “why” questions on physicians’ adoption of certain note-entry and reading/retrieval styles, an ethnographic study entailing Internal Medicine residents, with a mixed data analysis approach was performed. Participants were observed interacting with two different EHR systems in inpatient settings. Data was collected around the use and creation of History and Physical (H&P) notes, progress notes and discharge summaries.

Results

The highest variability in template styles was observed with progress notes and the least variability was within discharge summaries, while note-writing styles were most consistent for H&P notes. The first section to be read in a H&P and progress note were the Chief Complaint and Assessment & Plan sections, respectively. The greatest note retrieval variability, with respect to the order of how note sections were reviewed, was observed with H&P and progress notes. Physician preference for adopting a certain reading/retrieval order appeared to be a function of what best fits their workflow while fulfilling the stimulus demands. The time spent entering H&P, discharge summaries and progress notes were similar in both EHRs.

Conclusion

This research study unveils existing variability in clinical documentation processes and provides us with important information that could help in designing a next generation EHR Graphical User Interface (GUI) that is more congruent with physicians’ mental models, task performance needs, and workflow requirements.

1. Introduction

Clinical notes are an essential communication tool for summarization, synthesis and decision making for patient care. In addition to direct patient care, notes are valuable for other functions such as medical education, research, billing, quality-assessment and medico-legal inquiries/compensations (13). The importance of having high quality clinical notes was recognized in the 1960s by Dr. Lawrence Weed as part of the Problem-Orientated Medical Record (POMR) framework, which was key in the establishment of the SOAP (Subjective, Objective and Assessment & Plan (A/P)) note format and documentation of patient problems by organ systems (4). Currently used common clinical note types include History and Physical (H&P) notes, progress notes, consult notes, operative notes and discharge summaries.

Clinical notes documentation is considered to be a core aspect of a patient’s encounter and fundamental for health care delivery. While EHRs have enhanced direct access to patient data (5), clinicians continue to experience significant barriers in EHR usage, such as inefficiencies with structured data entry and retrieval, as we all as difficulty using and creating computerized patient documentation (1, 6). Free text entry in clinical documents is typically considered ideal for communication between providers and for presenting complex sets of facts, but can be laborious and time consuming to create in an electronic interface. On the other hand, structured data entry, which is typically more difficult to read and synthesize, enables the reuse of data for downstream applications such as quality improvement and research (2, 7, 8). While clinicians appreciate the flexibility and efficiency of narrative free-text entry with the use of “copy and paste” or “copy forward” functions, they are challenged by long and verbose clinical notes that can be laborious to review or synthesize and could potentially contain erroneous information not appreciated during the documentation process.

There is growing interest in understanding the different aspects of clinical documentation processes such as their integration with workflow (8, 9), structured versus free-text entry (2) and usability studies of EHR systems pertaining to creation and use of clinical documents (10). In recognition of the importance of clinical documentation, recording electronic notes in patient charts is included as one of the menu objectives in Stage-2 of the Meaningful Use Program (11). Also, the lack of standardization in EHR clinical documentation and display styles provides interface designers with an area of opportunity to re-design EHR systems (1215).

Several researchers have previously examined tools and measurements to understand clinical documentation processes and potential areas of opportunity to improve clinical note quality. This includes development of validated instruments for assessing inpatient clinical documentation quality (16, 17), techniques for generating clinical notes with clinically relevant information that is reusable and readable (1, 16, 18, 19), and use of eye tracking to discover how the visual attention of physicians is distributed while reading electronic notes (20).

In order to improve our understanding of empiric behaviors of physicians around clinical documentation use and generation, the goal of this study was to discover different styles of physician inpatient note-entry as well as reading/retrieval styles in two different EHR systems in two observed settings and to extrapolate potential factors associated with different behaviors/styles of system use. In addition, this study aims to ascertain and compare the various time to complete key tasks of clinical note documentation.

2. Methods

2.1. General Description and Setting

A participant observation ethnographic field study approach, supplemented with post-observation online surveys, was employed to collect data about the routine, day-to-day activities of participants/users in a naturalistic setting (21). While this approach does not offer a controlled experimental setting, the method was chosen since it provides a rich, realistic, and holistic view of the users’ routine by immersing in their environment. This immersion helps in gathering additional detailed information, which users can sometimes inadvertently fail to communicate overtly with other more interactive or controlled (e.g., laboratory-based) methodological approaches. Various similar observational study methodologies have been widely used in scientific research, including healthcare (2226).

Approval for this study was obtained from the University of Minnesota Institutional Review Board and from the Veterans Affairs Research and Development Committee. Internal Medicine resident physicians were observed interacting with two different EHR systems, Epic and Veterans Affairs Computerized Patient Record System (CPRS), in naturalistic inpatient environments, at the University of Minnesota Medical Center (UMMC) and Minneapolis Veterans Affairs Health Care System (VAHCS) respectively, at various times and days including on-call and off-call days. Since residents spend most of their time interacting with EHRs in workrooms, particularly performing clinical note documentation, the majority of observations were made there.

2.2. Study Sample

Residents (2nd through 4th years), enrolled in Internal Medicine Categorical or Internal Medicine Combined programs, were recruited for the study. Interns, medical students, advanced practice providers and other clinical staff were excluded. Participants were recruited after obtaining their verbal assent. Detailed characteristics of research participants are summarized in Table-1.

Table-1.

Summary characteristics of research participants

Characteristics UMMC *H1 VAHCS *H2
Female (%) 4 (66.6%) 3 (50%)
Male (%) 2 (33.3%) 3 (50%)
Mean age 31 (±3.6) 29.5 (±1.6)
Mean years in training 2.8 (±0.4) 3 (±0.6)
*

UMMC-Hospital (H1); VAHCS-Hospital (H2)

2.3. Data Collection

Qualitative and quantitative clinical documentation process data was collected focusing on clinical note data entry and reading/retrieval tasks. Direct observation was used to collect data regarding user behaviors, their workflow and EHR usage centering on different uses and tasks associated with clinical documentation.

Residents follow different call and day schedules at UMMC and VAHCS (Fig. 1). To account for this variability, each participant was observed over different call routines and times of the day. The majority of field notes were taken while residents were doing clinical documentation in their workrooms.

Figure 1. Typical call and day schedule of residents at UMMC-Hospital (H1) and VAHCS-Hospital (H2).

Figure 1

The figure shows approximate times, other than for morning report and noon conference which have set times. Nightfloat residents or a resident on sub-specialty month do not follow the above schedule

The total observation time was greater than 110 hours. Details about observation times are provided in Table-2.

Table 2.

Observation schedule and hours

Participants UMMC-H1 VAHCS-H2
On call hours Non-call hours Total On call hours Non-call hours Total
P1 7 4 11 7 6 13
P2 6 4 10 5 4 9
P3 7 3 10 5 5 10
P4 6 3 9 4 3 7
P5 6 4 10 6 4 10
P6 2 2 4 5 4 9
Mean hours 5.6 (±1.8) 3.3 (±0.8) 9 (±2.5) 5.3 (±1.0) 4.3 (±1.0) 9.6 (±1.9)

Field notes were taken on an electronic tablet through a time-stamped application called “Timestamped Field Notes Application version 3.0” (27). The data was later transferred to an encrypted device and stored on a secure PHI-compliant server. We also collected hard copies of note templates (H&P, progress note and discharge summaries), consumed by each participant, for post-hoc data analysis purposes. At the end of observations, an electronic semi-structured survey regarding user perceptions about EHR clinical documentation practices was administered. The survey contained multiple choice and open-ended questions on note styles, note documentation, workload and electronic interface usage. Each study participant filled out the survey once with a 100% response rate. The purpose of conducting the surveys was to collect useful benchmark data on physicians’ workflow, their preferences and perceptions about clinical documentation processes. Participants were provided with a nominal gift certificate ($50) for their participation.

2.4. Data Analysis

Ethnographic Content Analysis (ECA) was performed with integrated qualitative-quantitative research designs (28) using “NVivo version 10.1.3” (29). Observations performed on multiple days and times were examined iteratively in order to generate broader generalizations.

Observations and data parsing were primarily done by RR, a physician and health informatician and by GH, a health informatician and clinical research study coordinator. Each observation was used as a unit of analysis. Since this study is to be considered process driven (i.e., categories defined empirically by process as opposed to predefined), the data collection process was performed without any prior conceptual framework. After repeated reviewing of field notes, four higher-level themes were derived, each representing a respective parent node having several child nodes. The data was then coded at a more granular level i.e., note type (e.g., H&P, progress note, discharge summary, consult note), task performed (e.g., note-entry; notes reading/retrieval), style adopted (e.g., style 1, style 2, style 3 etc.) and time to task. The data and themes were validated by a set of senior clinicians (GMM, TA) and a user interface design expert (KH), arriving at agreement with the observers’ determination of nodal structure and general findings. Integration of different types and sources of data was also obtained for triangulation, thus increasing the validity to the overall findings. Triangulation was achieved by employing mixed method research design and collecting data in several different ways. Objective data was collected by observing participants in a naturalistic setting and taking down field notes. Subjective data was acquired directly from the participants using post-observation online surveys. Both objective and subjective data were later analyzed and compared.

Inter-rater reliability was established by calculating percentage agreement between the two coders from a subset of data representing 16% of the field notes, with mean percentage agreement of 90% and kappa value of 0.73. Any inconsistencies were addressed via review and consensus.

3. Results

3.1. Note-entry

3.1.1. Note Template Styles

The template is defined as a pre-structured documentation tool, providing a basic format that could be used repeatedly and are often employed for generating clinical documents (30). Note templates were ubiquitously used by physician residents in our study while performing clinical note-entry tasks. For H&P and progress notes, the templates were either created by the user or shared from other users, however, for discharge summaries, a certain level of template standardization was observed with small areas of customization.

Overall, five H&P template styles were seen, with common sections of: Chief Complaint (CC); History of Present Illness (HPI); Past Medical History (PMH); Past Surgical History (PSH); Family History (FM); Social History (SH); Allergies; Medications; Review of Systems (ROS); Physical Examination (PE); Laboratories; Imaging and Assessment & Plan (A/P). The most commonly observed styles were style 1 and style 2 (each style being preferred by 4/12 participants (33%) and used together in (22/32 (69%)) instances (Fig. 2). Most H&P templates had Chief Complaint located at the top of the note (29/32 times (91%)), with Assessment & Plan occasionally located at the beginning of a note (3/32 times (9%)) and with some order variation and preferences for other sections.

Figure 2. Note template styles for H&P, progress note and discharge summary as adopted by physicians.

Figure 2

* H1:UMMC-University of Minnesota Medical Center; H2:VAHCS-Veterans Affairs Health Care System; Sn: Number of participants; T=Total participants; Nn: Number of notes; CC: Chief Complaint; HPI: History of Present Illness; PMH: Past Medical History; PSH: Past Surgical History; ROS: Review of Symptoms; PE; Physical Exam; SH: Social History; FM; Family History; A/P: Assessment & Plan; S: Subjective; DD: Discharge Diagnoses; HCP: Hospital Course by Problem

Similarly, for progress notes, six styles were used including the common sections of Subjective (S); Objective (O) (e.g., Physical Examination, Laboratories/Imaging; Medications) and Assessment & Plan. Interval History, which is another name for the Subjective section, was also a common section title. The three different components of the Objective section also had several different order preferences. The most commonly used progress note templates were style 1 (4/12 participants (33%), used 19/73 times (26%)); style 2 (3/12 participants (25%), used 19/73 times (26%)) and style 3 (2/12 participants (17%), used 14/73 times (19%)) (Fig. 2). In all cases, progress note templates started either with the Subjective or Interval History section (54/73 times (74%)), or less commonly from Assessment & Plan (19/73 times (26%)).

For discharge summaries, there were five template styles with the following common sections: Discharge Diagnoses (DD); Pertinent Procedures and Imaging; Physical Examination; Hospital Course by Problem (HCP) and Discharge Instructions. Additional and less consistently used sections were Consults, Medications and History of Present Illness. For discharge summary templates, styles 1 and 2 were most commonly used (5/12 participants (42%) and 3/12 participants (25%); 21/48 (44%) and 9/48 (19%) times respectively) (Fig. 2). It was observed in all instances that the discharge summary templates had Discharge Diagnoses at the beginning with some order customization of other sections.

3.1.2. Note-Writing Styles

For writing notes, physicians preferred to utilize a range of styles demonstrating both within and between participant variability in writing styles for different notes types. H&P note-writing styles corresponded directly to the five H&P template styles. The most preferred ordering was to use style 1 (5/12 participants (42%), used 9/32 times (28%)); style 2 (4/12 participants (33%) used, 11/32 times (34%)); and style 3 (4/12 participants (33%), used 6/32 times (19%)) respectively (Fig. 3). The majority of users started writing notes with the Chief Complaint section (23/32 (72%)) and the minority of users starting with the Assessment & Plan section (9/32 (28%)). After completing the Assessment & Plan section, participants were observed to follow any of the other four writing patterns to complete the rest of the H&P note. The tendency for users to stick with a particular style each time was rather consistent with very minimal crossover.

Figure 3. Note-writing styles for H&P, progress note and discharge summary as adopted by physicians.

Figure 3

*H1-UMMC: University of Minnesota Medical Center; H2-VAHCS: Veterans Affairs Health Care System; Sn: Number of participants; T=Total participants; Nn: Number of notes; CC: Chief Complaint; HPI: History of Present Illness; PMH: Past Medical History; PSH: Past Surgical History; ROS: Review of Symptoms; PE: Physical Exam; SH: Social History; FM; Family History; A/P: Assessment & Plan; S: Subjective; DD: Discharge Diagnoses; HCP: Hospital Course by Problem. Dotted lines represent various patterns adopted

For progress notes, six common note-writing styles, corresponding roughly to the template styles, were employed. The preferred order for creating a progress note was style 1 (10/12 participants (83%), observed 40/73 instances (55%)) (Fig. 3). Within progress notes, most users started composing the note from either Assessment & Plan section (40/73 (55%) or Subjective/Interval History sections (33/73 (45%) times), followed by a variety of completion patterns.

Compared to the five template styles for discharge summaries, six common discharge summary note-writing styles were used, including one additional note-writing style. The most preferred style was style 1 (9/12 participants (75%), with 22/38 instances (58%)) (Fig. 3). All participants started to compose discharge summaries from either Hospital Course by Problem section (22/38 times (59%)) or the Discharge Diagnoses section (16/38 times (42%)). Those who preferred starting from Hospital Course by Problem completed the note following any of the other 5 patterns.

3.2. Notes Reading and Retrieval Styles

A number of reading styles were observed for each note type. The pattern was often related to the stimulus initiating the task. Physician preference for adopting a certain style appeared to be a function of what best fits in their workflow. Both within and between participant variability was observed in note reading/retrieval styles for various notes types and in response to different stimuli.

When reading information from H&P notes in both systems, providers’ preferred to start reviewing either from the Chief Complaint section (23/31 times (74%)) or from the Assessment & Plan section (8/31 times (26%)). For progress notes, the commonly observed preference was to start reading notes from the Assessment & Plan section (47/53 times (89%)), and less often from the Subjective or Interval History section (6/53 times (11%)). On the other hand, while reading a discharge summary, users started with the Hospital Course by Problem section (24/32 times (75%)) or Discharge Diagnoses section (8/32 times (25%)) (Fig. 4). Apart from these three main note types, consult notes were mostly read starting from the Recommendations section, which is analogous to the Assessment & Plan section of an H&P note, where providers enter their assessment and relevant plan suggestions for managing the patient.

Figure 4. Note retrieval/reading styles for H&P, progress note and discharge summary as adopted by physicians.

Figure 4

*H1-UMMC: University of Minnesota Medical Center; H2-VAHCS: Veterans Affairs Health Care System; Sn: Number of participants; T=Total participants; In: Number of Instances; CC: Chief Complaint; HPI: History of Present Illness; PMH: Past Medical History; PSH: Past Surgical History; ROS: Review of Symptoms; PE; Physical Exam; SH: Social History; FM; Family History; A/P: Assessment & Plan; S: Subjective; DD: Discharge Diagnoses; HCP: Hospital Course by Problem

Overall, the chronological order of reading a particular note after studying the initial section, often appeared dependent upon the type of the retrieving stimulus. For example, when looking for information about the vitals or laboratories and medications, providers almost exclusively preferred to read and synthesize this information directly from the primary data entry section of the chart containing the results and ancillary studies as opposed to the obtaining the information from re-entered data in the note. One exception to this observation was noticed when providers were reading a discharge summary or a H&P note from previous admissions, where it was observed that they tended to skim through all the entered data.

A summary of the notes template, writing and reading/retrieval styles are depicted in Fig. 5.

Figure 5. Summary of preferred note-entry and retrieval styles as adopted by physicians.

Figure 5

*CC: Chief Complaint; HPI: History of Present Illness, PE: Physical Exam; PMH: Past Medical History;

SH: Social History; FH: Family History; PSH: Past Surgical History; ROS: Review of Symptoms; A/P: Assessment & Plan; S: Subjective: O: Objective; A/P: Assessment & Plan; DD: Discharge Diagnoses; HCP: Hospital Course by Problem

3.3. Comparison of Observed and Self-Reported data

Observations on note templates, writing and reading/retrieval style were later compared with self-reported survey data. A sample of survey questions is depicted in Table-3.

Table 3.

Sample questions and response options used in survey

Survey Questions Response options
1 What style do your prefer while entering an H&P note? -Start form subjective data entry
-Start from Assessment & Plan
-Other
2 How much time do you think you spend on average writing an H&P? -<10
-10-20
-21-30
-31-40
->40
3 Do you use templates for entering H&P? -Yes
-No
-Other
4 What style do your prefer while reading an H&P? -Start from subjective data
-Start from Assessment & Plan
-Other
5 What are the major limitations of EHR’s Graphical User Interface in terms clinical note-entry tasks? How do you think they can be rectified? Free text
6 What are the major strengths of EHR’s Graphical User Interface in terms clinical note-entry tasks? How do you think they can be rectified? Free text

For note-writing and reading/retrieval styles, considerable discrepancies were discovered between physician self-report and their observed actions (Table-4). These observed variances in reading/retrieval styles could be explained by types of stimuli triggering the tasks. We also observed that physicians had a tendency to utilize the same template every time they entered a particular type of a note, having consistent results for both self-reported and observed data.

Table 4.

Comparison of participants self-report versus observed template style, writing, and reading/retrieval styles

Template style Writing Styles Note Reading/Retrieval Styles
  H&P *CC *A/P *CC *A/P *No particular style *CC *A/P **No particular style
Self-Report 11 1 10 2 0 6 6 0
Observed 11 1 7 1 4 6 1 5
  Progress notes *SOAP *APSO *S *A/P **No particular style *S *A/P *No particular style
Self-report 9 3 8 4 0 4 8 0
Observed 9 3 2 6 4 0 5 7
Discharge summaries *DD *HCP *DD *HCP **No particular style *DD *HCP *No particular style
Self-report 12 0 6 6 0 6 6 0
Observed 12 0 3 2 7 0 3 9
*

CC: Chief Complaint; A/P: Assessment & Plan; S: Subjective; SOAP: Subjective, Objective, Assessment & Plan; APSO: Assessment & Plan, Subjective, Objective; DD: Discharge Diagnoses; HCP: Hospital Course by Problem

Based upon our observations, overall, similar amounts of time were spent on each type of note in both Epic and CPRS, with the H&P taking the most time (mean 39 and 42 minutes, respectively), and progress notes taking the least time (mean 11 and 12 minutes respectively) (Fig. 6 and Table-5).

Figure. 6.

Figure. 6

Observed time in minutes for entering different types of notes in two EHRs subjective data

Table 5.

Time to complete a note in Epic and CPRS, a comparison between objective and subjective data

H&P Progress notes Discharge summary
Observed (Mean, Median, Range) Self-report (Frequently selected range) Observed (Mean, Median, Range) Self-report (Frequently selected range) Observed (Mean, Median, Range) Self-report (Frequently selected range)
Epic
P1 30,35,43 (36,35,30–43) 31–40 8,10,12,30 (15,11,8–30) 31–40 25,40 (33,33,25–40) 21–30
P2 30,35 (33,33,33–35) 10–20 5,5,6,7,15 (8,6,5–15) 10–20 30,30 (30,30) 21–30
P3 40,45,55 (47,45, 40–55) 31–40 4,4,12 (7,4,4–12) <10 20,36,55 (37,36,20–55) 21–30
P4 40,50 (45, 45, 40–50) 31–40 7,13,14,15,15 (13,14,7–15) 10–20 25,26,30 (27,26,25–30) 31–40
P5 45 (45,45) 21–30 7,10,30 (16,10,7–30) 10–20 35 (35,35) 21–30
P6 30,30,35 (32, 30,30–35) 21–30 5,6,8,8,11 (8,8,5–11) 10–20 9,20,34 (21,20,9–34) 10–20
(39,38,30–55)
SD-8
(31–40) (11,8,5–30)
SD-6.8
(10–20) (30,30,9–55)
SD-10.8
(21–30)
CPRS
P1 35,40 (38,38,35–40) 10–20 5,10,10,12 (9,10,5–12) <10 15,20,25,25 (21,23,15–25) 10–20
P2 50 (50,50) 21–30 14,15 (14.5,14.5) 10–20 17,25 (21,21,17–25) 21–30
P3 30,30 (30,30) 31–40 6,15,30 (17,15,6–30) 10–20 25, 27 (26,26,25–27) 31–40
P4 55 (55,55) 10–20 8,8 (8,8) <10 28,33 (31,31,28–33) 10–20
P5 32,45,50 (42,45,32–50) 31–40 10,12,15 (12,12,10–15) 10–20 25,33,40 (33,33,25–40) 31–40
P6 40,50 (45,45,40–50) 21–30 10,12 (11,11,10–12) 10–20 17,25,25,29 (24,25,17–29) 10–20
(42,40,30–55)
SD=9
Equivocal (12,115–30)
SD-5.7
(10–20) (26,25,15–40)
SD-6.3
(10–20)

4. Discussion

Our study demonstrates that physicians have several preferences for performing clinical note-entry and reading/retrieval tasks, which vary with note types, tasks and by stimuli. Progress note template usage among residents showed the greatest variability, while the discharge summary templates had the least (Fig. 2). On the other hand, note-writing styles were most consistent for H&P notes (Fig. 3). The first section to be read in the most consistent fashion was observed with H&P and progress notes (i.e., starting from Chief Complaint and Assessment & Plan sections respectively). Conversely, the greatest note retrieval variability (i.e., the chronological orders of how note sections were reviewed), was observed with H&P and progress notes (Fig. 4). Various note reading/retrieval styles appeared to result from the type of stimulus mandating specific information to be extracted from a particular note type and as well as a result of personal preferences leading to adoption of a few dominant styles.

The observed variability in note-entry and reading/retrieval styles, as adopted by physicians, could be explained from three different perspectives. First, the relevance and priority of data being entered or retrieved focused on accomplishing the task effectively and efficiently. For example, a physician initially wants to know “what” brought the patient to emergency department and then “why.” Subsequently, the physician then works towards establishing and documenting reasons for those “what” and “why” questions, proceeding later to “how” to solve those problems and finally to documenting the “summary of the whole encounter.” Secondly, the type of stimulus also dictates the chronological order of how a note might be reviewed (e.g., a progress note from the previous day is most often opened for writing the subsequent progress note). Finally, while we do not have direct evidence, other factors such as earlier training, colleagues’ styles, and the physician’s personality could all potentially contribute towards adopting different styles/patterns. The exact contribution of these factors remains unclear and could be an interesting area of further study.

Our findings also demonstrate differences between actual practice and perceived usage of note template organization and styles pertaining to the clinical documentation process. Among users of both EHR systems, the observed and perceived times on entering progress notes and discharge summaries were fairly similar with some inconsistencies in time data on H&P notes. Our observations were externally validated and consistent across two different established EHR systems and at two different inpatient sites with considerable inter-participant consistency.

This research study identifies variations, which exist in note-writing and reading/retrieval styles resulting from varied physicians’ preferences and workflow demands. Understanding the various styles/time to tasks consumed by users, which are centered on their preferences and the workflow demands, could be used to address the disparities existing in current EHR system design. For example, improved consistency in clinical notes documentation could be established with standardization of note template structure. A more modular, streamlined, task-oriented style, congruent with users’ preferences and their mental models, would be beneficial, particularly for the notes showing greatest variability i.e., progress notes and H&P. Designing a GUI, defined as “a program that allows a person to work easily with a computer by using a mouse to point to small pictures and other elements on the screen” (31), for clinical documentation, should reflect the users’ mental model which could potentially lead to more uniformity in note-writing styles. Similarly, designing an interface that provides users with task and/or stimuli specific presentation views, could potentially facilitate more efficient and effective data comprehension and retrieval from notes. Furthermore, while not examined directly here, clinical notes usage by physicians could also be reinforced or improved through refinement of automated methods to detect and visualize new information (32).

Our analysis of template styles revealed a number of predominant note organization patterns, which was somewhat but not fully congruent with the styles used for writing or reading/retrieving. For instance, an H&P most often had Chief Complaint or less often Assessment & Plan first in the note, which was the same as the writing style and initial reading style. However, the chronological order of how additional H&P sections were reviewed, was dependent upon the type of stimulus. For example, when writing a discharge summary, physicians tended to pull the H&P note from current admission, utilizing information from the Subjective sections e.g., Chief Complaint and History of Present Illness. We also observed that H&Ps were most commonly opened for the purposes of writing another H&P (Fig. 4).

Users reviewed progress notes most commonly starting from the Assessment & Plan section and less often from Subjective. This was the most commonly seen style regardless of whether they had read the H&P earlier or not. This observation is congruent with another study where an eye tracking methodology was used to discover how the visual attention of physicians is distributed while reading electronic progress notes (20). In terms of fixations and glances, physicians directed the most attention to the Assessment & Plan section with very little attentiveness given to other parts of the note (20). Moreover, similar to H&P notes, the chronological order for reviewing various sections within a progress note appeared to be heavily dependent upon the stimulus. For example, when paged regarding an alteration in a patient’s condition, a covering physician who might be less familiar with a given patient, tended to look first at the Subjective from that day to contextualize the patient’s condition and status. On the other hand, when writing a progress note, physicians would often read the Assessment & Plan section of a note followed by the Physical Examination section. In our observation, the highest number of progress notes were pulled up for the purposes of writing a subsequent progress note. These observations were similar between two locations.

On the other hand, while reading a discharge summary, the tendency was to read the Hospital Course by Problems first or in some instances Discharge Diagnoses. Similar to H&P and progress notes, the type of stimulus appeared to help dictate the physician reading styles e.g., when writing an H&P note for readmission, physicians preferred to review the Hospital Course by Problem from the previous discharge summary followed by Discharge Diagnoses and other available data. Discharge summaries were often pulled up to write an H&P or to write a new note on a patient getting readmitted to the hospital or a patient getting transferred to another unit.

In general, we also observed that when the goal was to retrieve data for vitals, labs and medications, physicians tended to gather data directly from primary data entry points rather than from electronic notes. An exception to this behavior was observed when a providers were reading discharge summaries or H&Ps from previous admissions where it was observed that they tended to skim through all the entered data.

We also observed that for H&Ps and progress notes, the templates were either created by the user or shared from other users, depending upon their preferences. On the contrary, discharge summaries contained a significant level of template standardization with small areas of customization. The tendency for users to stick with a particular template style was rather consistent with very minimal crossover between styles. Physicians had a tendency to utilize the same template every time they entered a particular type of a note, which was consistent in both self-reported and observed data, thus strengthening our inference about physicians sticking to a particular style. All the above observations were comparable between two sites, during different times of the day and whether the participants were being observed on on-call and/or off-call days.

Additionally, we also observed some discrepancy between physicians’ self-reported behavior as gathered from electronic surveys and our observations. The inconsistencies noted are mainly for writing and reading/retrieval styles for H&Ps, progress notes and discharge summaries. These observed conflicts between some of the perceived and observed reading/retrieval styles could be explained by the difference in the type of stimulus instigating a specific task.

There are several limitations associated with this study. Qualitative research is highly dependent on a researcher’s skill and more easily influenced by the researcher’s personal biases. We have tried to address this limitation to enhance trustworthiness in the study through content validation involving other co-authors (MDs, health informaticians and usability experts) and assessing inter-rater reliability between coders. Any inconsistencies were addressed via review and consensus. Another limitation is the small sample size posed by recruitment of only Internal Medicine residents in their second, third and fourth years. Because of our small sample size, our findings are limited by a lack of significant statistical analysis. In addition, this study presents more of a quantitative representation of qualitative data and provides readers with a broader view of the observed dissimilarities between the objective and subjective data. Further exploration is needed to make comment on statistically significant differences between observed and self-reported time. Additionally, we did not examine note retrieval styles at a macro level, including navigation between different types of notes along with what information was contained in each type of note. These findings should be corroborated with a larger set of physicians and possibly with providers working in non-academic settings or with established clinicians working in a wider variety of hospital types. Also, our time data for notes should be considered as approximate times. Use of a stopwatch, asking MDs to self-report time required for tasks, or directly extracting time stamped data from EHRs, are some other approaches that could have resulted in more accurate data collection. In addition, within surveys, we provided participants with preset ranges of time needed to complete a particular note rather than keeping the response open-ended, which could have provided us with more accurate time data.

We also anticipate that the ambulatory setting could have different findings, stemming in part from significantly different workflows. Future studies will also aim to assess usability features offered by each system in detail. In addition, this paper does not provide the relative amount of time for each section in either reading/retrieving or creating a note. More detailed time motion studies are required to further elaborate on time data and utilize this knowledge in creation of new, improved GUI. Ultimately, prospective studies linking note styles and different note GUIs within EHRs to their impact on associated care, patient outcomes and potentially clinician comprehension of the patient’s clinical status are needed.

5. Conclusion

In summary, different but congruent styles were utilized by physicians while performing data entry and reading/retrieval tasks for different types of inpatient clinical notes in two different EHR systems. The differences in note-entry styles and reading/retrieval styles appeared to be primarily based on physician preferences, note type and the stimulus type initiating a task. There were inconsistencies seen in physician self-reported and observed note-writing and reading/retrieval styles. Additionally, the times to write the full H&P, progress note and discharge summary were comparable in both systems with H&P taking the most time and progress notes taking the least time. This study provides EHR interface designers with valuable information to help define requirements and potential designs for improved EHR system interfaces for clinical notes that could be more aligned with the users’ mental model and task performance for clinical note documentation.

Acknowledgments

We would like to thank the following people for their assistance: Drs. Jessica Voight and Kate Gillen (Chief residents) along with all residents for their participation and valuable feedback, as well as the assistance of Fairview Health Services and the Minneapolis Veterans Affairs Health Care System. We would also like to thank Elizabeth Lindemann in helping us with the proofreading of the manuscript.

This study was supported by the Agency for Healthcare Research and Quality, Project # R01HS022085 (GM).

Footnotes

Author contributions

Rubina F. Rizvi: Main author who designed this study and was the key person involved in data collection, analysis and interpretation process. She drafted the full manuscript of paper and did the critical revisions in the manuscript in accordance with other co-authors feedbacks.

Kathleen A. Harder: Involved in most key decisions for the study and the paper. Her involvement was crucial to the design of the study and the choice of methodologies.

Gretchen M. Hultman: Supported the work by active engagement in all project meetings and determining the inter-rater reliability through coding of a subset of the qualitative data. She also supported the survey validation for the ethnographic study used for the paper.

Terrence J. Adam: Main reviewer of the paper aside from Dr. Melton. He was also responsible for arranging logistics for the ethnographic study at the Veterans Affairs Health Care System and as well as for the recruitments of the participants for the study.

Michael Kim MD: Supported the logistics for the ethnographic study as well as the participant recruitment at the University of Minnesota Medical Center.

Serguei V. Pakhomov: Contributed to the initial design and selection of methodologies for the study.

Genevieve B. Melton MD, PhD: PI for the project “Discovery and Visualization of New Information from Clinical Reports in the EHR.” Dr. Melton guided and supervised every step of this work. She is the main mentor and advisor for the paper.

Conflicts of Interest

The authors declare that they have no conflicts of interest in the research.

Human Subjects Protection

Residents interacting with two different EHR systems (Epic and CPRS) were investigated at the University of Minnesota Medical Center (UMMC) and Minneapolis Veterans Affairs Health Care System (VAHCS) following approvals and in compliance with Institutional Review Board (IRB) #1308E41121 and Research and Development committee (R&D) #R140720X standards.

The content is solely the responsibility of the authors and does not represent the official views of the Agency for Healthcare Research and Quality.

References

  • 1.Embi PJ, Yackel TR, Logan JR, Bowen JL, Cooney TG, Gorman PN. Impacts of computerized physician documentation in a teaching hospital: Perceptions of faculty and resident physicians. Journal of the American Medical Informatics Association [Internet] 2004;11(4):300–9. doi: 10.1197/jamia.M1525. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Rosenbloom ST, Denny JC, Xu H, Lorenzi N, Stead WW, Johnson KB. Data from clinical notes: A perspective on the tension between structure and flexible documentation. J Am Med Inform Assoc [Internet] 2011 Mar-Apr;18(2):181–6. doi: 10.1136/jamia.2010.007237. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Solomon DH, Schaffer JL, Katz JN, Horsky J, Burdick E, Nadler E, Bates DW. Can history and physical examination be used as markers of quality?: An analysis of the initial visit note in musculoskeletal care. Med Care [Internet] 2000;38(4):383–91. doi: 10.1097/00005650-200004000-00005. [DOI] [PubMed] [Google Scholar]
  • 4.Weed LL. Medical records, medical education, and patient care: the problem-oriented record as a basic tool. [Internet] Press of Case Western Reserve University; 1970. [Google Scholar]
  • 5.Smith K, Smith V, Krugman M, Oman K. Evaluating the impact of computerized clinical documentation. Comput Inform Nurs [Internet] 2005 May-Jun;23(3):132–8. doi: 10.1097/00024665-200505000-00008. [DOI] [PubMed] [Google Scholar]
  • 6.Friedberg MW, Chen PG, Van Busum KR, Aunon F, Pham C, Caloyeras J, Mattke S, Pitchforth E, Quigley DD, Brook RH. Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy [Internet] Rand Corporation; 2013. [PMC free article] [PubMed] [Google Scholar]
  • 7.Ash JS, Bates DW. Factors and forces affecting EHR system adoption: Report of a 2004 ACMI discussion. J Am Med Inform Assoc [Internet] 2005 Jan-Feb;12(1):8–12. doi: 10.1197/jamia.M1684. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Rosenbloom ST, Miller RA, Johnson KB, Elkin PL, Brown SH. Interface terminologies: Facilitating direct entry of clinical data into electronic health record systems. J Am Med Inform Assoc [Internet] 2006 May-Jun;13(3):277–88. doi: 10.1197/jamia.M1957. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Johnson KB, Ravich WJ, Cowan JA., Jr Brainstorming about next-generation computer-based documentation: An AMIA clinical working group survey. Int J Med Inf [Internet] 2004;73(9):665–74. doi: 10.1016/j.ijmedinf.2004.05.009. [DOI] [PubMed] [Google Scholar]
  • 10.Rose AF, Schnipper JL, Park ER, Poon EG, Li Q, Middleton B. Using qualitative studies to improve the usability of an EMR. J Biomed Inform [Internet] 2005;38(1):51–60. doi: 10.1016/j.jbi.2004.11.006. [DOI] [PubMed] [Google Scholar]
  • 11.Centers for Medicare & Medicaid Services (CMS), HHS. Medicare and medicaid programs; electronic health record incentive program–stage 2. final rule. Fed Regist [Internet] 2012 Sep 4;77(171):53967–4162. [PubMed] [Google Scholar]
  • 12.Buller-Close K, Schriger DL, Baraff LJ. Heterogeneous effect of an emergency department expert charting system. Ann Emerg Med [Internet] 2003;41(5):644–52. doi: 10.1067/mem.2003.182. [DOI] [PubMed] [Google Scholar]
  • 13.Clayton PD, Naus SP, Bowes WA, 3rd, Madsen TS, Wilcox AB, Orsmond G, Rocha B, Thornton SN, Jones S, Jacobsen CA, Udall MR, Rhodes ML, Wallace BE, Cannon W, Gardner J, Huff SM, Leckman L. Physician use of electronic medical records: Issues and successes with direct data entry and physician productivity. AMIA Annu Symp Proc [Internet] 2005:141–5. [PMC free article] [PubMed] [Google Scholar]
  • 14.Hammond KW, Helbig ST, Benson CC, Brathwaite-Sketoe BM. Are electronic medical records trustworthy? observations on copying, pasting and duplication. AMIA Annu Symp Proc [Internet] 2003:269–73. [PMC free article] [PubMed] [Google Scholar]
  • 15.Logan JR, Gorman PN, Middleton B. Measuring the quality of medical records: A method for comparing completeness and correctness of clinical encounter data. Proc AMIA Symp [Internet] 2001:408–12. [PMC free article] [PubMed] [Google Scholar]
  • 16.Stetson PD, Morrison FP, Bakken S, Johnson SB. Preliminary development of the physician documentation quality instrument. Journal of the American Medical Informatics Association [Internet] 2008;15(4):534–41. doi: 10.1197/jamia.M2404. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Stetson PD, Bakken S, Wrenn JO, Siegler EL. Assessing electronic note quality using the physician documentation quality instrument (PDQI-9) Appl Clin Inform [Internet] 2012;3(2):164–74. doi: 10.4338/ACI-2011-11-RA-0070. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Rosenbloom ST, Crow AN, Blackford JU, Johnson KB. Cognitive factors influencing perceptions of clinical documentation tools. J Biomed Inform [Internet] 2007;40(2):106–13. doi: 10.1016/j.jbi.2006.06.006. [DOI] [PubMed] [Google Scholar]
  • 19.Lin HS, Stead WW. Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions [Internet] National Academies Press; 2009. [PubMed] [Google Scholar]
  • 20.Brown P, Marquard J, Amster B, Romoser M, Friderici J, Goff S, Fisher D. What do physicians read (and ignore) in electronic progress notes? Appl Clin Inform [Internet] 2014;5(2):430–44. doi: 10.4338/ACI-2014-01-RA-0003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Schensul SL, Schensul JJ, LeCompte MD. Essential ethnographic methods: Observations, interviews, and questionnaires. [Internet] Rowman; Altamira: p. 1999. [Google Scholar]
  • 22.Chesluk BJ, Holmboe ES. How teams work–or don’t–in primary care: A field study on internal medicine practices. Health Aff (Millwood) [Internet] 2010 May;29(5):874–9. doi: 10.1377/hlthaff.2009.1093. [DOI] [PubMed] [Google Scholar]
  • 23.Saleem JJ, Plew WR, Speir RC, Herout J, Wilck NR, Ryan DM, Cullen TA, Scott JM, Beene MS, Phillips T. Understanding barriers and facilitators to the use of clinical information systems for intensive care units and anesthesia record keeping: A rapid ethnography. Int J Med Inf [Internet] 2015;84(7):500–11. doi: 10.1016/j.ijmedinf.2015.03.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.McMullen C, Ash J, Sittig D, Bunce A, Guappone K, Dykstra R, Carpenter J, Richardson J, Wright A. Rapid assessment of clinical information systems in the healthcare setting. Methods Inf Med [Internet] 2011;50(4):299–307. doi: 10.3414/ME10-01-0042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Dorfsman ML, Wolfson AB. Direct observation of residents in the emergency department: A structured educational program. Acad Emerg Med [Internet] 2009;16(4):343–51. doi: 10.1111/j.1553-2712.2009.00362.x. [DOI] [PubMed] [Google Scholar]
  • 26.Kopp BJ, Erstad BL, Allen ME, Theodorou AA, Priestley G. Medication errors and adverse drug events in an intensive care unit: Direct observation approach for detection. Crit Care Med [Internet] 2006;34(2):415–25. doi: 10.1097/01.ccm.0000198106.54306.d7. [DOI] [PubMed] [Google Scholar]
  • 27.Timestamped field notes [Internet] cited Dec 4, 2015] Available from: http://www.neukadye.com/mobile-applications/timestamped-field-notes/
  • 28.Srnka KJ, Koeszegi ST. From words to numbers: How to transform qualitative data into meaningful quantitative results. Schmalenbach Business Review [Internet] 2007;59(1):29–57. [Google Scholar]
  • 29.NVivo-QSR International. [Internet]. cited Dec 6, 2015] Available from: http://www.qsrinternational.com/product.
  • 30.Template-Definition [Internet] cited Dec 6, 2015] Available from: http://www.merriam-webster.com/dictionary/template.
  • 31.Graphical User Interface. [Internet]. cited Dec 6, 2015] Available from: http://www.merriam-webster.com/dictionary/graphical user interface.
  • 32.Farri O, Rahman A, Monsen K, Zhang R, Pakhomov S, Pieczkiewicz D, Speedie S, Melton G. Impact of a prototype visualization tool for new information in EHR clinical documents. Appl Clin Inform [Internet] 2012;3(4):404–18. doi: 10.4338/ACI-2012-05-RA-0017. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES