Specifying Design Criteria for Electronic Medical Record Interface Using Cognitive Framework

Abstract

As the healthcare industry transitions from paper to electronic medical records (EMRs), medical informatics researchers face the task of ensuring that the electronic presentation of the information remains usable and effective while capitalizing on the ability of EMRs to tailor information to different users. In our research, we focus on utilizing formal cognitive science methodology to guide the conversion of paper-based narrative discharge summaries to a more dynamic, structured electronic version. In this paper, we present the results of a cognitive analytic study (1) that determines a ‘core’ component in medical narratives and (2) that compares the use of structured and narrative texts by physicians with varying expertise. Specifically, we studied six psychiatrists at three levels of expertise— experts, intermediates, and novices. The subjects were given two clinical case scenarios with discharge summaries and asked to verbalize their thoughts as they read through the summaries. The interview transcripts were analyzed for recalls and inferences generated in the verbalization. Based on experts’ verbalizations, the discharge summaries were organized into a more structured form and used in the interview of other subjects. Novice-level subjects had more recall with the structured than with the narrative format. More errors were also made in recall with the narrative than with the structured text. We discuss how these results are valuable in designing an EMR interface to reduce errors and to support users of different expertise.

1. Background

The paradigms of healthcare are shifting in favor of enhancing performance and maximizing efficiency by utilizing the emerging information technologies. Prominent amongst these technologies is the Electronic Medical Record (EMR). The transition from the traditional paper-based medical record to the EMR is complex – how do we transform the information available on paper to create an electronic version? There have been naïve attempts at mere replication, but making the computer screen display an exact copy of the paper chart may actually slow down the care-delivery process. For example, in terms of data entry, the healthcare provider may spend more time in selecting the appropriate option from pull-down menus, radio buttons and checkboxes in comparison to jotting down a few words on a piece of paper. Similarly, in terms of information retrieval, the effort of authorizing, clicking hyperlinks and menu selections may be tedious when compared to a simple act of flipping through a set of papers. Thus, an important goal in the process of shifting from a paper chart to an electronic version on a computer is to minimize the ‘friction’ on the user. Cognitive science methodologies can help us understand the end-user in terms of human comprehension, problem-solving behavior and his or her interaction with EMR technology. We use this approach to specify design criteria for the design of the EMR. In this paper, we present the results of a cognitive analytic study that compares the use of narrative and structured texts at different levels of physician expertise, and we show how the results can assist in the design of the EMR interface for discharge summaries.

Tange and colleagues (1) define “medical narratives” as all the qualitative (and semi-quantitative) data gathered by the physician. Their term comprises the physician-gathered ‘core parts’ of the medical record (medical history, physical examination, progress notes and episode summaries (discharge letters). Natural prose is the typical way to express ones observations and thoughts in such documents. The ‘fluid’ nature of their content underlies the complexity of the process of information transfer from paper to electronic environment. The problem with narrative, unstructured format is that it is susceptible to being distorted to the extent of being incoherent in this process (2). This very nature of narrative medical texts motivates us to develop a solution to the problem of ‘friction’ stated above by finding an ingenious structure of narrative medical texts that can maximally utilize the scope of an electronic environment and aid the provider in care delivery. Since the interpretation of narrative text can vary according to the domain expertise of the reader, finding the ‘core’ component of medical text to represent and structure in an EMR would enable a non-expert to interpret in as much similar way as an expert. In our research on the design of EMR, we have chosen to focus on a specific narrative section of the patient record, the discharge summary. There has been very little research on the interface design to access and review discharge summaries in electronic medical records. Attempts at generating a structure for discharge summaries in the past have been mainly based on opinions of a group of domain experts via surveys or questionnaires. This approach is a useful guide but suffers from personal biases and is not theoretically-driven, since it mostly represents the expert’s intuition or experience. Such attempts only address the issues with the organization of discharge summary content in a generic sense and are not specific to the electronic environment. In our research, we use a methodology based on principles of cognitive science in which we analyze the thought processes and conceptual representations of end users (physicians) to identify the maximally utilized ‘core’ structure of the text. In Section 2, we present the theoretical framework of our approach. In Section 3, we provide the methodology of our study examining the variations in the interpretation of psychiatric discharge summaries by format and level of expertise. In Section 4, we present our results that demonstrate a ‘core’ structure, and, in Section 5, we give examples as to how this ‘core’ structure can be represented in a computerized interface and favor a set of interpretations that remain constant irrespective of the nature of expertise of the reader.

2. Theoretical Framework

A text is narrative when it is recorded in natural prose, which is typically the way to express ones observations and thoughts. It does not guarantee, however, that the reader would understand this information exactly as the author meant it (1, 3). Natural language text has two central and complementary component processes — production and interpretation. Both of these can be analyzed in terms of natural linguistic structures (e.g., syntax), propositions, concepts and discourse dynamics (4). Information retrieval from such text has two steps. First, the user has to search through the text, which is guided by its internal structure, to select the section that might contain the relevant information. Second, the user has to read the content of this section to retrieve the information needed (5). Structured text, in contrast, is a format that has an inherent order to it and is highly organized. The content is clearly divided into subsections, and the coherence enables the reader to find and focus on relevant information efficiently. Structured texts may be easier and faster to use in some ‘real-life’ situations (like clinical practice) since they enable rapid selection of relevant information. However, due to their structured nature, they tend to be condensed and may be unable to provide as much information as the narrative texts. Structured text does not produce any significant change in an expert’s efficiency (when compared to non-experts); experts are more capable of ‘filtering’ relevant information out of the given text, irrespective of the format. Although medical narrative texts use natural prose, they also have a certain degree of inherent organization in them. The word ‘structured’ denotes a predominantly organized format, not an absolute one. So, ‘structured’ and ‘narrative’ effectively refer to ‘mostly structured’ and ‘mostly narrative’ texts.

Patel and colleagues (4, 6) point out that processing natural language text requires a distinction between the ‘text base’ and ‘situation model’. A ‘text base’ is a propositional representation of the meaning of the text itself. The reader generates meaning from the text by transforming the written information into some semantic form or conceptual message. In similar context, understanding may be regarded as a process whereby readers attempt to infer the knowledge structure of a writer through the text, based on what they already know. Thus the mental model developed by an individual who is reading a text is not limited to the information contained in the text itself but is extended to incorporate the reader’s prior knowledge. In this sense, the reader constructs a ‘situation model’ of the scenario described in the text. It is, then, from the interaction between the text-base and the situation model that the conceptual representation emerges (6). This representation varies greatly from reader to reader as prior knowledge and experiences (i.e., expertise) differ. Unquestionably, expertise also plays a vital role in determining physician’s comprehension, problem-solving and decision-making abilities.

We have applied the theoretical concepts of variation in format and expertise, as described above, to our study of discharge summaries. The doctor-patient interaction generates, on one hand, a patient chart as the ‘text base’ since the chart is representing the clinical information gathered from and provided by the patient. On the other hand, the discharge summary created as the result of the interaction, is the representation of biomedical and clinical information originating from the physicians experience, and can be equated to the ‘situation model’. Using the cognitive framework, the discharge summary is read in a subsequent patient visit as a part of “text-base” for that particular patient. Further adding to the complexity, the discharge summary may be written by a domain expert, but then viewed by a non-expert (e.g., a physician who is not specialist in the same field as the author of the summary), who is using the information to make an initial assessment of the patient. This non-expert then interprets the information in the discharge summary to produce another high-level abstract of the clinical case, thus turning this text base into a ‘situation model’ again. This second situation model may or may not be accurate, due to the variations in expertise and interpretation of natural language text as described in preceding sections.

An important goal of studying any medical text or artifact is to understand the set of interpretations that remain constant and relevant irrespective of the nature of expertise of the reader. Therefore, it is imperative that, in exploiting the re-organization of information provided by the electronic environment, we facilitate a constant and relevant set of interpretations. In our view, it may be possible to represent the discharge summary text in a format that can counterbalance the variations in domain expertise and minimize interpretational errors. This would support a non-expert in interpreting in a similar way to an expert, enabling standard and consistent care-delivery throughout the expertise continuum. Studies have also shown that physicians prefer structured format and errors can be avoided if standardized formats are completed. Using a structured summary can help also to focus on the most appropriate information, facilitates retrieval, has educational value and promotes brevity.

3. Methods

3.1. Methodological framework

In the cognitive science field of comprehension, propositional analysis is a formal method for investigating representation of meaning in memory. Propositional analysis, however, does not end with the identification of the concepts alone. One can go further and identify the agent and goals of the action and the instrument used. For our study, it was sufficient to stay at a less detailed level of analysis, since the purpose was only to identify main concepts and categorize them into recall or inferences.

The generation of inferences in reading a text is linked to the understandability and the coherence of a text (6). Structure arguably makes a text more coherent and easier to understand by its organized nature. The amount of prior knowledge (expertise) in reading a text is also related to the process of inference generation and comprehensibility. Domain experts tend to generate higher level of inferences, whereas novices tend to create lower-level inferences or none at all. Recall is usually low for experts and high for novices, unless the text is very simple, in which case experts have high recall. Propositional analysis can help us understand the nature of expert–novice differences in text understanding and how a given information can be tailored to a particular target audience, thereby reducing both excessive use of working memory (especially in novices) and the possibility of incorrect inferences.

3.2. Study design

We undertook a study of interpretation of psychiatric discharge summaries with clinically minimal risks. An attending physician in the psychiatric emergency department at Columbia Presbyterian Medical Center (A.D.) served as a domain expert, and prepared two narrative discharge summaries based on textbook cases. He also developed for each discharge summary a written clinical scenario describing an initial evaluation of a patient presenting to the psychiatric emergency room after recently being discharged from an inpatient unit. Each discharge summary corresponded to a hypothetical patient’s admission, and was modeled by the domain expert to be similar in style and length to those available in ‘WebCIS’, the EMR at Columbia Presbyterian Medical Center. The corresponding proposition ‘density’ in each discharge summary was also designed to be comparable. Six psychiatrists were recruited with varying emergency room expertise – two experts, two intermediates, and two novices. The two experts chosen were attending-level physicians in the emergency room. The intermediates were residents in their fourth year of training, and had completed a year-long rotation in emergency psychiatry. The novices were second-year residents who had completed a one-month emergency room rotation in their first year of residency. To undertake propositional analysis, subjects were given a text description of the clinical scenario and the corresponding discharge summary and were asked to ‘think aloud’ as they read the information. The study was divided in two phases. To derive the ‘structured’ form of both the summaries, the first phase of the study comprised of analyzing expert’s interpretations from the discharge summary using think-aloud techniques, generating two protocols per subject. Based on the experts’ interpretations from the two narrative discharge summaries, a structured format was produced in consultation with the domain expert. This format had equal or fewer propositions (to achieve equal propositional density) in each section and was organized in a way that each salient information cluster was printed as a bulleted point under its respective section heading. In the second phase of the study, each of the intermediates and novices was given either the ‘narrative’ or ‘structured’ form of one case. Using a cross-design technique on a subsequent session, the subjects were then given the alternate form (narrative or structured) of the second case.

All interviews were recorded and transcribed. The transcripts were then divided into key propositions and a detailed manual mapping of the propositions between transcript and text was done for each section of the discharge summary. In effect, each concept verbalized by the subject was mapped to the presence or absence of a corresponding concept in the discharge summary. The mapping was analyzed in terms of recall and inferences amongst the total number of propositions in each section of the summary. Details of the methods are given in (6).

4. Results

4.1 Information “most” inferred

We present in Table 1, for each expert and each case, the percentage of inferences made in each section of the discharge summary. The results of the propositional analysis allowed us to identify the ‘core’ component of the discharge summaries, the sections of text that are consistently utilized by both the expert subjects and accounted for majority of the interpretations (76%) generated. As shown in Table 1, these sections are “History of Present Illness”, “Past Psychiatric History” and “Hospital Course.”

Table 1.

Discharge Summary Sections	Percentage of propositions inferred in each section
	Expert 1		Expert 2
	Case 1	Case 2	Case 1	Case 2
Identifying data	7	11	9	0
H. of present illness	40	32	20	34
Past psych. history	21	32	40	49
Social history	5	5	20	0
Mental Status	14	0	3	3
Admitting diagnosis	0	5	0	0
Hospital course	12	11	9	11
Discharge diagnosis	12	5	0	3
Discharge plan	0	0	0	0

4.2 Inference and recall by intermediates & novices

Figure 1 shows the percentage of recalled propositions by novices for each of the identified ‘core’ sections of the discharge summary by type of format (narrative or structured). The most striking result in this graph is that novice subjects showed higher recall in structured format than narrative – increasing their performance level closer to that of the experts.

Figure 1.

Recall for novices.

Figure 2 provides the result for novice subjects in terms of their inferences from the narrative and structured discharge summaries. As shown, the novices consistently infer more from the structured format in each of the ‘core’ sections, so this forrmat may support more ‘expert-like’ behavior in novices. Differences between structured and narrative formats for the intermediate subjects had, in contrast, no consistent patterns in both recall and inference (See Figure 3 in Appendix).

Figure 2.

Inferences for novices.

4.3 Errors

Seven comprehension errors were made in the ten interviews of subjects who were given narrative formats. The most prominent amongst these was the ‘family history’ content of the narrative discharge summaries. Although this information was included under the section titled “Past Psychiatric History”, two of the eight subjects made errors while reading it. Interestingly, none of the subjects missed, ignored or misinterpreted it when an explicit heading and section titled “Family History” was included in the structured discharge summary.

5. Discussion

In this paper, we focus on the problem that healthcare providers face in transitioning from paper charts to the EMR, and present a novel approach, based on cognitive analytic methodology, for designing the EMR interface for medical narratives. This study demonstrates the value of our methodology in structuring electronic discharge summaries to support varying levels of expertise and to reduce errors in recall of information. Based on expert use and interpretation, we identified a ‘core’ component of psychiatric discharge summaries in their use for emergency care. In the design of an EMR interface for this setting, these pertinent sections should be in detail on the first screen that a physician views when he or she reviews the patient’s discharge summary. The rest of the sections could be presented as ‘hyperlinks’, which could be opened up by the viewer, if desired. The rationale behind this distinction is that the majority of the inferences from the given cases were made from these sections, and it is reasonable to expect the same in an electronic interface since the informational content would be constant across both mediums. The user would thus save time and effort in looking for the most needed information.

In our study, we have also examined the use of the narrative and structured discharge summary by non-expert physicians. Non-experts had more recall with the structured format than with narrative format, which is consistent with the subjects bringing their own knowledge to construct a narrative-like story from bulleted information. In terms of inferences, our study results for intermediate subjects confirmed the previously demonstrated “intermediate effect” (7). Intermediate subjects, whose expertise is in between experts and novices, tend to generate more inferences that are extraneous. Intermediate subjects show a great deal more knowledge than novice subjects, but they have not yet acquired the expert strategies to deal with this information. In terms of accuracy, all subjects were accurate in their recall, as expected, since they all have ‘generic expertise’ of being physicians, whereas differences in inference accuracy was not expected, since they are specialized in psychiatry with different levels of expertise (specific expertise) (7). Expert, intermediate and novice subjects all made errors, but the nature of errors was different. In an electronic version of the text, customized ‘views’ corresponding to the expertise of the viewer could be helpful to counteract these errors to some extent, and future research should be directed in this direction. The EMR design could also be utilized to make omission-prone information more conspicuous to the users, thereby minimizing subsequent errors. Based on methodology similar to the one used in our study, one could identify and order the content of a narrative text in terms of institutional/user priority and make the important content more ‘eye-catching’ (e.g., by color/font variation or making a new heading/section).

The issues addressed in this paper can be generalized beyond the specific case of transferring narrative discharge summary content from paper to the EMR. The healthcare domain faces similar challenges in analogous situations where one needs to transmogrify information to a more organized format that is optimally sectioned to facilitate easy data entry and information retrieval. Some healthcare forms filled out by providers in a patient encounter may be a worthy example. These patient visit forms are antiquated in terms of their layout and organization. Often the more important information sections are scattered all over the form, rather than being the initial ones on the first page. Another situation could be where the traditional documents generated in a patient visit (e.g., signout notes) need to be standardized across the institution. How does one decide the final layout of these standardized, structured versions? Commonly, a group consensus of local domain experts of the institution is considered the basis for such formats. Our methodology provides a more logical and scientific alternative to that approach, since it is based on the analysis of conceptual models of end-users of such documents, the physicians themselves. Using our approach, one could identify the pertinent ‘core’ components of relevant information in the text and utilize them to optimally segment the text in terms of importance of content. As in the example of discharge summaries, one could ‘extract’ the structure of a given narrative text from the knowledge and conceptual representation of the end users themselves, thereby increasing the applicability and effectiveness of such structured formats manifold. In our future research, we plan to examine these issues further, as well as focus on the accuracy of inferences made in using medical narratives in an electronic form rather than on paper.

APPENDIX

Figure 3.

Recall for intermediates.