Making Sense of the Cognitive Task of Medication Reconciliation Using a Card Sorting Task

Abstract

Objective:

To explore cognitive strategies clinicians apply while performing a medication reconciliation task, handling incomplete and conflicting information.

Background:

Medication reconciliation is a method clinicians apply to find and resolve inconsistencies in patients’ medications and medical conditions lists. The cognitive strategies clinicians use during reconciliation are unclear. Controlled lab experiments can explore how clinicians make sense of uncertain, missing, or conflicting information and therefore support the development of a human performance model. We hypothesize that clinicians apply varied cognitive strategies to handle this task and that profession and experience affect these strategies.

Method:

130 clinicians participated in a tablet-based experiment conducted in a large American teaching hospital. They were asked to simulate medication reconciliation using a card sorting task (CaST) to organize medication and medical condition lists of a specific clinical case. Later on, they were presented with new information and were asked to add it to their arrangements. We quantitatively and qualitatively analyzed the ways clinicians arranged patient information.

Results:

Four distinct cognitive strategies were identified (“Conditions first”: n = 76 clinicians, “Medications first”: n = 7, “Crossover”: n = 17, and “Alternating”: n = 10). The strategy clinicians applied was affected by their experience (p = .02) but not by their profession. At the appearance of new information, clinicians moved medication cards more frequently (75.2 movements vs. 49.6 movements, p < .001), suggesting that they match medications to medical conditions.

Conclusion:

Clinicians apply various cognitive strategies while reconciling medications and medical conditions.

Application:

Clinical information systems should support multiple cognitive strategies, allowing flexibility in organizing information.

Background

Medication reconciliation is “discovering and correcting” inaccuracies in medication lists (Cook, 2017). Establishing and maintaining an accurate list of medications is difficult, especially when these lists are a result of varied processes and different sources (e.g., community clinics, hospital release forms, patient and family, etc.; Magalhães, Santos, Rosa, & Noblat Lde, 2014; van der Gaag, Janssen, Wessemius, Siegert, & Karapinar-Çarkit, 2017). Medication lists are prone to gaps (Cook, Render, & Woods, 2000). Reconciliation is far from trivial. Many people, particularly the severely ill, take a dozen or more medicines at home. The average number of medications at admission to hospital is around eight; the average at discharge is more than nine (Kramer et al., 2014). Patients’ medications change as their medical conditions change, as they accrue new diagnoses, and as they transition from one venue to another. Often there are multiple, incomplete, or even conflicting sources of information about what medications a patient is taking. Establishing a consistent and accurate list reconciles these sources (Rose, Fischer, & Paasche-Orlow, 2017). Medication reconciliation plays an important role in patient safety (Leguelinel-Blache et al., 2014) as it is a method to highlight conflicts and inconsistencies in patient care. Therefore, it is mandated in many countries (Kwan, Lo, Sampson, & Shojania, 2013). Frequently, this requirement is met by comparing lists from different patient encounters.

The current experiment seeks to evoke reconciliation by presenting a data field that requires evaluation of two related but dissimilar sets. Medication and medical condition lists are two collections and are closely related. A patient’s use of particular medications implies the presence of one or more particular medical conditions. Although there are durable patterns of medications and conditions, individual medication regimens vary widely. The problem with the accuracy of a list of medications that results from comparing medication lists in the context of patient medical conditions is that it ignores the common reality that these lists might be incomplete or incorrect, particularly if obtained by an external provider or if they depend on incomplete memory. Thus, medication reconciliation is not a structured process like comparing two drug lists but rather a partially unstructured cognitive task (Magalhães et al., 2014). Clinicians develop and use mental models to discover and resolve inconsistencies across information collections (Vashitz et al., 2013). Studies of human cognition praise the innovative and complex cognitive processes humans apply when they are facing ambiguous and unpredictable situations (Klein, 2011). Like many such decision-making processes, these process might be subconscious (Dijksterhuis, 2004), and experts who make these decisions may be remarkably incapable of articulating what exactly they are doing (Klein, 1999).

Improving our understanding of the cognitive strategies clinicians apply during reconciliation will contribute to the development of innovative tools that might support clinicians during this critical task. User-centered design describes design processes to account for the way end users would use a system (Abras, Maloney-Krichmar, & Preece, 2004). The designer facilitates tasks the user has to complete in the system, especially in working environments that require quick and accurate user response. Achieving user-centered design requires detailed information about the tasks and processes that the user engages with the system, and this means understanding their cognition and mental models (Thorvald, Högberg, & Case, 2012). User-centered design has been implemented in health care information systems, particularly in the context of visual presentation of electronic health records (Duarte & Guerra, 2012; Thyvalikakath et al., 2014), and studies show that this method can improve the design of medical equipment (Lin et al., 1998). The challenge of applying a user-centered approach is identifying the cognitive strategies the user is employing while performing his task.

Medication reconciliation is an activity providers should and often do, mandated by their hospitals or health accreditation bodies. To the best of our knowledge, there is no standard way of teaching how to perform this task, and therefore, it might be practiced in a different way by clinicians of different professions who studied and worked in varied places. The task is often facilitated and documented by forms or activities in the electronic medical record systems. Although the technical tools that support and record this activity might vary between health care organizations, it is the cognitive activity that interested us in this task. Given inconsistencies, lack of a standard source of information, and known gaps in the continuity of care (Cook et al., 2000), further insights would come from understanding the cognitive strategies clinicians apply to make sense of uncertain or missing information. To evaluate the cognitive strategies such as medication reconciliation, one needs to understand how clinicians process information during a task. Particularly, we wish to know how clinicians arrange information, so they can make sense of it quickly and efficiently.

Previous cognitive research has benefited from studying behaviors in natural environments (Hutchins, 1995) as a way to understand behavior in context (Bradel et al., 2013). In our previous studies, we simulated a reconciliation task in a lab experiment using a card game demonstrating that clinicians follow common cognitive strategies when they perform medication reconciliation tasks (Vashitz et al., 2011) and that they preferentially sorted medical information by organ systems (Vashitz et al., 2013). New insights on variability in strategies, and possible ways to uncover missing information, would enhance this understanding as a foundation for future innovative user-centered design.

Aims

We aimed to model human performances as they thought about mediating medications in the context of patient care. We studied the cognitive strategies clinicians applied during a medication reconciliation task and draw recommendations for the design of health care information systems. We were particularly interested in strategies they applied when making sense of conflicting or missing information, which is often the case in real-life medication reconciliation. We aimed to examine whether profession and experience affect these strategies (e.g., nurses vs. physicians and beginner vs. expert), as would be expected due to the fact that this is a partially unstructured cognitive task that might be affected by varied conditional factors.

To achieve these aims, the study was designed to test the following primary hypotheses:

• Hypothesis 1: Clinicians would use varied strategies to arrange the information in a way that would make sense to them—we expect to find few distinguished strategies to arrange the reconciliation information.

• Hypothesis 2: Clinicians with different professions (physicians vs. nurses) or different levels of experience (beginner vs. expert) would approach this task differently—we expected our results to have significant differences between varied clinician types.

• Hypothesis 3: Once clinicians receive additional information, they would update the way they understand the situation (i.e., their mental models) according to the updated information—we expected all clinicians to adjust the way they arranged the information once new information is revealed.

Method

The Experiment

Our experimental method relies on the simplicity of the task presented to clinicians and the efficiency with which they complete the task. We extracted information on a single clinical case—a 66-year-old female presenting for resection of a base-of-tongue lesion. Patient medications and medical conditions were copied, and the participant had to arrange the information in front of them. The simulated medication reconciliation in this experiment is based on a previous simulated reconciliation task (Vashitz et al., 2011). In the current study, we adapted the physical card sorting task (CaST) to digital mobile media, to improve dissemination and data collection. The experimenter recruited a convenience sample of on-duty clinicians from anesthesia and critical care specialties, asking for their voluntary participation in the short study. The experimenter provided information about the clinical case, asked the participants to organize the information presented on a dedicated software running on a 10″ Android^TM tablet computer screen by moving cards around, and waited until they stated that they completed the task. Adaptation of the original experiment to a software-based experiment running on a tablet enabled us to bring the simulated reconciliation task to the clinical environment and thus capture a larger number of providers from varied professions. Although the tablet computer we used might have different screen dimensions than other systems that might record medication reconciliation in health care systems, our tablet experiment was aimed at the cognitive mental task and therefore less affected by the display form factor. The current experiment was conducted in a large American teaching hospital. It is complied with the American Psychological Association Code of Ethics and was approved by the Institutional Review Board at The University of Chicago. Informed consent was obtained from each participant.

In Part 1 of the experiment, the reconciliation task consisted of 18 initial cards: nine medications and nine medical conditions, arranged in consistent but unorganized scattered arrangement. The initial arrangement (zero state) is shown in Figure 1. Clinical conditions were labeled with a date, if known, and medications were listed by generic name with a trade name in parentheses.

Figure 1.

The initial experimental setting. DVT = Deep Venous Thrombosis; COPD = Chronic Obstructive Pulmonary Disease; CA = Carcinoma; MI = Myocardial Infarction; CAD = Coronary Artery Disease.

The experimental task is semantic and virtually impossible for a nonclinician. Although there may be one-to-one drug–disease associations, it is common for a disease to be treated with more than one drug, and a drug may have effect on more than one medical condition. The relationships between conditions and medications are varied, sometimes fluid, and based on assumptions. Participants were asked to handle the information as though they were taking this patient under their care, and that the information was all that was initially available to them. They were instructed to arrange the cards as they saw fit to help organize the information and to discuss their thoughts aloud as they made arrangements. They did not have any limit on the duration of their performance. Seeing how providers move the cards helps reveal some assumptions. They were also prompted to explain what they were thinking if they did not articulate their thoughts initially. The Android^TM software recorded the card movements and made an audio recording of the conversation for later playback.

In Part 2 of the experiment, three new cards (two medications: clopidogrel and digoxin, and one medical condition: cerebrovascular accident) were added to the set. We wanted to understand how clinicians confront uncertain or missing data and how they handle the additional information that was provided. Participants were asked to add the new cards to the arrangement and articulate their thoughts. These new cards added context to the existing information. The potassium is an outlier in the initial arrangement, as it has no obvious pairings in the other medications and conditions. It is usually given for a specific reason, most commonly to prevent side effects of another medication, such as a diuretic or the drug digoxin. Digoxin is introduced in Part 2 of the experiment. Watching the relationships change with new information adds clarity to some of the uncertainties and weaknesses behind the connections between the cards.

We recruited 153 clinicians with varied clinical professions and years of experience. The two main profession groups were physicians and nurses. Experience was categorized to four groups as defined by Benner (1982): beginner, competent, proficient, and expert. We defined beginners as clinicians in their first year of working in the intensive care unit (ICU) because the learning curve in the first year in the ICU is steep. Participants with 1 to 5 years of experience were assigned to the competent group, 5 to 10 years to the proficient group, and more than 10 years to the expert group. This categorization is based on the perceptions of clinicians that work in the ICU. We excluded three medical students who had just begun their clinical rotations, as they lacked clinical experience in the ICU and therefore would be considered novice (Benner, 1982). Twenty participants who did not complete the scenario due to calls and other interruptions that limited the time they could dedicate to the experiment were also excluded. Of all 130 participants analyzed, 73 were nurses and 57 physicians (Table 1). Due to software-related issues in the data collection program, the card movements of 20 participants were not captured during the second part of the experiment. Therefore, we included in our analysis of the second part of the experiment data from 110 participants.

Table 1:

Participant Characteristics

Years of Experience	Attending Physician	Resident/Fellow—Anesthesiology	Resident/Fellow—Other	Nurse—ICU/CRNA	Nurse—PACU/Pre-op	Total
0–1	0	4	7	3	1	15
1–5	1	21	3	13	4	42
5–10	5	2	4	11	8	30
>10	10	0	0	15	18	43
Total	16	27	14	42	31	130

Note. ICU = intensive care unit; CRNA = Certified Registered Nurse Anesthetists; PACU = postanesthesia care unit.

Our exploration followed three analyses of the data. In Part 1 of the experiment, we explored the participants’ first sensemaking of the patient’s data. All the information participants had about the patient was in 18 cards, with a very limited background history. The three cards that were added in Part 2 of the experiment provided additional information about the patient, and therefore, this part of the experiment was an opportunity for the subjects to rearrange the medications and conditions based on this. Data analysis first explored card arrangement order at the end of the first part of the experiment, looking at the order each card arrived at its final position (grouped by card type—condition or medication). In the second part of the analysis, we measured how participants handled the potassium card. We looked at how often potassium was moved compared with other cards, and the total distance (pixels) it traveled to its new final position after the introduction of the digoxin card. Unlike our previous studies that looked at arrangement position (Vashitz et al., 2011; Vashitz et al., 2013), we designed our current software to focus on cards arrangement order and the distance cards traveled. Finally, to support our understanding of participants’ considerations regarding potassium and its relation to other cards, we supported our data with a verbal protocol, a common mixed methods combination of qualitative and quantitative data (Carayon et al., 2015). We analyzed the audio recordings during Part 1 of the experiment specifically addressing this card. We expected the verbal protocol to provide additional information, demonstrating confusion and uncertainty specifically related to potassium.

Statistical Analysis

We drew two descending plots that describe the order cards arrived at their final position. One plot represented clinical condition cards, and the other medication cards (Figure 2 presents few of these plots). To understand how clinicians arranged medication versus condition information, we compared the relative plot patterns for each of the participants. As the plots represent the order in which cards arrived at their final position at the end of the first part of the experiment, the distance between the two plots represents the difference in the number of cards from each type that arrived at their final position. We measured the maximum distance between the two plots, the number of times the two plots crossed each other, and the cumulative distance between the plots and developed a small computer program that classified each plot to one of four defined strategies that best describes the way the participant arranged the cards (Table 2).

Figure 2.

Sample results that represent the four strategies for arranging the information (Participant 1597 for condition first; Participant 4912 for medication first; Participant 2440 for crossover; Participant 1842 for alternating).

Table 2:

Criteria for the Four Strategies Classification

Criteria	Maximum Distance Between the Two Plots	Number of Times the Two Plots Crossed Each Other	Cumulative Distance Between the Plots
Strategies
Conditions first	High	Low	High
Medications first	High	Low	High
Crossover	Medium	Medium	Medium
Alternating	Low	High	Low

For the second part of the experiment, we used Wilcoxon rank sum test (Mann–Whitney) with continuity correction to test card movement frequencies significance. We also compared the mean Euclidean travel distances (pixels) of all 18 cards using Friedman rank sum test (Daniel, 1990) against the null hypothesis that participants treated all cards in the same way and that on average all cards would travel the same distance (pixels). After the main test was rejected, we performed a many-to-one comparison of the distance (pixels) the potassium card traveled compared with all other medication and condition cards. The p values of these many-to-one comparisons were adjusted using Benjamini–Hochberg (BH) step-up procedure to control the false discovery rate (Benjamini & Hochberg, 1995).

Results

Medication Reconciliation Strategies

We identified participants for each of the four strategies for reconciling conditions and medications and matched 110 of the 130 participants. Sample results that represent these strategies are presented in Figure 2. The other 20 participants did not have any consistent pattern and did not match any of the four main strategies we identified. In the most common strategy (76 participants), “Conditions first,” clinicians first sorted most conditions and, once finished, matched relevant medications to each condition. These participants based their cognitive process on the condition, using it as a cognitive anchor for this strategy. The opposite strategy, “Medications first,” occurred for only seven participants. In the “Crossover” strategy, the participants started by sorting some cards from one type (condition or medication) and then crossed to the other type of card, sorting most of the cards from the other type. The cognitive anchor for this strategy switched during the sorting task. Seventeen participants employed this strategy. In an “Alternating” strategy, the clinician chose a single condition and matched a relevant medication to this condition. Similarly, the clinician may have chosen a single medication and matched a relevant condition to it. Then they turned to the next card and continued this stepwise selection. Ten participants employed this strategy.

Effect of Profession and Experience

Analyzing the median distance between the two descending plot measurements reveals an interesting finding—clinicians’ experience had a significant effect on the strategy that they applied to arrange the cards $({\chi }_3^2=10.16,p=.02)$. Comparing clinicians’ profession—physicians (attending physicians and residents) versus nurses (ICU/Certified Registered Nurse Anesthetists [CRNA] and postanesthesia care unit [PACU]/pre-op)—did not have a similar significant effect. Due to the way these measurements were calculated, they were highly correlated, and therefore effects were similar for both professions.

Impact of New Information

Comparing the frequency each of the 18 cards moved during the second part of the experiment (Figure 3), the mean sum of medication card movement was 75.2 compared with 49.6 for condition cards (Wilcoxon rank sum test, W = 2.5; p < .001). This suggests that conditions served as anchors and that participants match medications to conditions rather than conditions to medications. When analyzing these results by the clinicians’ experience groups, the mean frequency of medication card movements for experienced clinicians (more than 1 year of experience) was 77.2 compared with 49.3 for condition cards (Wilcoxon rank sum test, W = 2; p < .001). The results for the inexperienced clinicians were statistically not significant (61.1 movements for medications compared with 51.6 for condition cards).

Figure 3.

Frequency of movements of the 18 cards during Part 2 of the experiment. MI = Myocardial Infarction; COPD = Chronic Obstructive Pulmonary Disease; DVT = Deep Venous Thrombosis; CAD = Coronary Artery Disease.

The potassium card traveled the longest distance from its original position once new information was presented at the second part of the experiment (Figure 4); it traveled 93.2 pixels compared with 80.6 pixels for alprazolam, the second the longest distance traveled $({\chi }_{17}^2=41.9,p=.0001)$.

Figure 4.

Mean distances (pixels) the existing cards traveled in Part 2 of the experiment. MI = Myocardial Infarction; COPD = Chronic Obstructive Pulmonary Disease; DVT = Deep Venous Thrombosis; CAD = Coronary Artery Disease.

Analyzing the verbal protocols to try to understand clinicians’ interpretation of the potassium card, we found several revealing descriptions. Fourteen clinicians mentioned potassium, most expressing uncertainty about its meaning or suggesting that condition or medication data were missing. Statements include,

Potassium? Not so sure why potassium is there and not like here or something. (Participant 3802)

I don’t know if she’s on potassium for, if she’s hypokalemic. (Participant 556)

I’ll probably throw that up here by the cardiac drugs because it’s probably associated there. (Participant 1352)

One participant clearly tried to relate potassium to another medication:

. . . but albuterol can deplete potassium. (Participant 525)

One participant made the connection between potassium and a possible missing diuretic:

Coronary artery disease with her aspirin . . . potassium chloride. I would think that she would be on that if she was on a diuretic. But she’s not. (Participant 1115)

These comments reflect uncertainties, a sense that something is missing, and, in a few cases, a search for an explanation to address a gap in knowledge.

Discussion

The value of these data is 2-fold: first, they are abstractions of the clinical task that can serve as the basis for investigating clinicians’ cognitive strategies; second, they immediately suggest specific improvements that can be implemented to health care information systems that should improve task performance.

Consistent with our previous studies (Vashitz et al., 2011), the results address Hypothesis 1, showing that clinicians use varied, yet definable strategies to reconcile conditions and medications. The current study identifies new strategies and measures the difference between the groups that adopt each strategy. These strategies suggest the linkages between medications and medical conditions are an important part of clinicians’ techniques for sorting information. For most participants, conditions serve as an anchor, and they match medications to these anchors. In addition, the results suggest that experience affects the way clinicians perform the task of reconciliation. These results suggest that health care information systems should allow multiple methods to arrange clinical information and that the information should be arranged both by medications and by medical conditions.

We are able to infer how clinicians handle the possibility of missing information. The results demonstrate re-sorting when new information is presented, addressing Hypothesis 3. Additional information leads participants to make changes, adjusting the way they understand the situation according to the updated information. That new information disrupts an established arrangement suggests the participants may have been modifying their mental models of the patient. Learning about a new medication or a medical condition might close knowledge gaps or even introduce new ones, as the mental model becomes more detailed. Understanding these gaps and inconsistencies has an important role in improving patient safety.

As a marker for missing information, the treatment of the potassium card suggests clinicians notice inconsistencies. The card traveled more than the other cards, and several participants voiced concerns about the card and that information might be missing. Our results suggest that there is an opportunity to help clinicians explore their concerns about information that does not fit. These results suggest that health care information systems should support placeholders for missing information and afford the assumption that as more information is revealed, new interaction might need to be documented.

Addressing Hypothesis 2, results for strategies for arrangement and re-sorting with new information correlate with clinicians’ experience, suggesting that clinicians develop intuitions for how to handle these cognitive tasks over time. Another interesting finding is that our data do not support significant differences between physicians and nurses. It could be that similar approaches develop during nursing and physician training. Alternatively, working closely together, it may be that any differences are only seen early in a nurse’s or a physician’s career, as their clinical experiences would lead them to develop similar intuitions later. These findings support the user-centered design idea of supporting varied user groups through separate modes of operations.

The strengths of our methods include the use of a clinical case based on real patients and the use of the CaST method and the verbal protocols that unveil the way the human operator is processing the information required for medication reconciliation. The larger sample of active clinicians, made possible by using a portable electronic medium, also added to the strength of this study. There are potential weaknesses and limitations to our study. First, like any other simulated study, we had to simplify the task, and as part of this simplification, we might have weakened clues that affect the mental model development. More importantly, because cognitive strategy is not necessarily in the participants’ consciousness, we can only infer how they select actions and use information. The limited demographic information that was collected about the participants also limits the types of conclusions we could draw from our results. Our conclusions are also limited by the fact that the study was conducted in one medical center, using one patient scenario.

Conclusion

Understanding the way clinicians think while performing a complicated task such as medication reconciliation opens new opportunities for user-centered design and improved patient safety. It provides health care information systems designers with knowledge they can translate into better tools that support clinicians when reconciling patients’ medications and medical conditions lists. It is an early first step toward better interface design that will support the users’ information integration processes. The study also highlights gaps that need to be addressed to improve patient safety. Based on our findings, we suggest that improved health care information systems should (1) allow flexibility in the way clinicians use information technology to organize medication and condition information and support multiple reconciliation strategies, (2) allow clinicians to toggle between medication and medical condition data and enable clinicians to link medical conditions with medications, (3) support adding comments and remarks near pieces of information that do not make sense until more information is available, and (4) allow clinicians to edit and re-sort information easily as new information becomes available.

Furthermore, these findings suggest that experience may be a factor in the cognitive strategy clinicians use. Our methods and results might be informative to develop learning strategies and to enhance recognition of problems such as missing information and improve education in this complex topic. Further research should seek to understand the effect that information layouts have on sensemaking and efficiency, how the reconciliation process is done by other professions (e.g., pharmacists), and the role experience and training have in building resilient strategies for varied professions.

Key Points

• Clinicians apply various cognitive strategies while reconciling medications and medical conditions.

• The way clinicians handle new information about the patient is affected by their experience, but not by their profession (physicians vs. nurses).

• Health care information systems for clinicians should support multiple cognitive strategies, and flexibility in organizing information.

Biography

Yuval Bitan received his PhD in industrial engineering and management in 2003 from Ben-Gurion University of the Negev, where he is currently a faculty member in the Department of Industrial Engineering and Management.

Yisrael Parmet received his PhD (2000) in statistics from Tel-Aviv University. Since 2002, he is a faculty member in the Department of Industrial Engineering and Management at the Ben-Gurion University of the Negev.

Geva Greenfield received his PhD (2010) in industrial engineering from Ben-Gurion University. He is currently with Imperial College London.

Shelly Teng is a bioengineering and biomedical engineering student at the Department of Bioengineering, University of Pennsylvania.

Richard I. Cook received his MD from the University of Cincinnati in 1986. He is currently with Ohio State University.

Mark E. Nunnally received his MD from the University of Washington in 1998. He is currently with New York University.