A user-centered evaluation of medication therapy management alerts for community pharmacists: Recommendations to improve usability and usefulness

Abstract

Background:

Community pharmacists provide comprehensive medication reviews (CMRs) through pharmacy contracts with medication therapy management (MTM) vendors. These CMRs are documented in the vendors’ web-based MTM software platforms, which often integrate alerts to assist pharmacists in the detection of medication therapy problems. Understanding pharmacists’ experiences with MTM alerts is critical to optimizing alert design for patient care.

Objectives:

The objectives of this study were to 1) assess the usability and usefulness of MTM alerts for MTM vendor-contracted community pharmacists and 2) generate recommendations for improving MTM alerts for use by community pharmacists.

Methods:

This was a convergent, parallel mixed-methods evaluation of data collected from 3 sources, with individual pharmacists contributing data to one or more sources: 1) community pharmacists’ submissions of observational data about MTM alerts encountered during routine MTM provision, 2) videos of naturalistic usability testing of MTM alerts, and 3) semi-structured interviews to elicit pharmacists’ perspectives on MTM alert usefulness and usability. MTM alert data submitted by pharmacists were summarized with descriptive statistics. Usability testing videos were analyzed to determine pharmacists’ time spent on MTM alerts and to identify negative usability incidents. Interview transcripts were analyzed using a hybrid approach of deductive and inductive codes to identify emergent themes. Triangulation of data (i.e., determination of convergence/divergence in findings across all data sources) occurred through investigator discussion and identified overarching findings pertaining to key MTM alert challenges. These resulted in actionable recommendations to improve MTM alerts for use by community pharmacists.

Results:

Collectively, two and four overarching key challenges pertaining to MTM alert usability and usefulness, respectively, were identified, resulting in 15 actionable recommendations for improving the design of MTM alerts from a user-centered perspective.

Conclusions:

Recommendations are expected to inform enhanced MTM alert designs that can improve pharmacist efficiency, patient and prescriber satisfaction with MTM, and patient outcomes.

Introduction

The Centers for Medicare and Medicaid’s requirement that Medicare Part D plans offer MTM services to certain beneficiaries has raised national attention to MTM’s potential to improve medication outcomes among older adults in the United States.¹ A key component of MTM is a comprehensive medication review (CMR); in a CMR, a healthcare professional interviews the patient and/or their caregiver(s) to conduct a thorough assessment of the patient’s current medications, medication needs, and medication-related outcomes, to identify medication therapy problems (MTPs) and intervene, as appropriate, with the patient and/or their prescribers.²

While many types of healthcare professionals provide MTM for Part D beneficiaries, pharmacists are the most common provider.³ Community pharmacists, specifically, provided approximately 22% of all CMRs delivered for Medicare Part D beneficiaries in 2014.⁴ Community pharmacists provide MTM to patients most commonly through pharmacies’ contracts with MTM vendors. Moreover, the number of patients receiving MTM from community pharmacists has likely increased as the percent of Part D plans utilizing community pharmacists for MTM through contracts between community pharmacies and MTM vendors has grown from about 26% in 2014 to about 65% in 2018.^3,5

MTM vendors typically require contracted community pharmacists to document and bill MTM services, including CMRs, in the vendor’s commercial, web-based MTM software platform. These platforms often include automated, electronic MTM alerts (e.g., warning of potential medication non-adherence) intended to assist the community pharmacist in the identification and resolution of MTPs for patients.

Scant literature exists describing alerts used in community pharmacy practice^6,7 and even less literature exists describing alerts specifically used during pharmacist-provided medication reviews, including CMRs for MTM.^8-14 With regards to alerts for pharmacist-provided medication reviews, previous research has explored the role of alerts vs. patient-provided information in pharmacist detection of MTPs,^8,9 the effect of new alerts on the identification of MTPs during medication reviews in the Netherlands,¹⁰ the development and testing of commercial medication review software to facilitate medication reviews in Australia,^11-13 and the utility of clinical decision support for telephonic MTM, with and without pharmacogenetic testing data.¹⁴ Notably, only the pharmacogenetic testing data alert research was conducted in the US. The US is quite different than the rest of the world in terms of MTM practice and policy because of the use of MTM vendor-owned documentation software platforms/alerts by community pharmacists, and information fragmentation across care settings. Moreover, much of this previous literature describing alerts for medication reviews was collected from and about the software systems,^8-14 rather than capturing the perspectives and experiences of the end-users who use the MTM software systems for care delivery.

Previously, the authors examined the role of Chronic Care Model elements^15,16 in MTM delivery, including clinical decision support, such as alerts.¹⁷ That work identified potential problems with MTM alerts, such as concerns with alerts falsely identifying MTPs.¹⁷ However, pharmacists’ experiences as end-users of alerts largely remain unknown. Recent literature reviews emphasize the importance of applying user-centered design principles in the development of medication-related alerts.^18-20 As stated in a 2018 review of medication-related alerts: “Capturing users’ experiences with medication-related [alerts] … is important if problems are to be exposed and refinements made to a system.“¹⁹ A recent paper summarizes ongoing challenges for MTM stakeholders, noting inconsistent evidence of positive impacts on patient health outcomes and challenges pertaining to pharmacist efficiency in MTM provision.²¹ Therefore, refining MTM alerts used by community pharmacists could be of value to stakeholders.

This study fills a gap in the literature through the conduct of a convergent, parallel mixed-methods²² investigation of pharmacist (i.e., end-user) perspectives and experiences with MTM alerts.

Objectives

The objectives of this study were to 1) assess the usability and usefulness of MTM alerts for MTM vendor-contracted community pharmacists and 2) generate recommendations for improving MTM alerts for use by community pharmacists. Findings are expected to inform enhancements to MTM alert design which could result in improved efficiency in community pharmacist MTM delivery, as well as better patient outcomes.

Methods

Conceptual frameworks

This study was informed by three complementary frameworks: 1) the Systems Engineering Initiative for Patient Safety (SEIPS) v. 2.0.,²³ 2) the MTP taxonomy framework described by Cipolle et al.,²⁴ and 3) Russ et al.‘s prescriber-alert interaction model, adapted for pharmacist-MTM alert interactions.²⁵ In the SEIPs 2.0 model, the work system includes “tools and technology,” which along with other factors, shape work processes and desired and undesirable patient outcomes.²³ For the purposes of the current study, MTM alerts were the “tools and technology” of interest. The Cipolle framework was applied to organize MTM alerts into the following 4 categories: 1) indication (e.g., medication is not medically warranted), 2) effectiveness (e.g., suboptimal dose), 3) safety (e.g., drug-drug interaction), or 4) adherence (e.g., the patient forgets the medication.)²⁴ Alert categorization is described in detail below. Finally, Russ’ model describes 9 factors that influence how prescribers perceive and respond to medication alerts. These factors are: 1) alert system logic, 2) alert system redundancy, 3) alert content, 4) alert display, 5) cognitive factors, 6) pharmaceutical knowledge, 7) medication management, 8) workflow, and 9) alert system reliability.²⁵ As described below, this framework was applied during the collection and analysis of interview and usability testing data, and triangulation of findings across data sources.

Setting and participants

This research was the second part of an overarching study that examined MTM alert design to identify recommendations for improvement. The first part conducted a heuristic evaluation of MTM alert screenshots to determine alert designs’ alignment with human factors principles, and the findings from that evaluation were recently published.²⁶ The second part, described herein, explored the experiences of pharmacists as end-users. This part of the study analyzed data from three sources, described below. The article herein is the only article for this second part of the overarching study that presents data describing the experiences of pharmacists with the MTM alerts.

As described for the first part of the study,²⁶ community pharmacists were recruited from 2 regional practice-based research networks (PBRNs): the Medication Safety Research Network of Indiana (Rx-SafeNet) and the Minnesota Pharmacy PBRN (MPPBRN).^27,28 They were eligible to participate in this study if they reported routine (i.e., 2 or more each week) CMR provision through contracts with one or both of two national MTM vendors which utilize web-based platforms with integrated MTM alerts. The goal was to recruit a total of 10–15 pharmacists. This number was expected, from investigators’ experience, to be sufficient for the detection of usability errors during usability testing sessions²⁹ and in the identification of themes from semi-structured interviews, described below.

Data collection

This was a convergent, parallel mixed-methods evaluation²² of quantitative and qualitative data collected from 3 sources. Individual pharmacists contributed data to one or more sources: 1) community pharmacist submission of quantitative observational data about MTM alerts generated by MTM vendors and encountered during routine MTM provision (all pharmacists were asked to participate), 2) naturalistic usability testing of MTM alerts to produce quantitative and qualitative data about the user experience (a sub-set of pharmacists were asked to participate), and 3) semi-structured interviews to elicit qualitative data for pharmacist perspectives on MTM alert usefulness and usability (all pharmacists were asked to participate). Study procedures were approved by the Purdue University Institutional Review Board.

MTM alert data submission

Pharmacists submitted MTM alert data to the study team as previously described for the first part of the study.²⁶ For this part of the study, data collected pertained to pharmacists’ experiences with MTM alerts as part of the process of identifying and resolving MTPs during a CMR. Specifically, each pharmacist was asked to submit alert data for 3 patients, making an effort to include patients with alerts warning about different MTP categories from the Cipolle framework.²⁴ The primary sampling goal (as described previously²⁶) sought to obtain a representative cross-section of MTM alerts for both national vendors across all MTP categories. As data collection proceeded, the study team provided guidance to the pharmacists on which alert categories to prioritize for submission based upon what alerts had been received up to that point.

Data were submitted by pharmacists in REDCap, which is a secure web-based application designed for the collection of research data.³⁰ Investigators reviewed MTM alert data to ensure completeness of the pharmacists’ submission and request clarifying information from pharmacists when needed. During this time, a pharmacist investigator categorized the submitted alerts using the Cipolle framework as a guide, specifically categorizing MTM alerts as warning about one of 7 MTPs within one of the framework’s 4 MTP categories. Indication-related MTPs included the following: 1) unnecessary medication therapy and 2) needs additional medication therapy; effectiveness-related MTPs included 3) ineffective medication/needs different medication and 4) dose too low; safety-related MTPs included 5) dose too high and 6) adverse drug reaction; and adherence-related MTPs included 7) any adherence-related MTP.²⁴

MTM alert data were primarily quantitative and the specific following data elements were submitted by pharmacists: pharmacist demographics (age, sex, race/ethnicity, education/training, employment information, and MTM experience); patient demographics (age, sex, race/ethnicity, insurance); information about the MTM encounter (MTM vendor, initial/follow-up CMR, mode of delivery [e.g., face to face], status of CMR [e.g., scheduled], roles of other pharmacy staff in CMR provision, number of MTM alerts generated for patient); information about each MTM alert submitted (screenshot images of the alert, timing of alert’s appearance, assessment of whether [yes/no and why] MTM alert appeared at the “ideal” time in MTM workflow); pharmacist response to each MTM alert (number and types of MTPs identified with the alert, actions taken with patients and prescribers in response to MTPs); MTPs identified without the alert; and ratings of pharmacists’ satisfaction with the MTM vendor platforms, assessed via the system usability scale (SUS) which contains 10 Likert-type scale items (http://cui.unige.ch/isi/icle-wiki/_media/ipm:test-suschapt.pdf) with response options ranging from 1 = strongly disagree to 5 = strongly agree,³¹ completed for each unique vendor for which alert data was submitted. Data were collected from April 2017 to March 2018.

Naturalistic usability testing

Naturalistic usability testing^32-34 was conducted, meaning that the usability evaluation was conducted in the natural work environment while pharmacists used MTM vendor software at their pharmacy for their own patients. Usability testing focused specifically on the pharmacist-MTM alert interactions, and was conducted with a subset of pharmacists who participated in the overall study. A purposeful subset of pharmacists was invited for usability testing, targeting a range of practice settings (e.g., geographic region covered by PBRN) and characteristics (e.g., gender, MTM experience). The sample size goal was to recruit 5 pharmacists for usability testing, because, based upon usability literature, five participants can uncover 55–99% of major usability problems²⁹ and our usability goals were more descriptive in nature, intended to be triangulated with the other data sources, rather than fully comprehensive of all possible MTM alert usability problems.

Usability testing sessions were conducted remotely³⁵ between October 2017 and April 2018 with pharmacists, using Webex conferencing software³⁶ and were facilitated by a moderator experienced in user-centered design and usability testing. To maintain consistency, the moderator utilized a standardized script and refined this script via pilot testing prior to data collection (Appendix A). As part of the script, pharmacists received brief instruction, including a standardized training video,³⁷ on how to think aloud³² in order to verbalize their thought process as they interacted with the MTM alerts generated for CMR-eligible patients.

Since we conducted usability testing in the natural clinical environment, patients’ protected health information was visible and inherently recorded as part of the usability videos. Thus, patients provided written consent and HIPAA authorization for their data to be used in this research. Patients were eligible if 1) they were 18 years of age or older and 2) were eligible for a CMR through one of the MTM vendors used by a participating study pharmacist. Patients’ demographics were not collected because these sessions were conducted to examine MTM alert usability from a pharmacist perspective.

Following methods from our prior research,³⁸ the moderator used Morae usability software [Techsmith, v 2.0, Okemos, MI]³⁹ to record each usability session. Morae is commonly used to capture usability testing data.^38,40 This software captures participating pharmacists’ computer screen actions, (shared over Webex), pharmacists’ facial expressions, and voice as they engaged in ‘think aloud’ and worked on MTM related activities and responded to alerts. These video recordings, captured by Morae, contained both quantitative and qualitative elements and were the data source for subsequent usability analysis.

Semi-structured interviews with pharmacists

All recruited community pharmacists were invited to participate in a one-on-one, audio-recorded, semi-structured telephone interview to elicit their perspectives on MTM alert usability and usefulness. Interviews were conducted between November 2017 and January 2018. An interview guide (Appendix B) was developed with questions designed to elicit qualitative data pertaining to each factor of the modified Russ prescriber-alert interaction model.²⁵ For example, one question related to workflow was, “In what ways, if any, does the MTM system affect how efficiently you complete MTM cases?” An example question related to alert system reliability was, “To what extent does the MTM system’s prioritization of alerts agree with how you would assign priority?” The guide was pilot tested with 2 pharmacists, audio-recordings of these pilot tests were reviewed and discussed by 3 investigators, and minor revisions for clarity were made prior to commencing study interviews. One investigator, a pharmacy fellow with formal training in qualitative data collection and experience in MTM delivery, conducted all interviews. Interview audio-recordings were transcribed verbatim by a professional transcriptionist and transcriptions were reviewed by the pharmacy fellow for accuracy prior to qualitative analysis.

Data analysis

Quantitative and qualitative data from each source were first analyzed independently, concurrently, as described in the below sections.

MTM alert data submission

Prior to analysis, the pharmacy fellow verified the MTM alert categorization (i.e., specific MTP warned about²⁴) made by another pharmacist investigator during the initial review. The MTM alert data (pharmacist/patient demographics, alerts submitted, MTPs identified, mean number of pharmacist actions taken with patients and prescribers in response to MTPs by MTP type, SUS scores) were then summarized using descriptive statistics, computed in SPSS [IBM SPSS v 24.0, Armonk, NY.]⁴¹ SUS scores were computed by scaling, in a standardized fashion, all response values from 0 to 4. These values were then summed and multiplied by 2.5 to create a possible score range of 0–100.³¹ An overall total mean score was computed from SUS ratings across all pharmacists’ responses.

In addition, “congruency” between the MTM alerts that the pharmacist submitted vs. any MTPs identified by the pharmacists in response to the MTM alert was assessed. MTM alerts were defined as congruent if the actual MTP identified by the pharmacist with the alert was the same as the MTP warned about (i.e., one of seven specific MTPs) by the MTM alert. Congruency was evaluated in order to identify false alarms (i.e., alerts firing with no MTPs identified in response) and situations where a different MTP was identified from that warned about by the MTM alert.

Naturalistic usability testing

Analysts reviewed video recordings in Morae to assess MTM alert usability. Morae software provides tools for analysts to mark their usability findings on the video recording, enter manual notes, and more easily calculate metrics (e.g., time to complete a specific task).⁴⁰ In our analysis, in order to note the time that pharmacists spent on each MTM alert, the “start time” was defined as the time when the pharmacists first selected the link to access the MTM alert and “stop time” was defined as the time when the pharmacist resolved the alert, and selects “OK” to acknowledge as such, OR when the participant was unable to resolve the alert and intentionally or unintentionally closed the window so that the MTM alert disappeared. Time data were recorded in Microsoft Excel [2016, Redmond, WA].⁴² Usability sessions were then analyzed using descriptive statistics to summarize the number of MTM alerts encountered by pharmacists, alerts’ resolution status (not reviewed or addressed; reviewed but not addressed; left pending; resolved), time spent on MTM alerts, and the number and type of negative usability incidents encountered (e.g., alert did not support or adequately integrate with the pharmacist’s preferred workflow). Usability incident data were analyzed by 2 usability specialists and their analyses were informed by comments from 2 pharmacist investigators who also reviewed and commented on all of the usability recordings. Facial expressions alone were not coded; rather, they provided non-verbal information to help inform analysts’ assessment of usability incidents (e.g, further confirming usability frustrations). Usability incidents were categorized according to factors described in the Russ prescriber-alert interaction model.²⁵

Semi-structured interviews with pharmacists

Interviews were analyzed by 2 pharmacist researchers with formal MTM and qualitative analysis training using a hybrid deductive and inductive approach to code development.⁴³ Specifically, a starting list of broad, conceptual codes was developed to sort text into the 9 factors of the modified Russ prescriber-alert interaction model.²⁵ Sub-codes were then developed inductively in order to identify intra- and inter-factor themes. Each investigator independently reviewed each transcript line-by-line to complete coding in the qualitative data analysis software, MAXQDA (VERBI, v. 12, Berlin, Germany).⁴⁴ Coding discrepancies were then compared and resolved through consensus, with a codebook and audit trail maintained throughout the process to track coding decisions.⁴⁵ A third pharmacist investigator trained in MTM and qualitative analysis participated in codebook development, independently reviewed half of the transcripts during coding, and discussed impressions with the 2 analysts. Final themes were identified through discussion.

Data triangulation

Each of the above analyses produced data summaries describing the findings from each of the 3 sources of data. These individual findings were then discussed by 6 investigators with expertise spanning MTM, nursing, health services research, human factors engineering, and alert design. The investigators met for approximately 12 h to collectively review and triangulate (i.e., identify areas of convergence/divergence across data sources) all analyses, identify overarching findings pertaining to MTM alert challenges, and develop recommendations for MTM alerts from an end-user perspective.

Results

Participants

A total of 10 pharmacists were successfully recruited and consented to participate, representing 8 community pharmacies. Of these, 9 pharmacists submitted demographic and MTM alert data in REDCap, 5 participated in usability testing, and 8 completed a semi-structured interview. All participating pharmacists were non-Hispanic Caucasians holding a PharmD, and 67% were female. About half (4, 44.4%) reported completion of a post-graduate year one residency program; 8 (88.9%) had completed at least one certificate program. Six participants practiced in an independent community pharmacy and 3 practiced in a chain pharmacy (4 or more locations.) They reported a mean (SD) of 4.2 (3.4) years of experience providing CMRs for patients and completing 8.2 (7.4) CMRs per month. For all results below reporting a mean, data are reported as mean (SD).

MTM alert data

Pharmacists submitted data for a total of 77 MTM alerts generated by 2 MTM vendors’ software platforms, representing 28 patients eligible for a CMR. On average, each pharmacist submitted 2.8 (1.4) MTM alerts to the study team per patient. Patients for whom alert data were submitted were 67 (17.7) years old on average with varying types of MTM payers: 17 (60.7%) Medicare Part D, 8 (28.6%) Medicaid, and 3 (10.7%) commercial insurance. Pharmacists indicated that most (89.3%) patients were receiving their first CMR. Pharmacists reported that 69 alerts (89.6%) appeared at the ideal time in their MTM workflow.

Submitted MTM alerts are summarized in Fig. 1. Of the 77 submitted alerts, 69 could be categorized using the Cipolle framework,²⁴ resulting in the following categorizations: 24 (35%) targeted indication-related MTPs; 5 (7%) targeting effectiveness-related MTPs; 20 (29%) targeting safety-related MTPs; and 20 (29%) targeting adherence-related MTPs. Specific alerts targeting indication-related MTPs included 13 related to “unnecessary medication therapy” and 11 related to “needs additional medication therapy.” Alerts targeting effectiveness-related MTPs were categorized specifically as “needs different drug product” (4 alerts) and “dose too low” (1 alert). Specific alerts targeting safety-related MTPs included 19 related to “adverse drug reaction” and 1 related to “dose too high.” Finally, 20 alerts were categorized as targeting “non-adherence” as an MTP. The other 8 alerts submitted were categorized as targeting cost MTPs/disease state management (n = 6) or did not include sufficient information for categorization (n = 2).

Fig. 1.

MTM alert data collection and MTPs identified.

About half (49.3%) of alerts resulted in the pharmacist identifying a MTP that the alert warned about. Pharmacists documented 3.2 (2.7) total MTPs per patient, with 1.7 (1.1) MTPs identified with the assistance of an MTM alert and 1.6 (2.1) MTPs identified without an alert. MTPs identified and actions taken are summarized in Table 1. All of the alerts warning about effectiveness-related MTPs resulted in the identification of an MTP, but this was not always an effectiveness MTP. Alerts warning about adherence-related MTPs was the only alert category without any incongruent MTPs (i.e., an MTP different than that warned about by the alert) identified. Most (51.7%) of the false alarm alerts were medication indication-related, with many warning of unnecessary or “duplicate” medication therapy, e.g., the use of multiple medications for hypertension. A total of 46 MTPs were identified by pharmacists without an alert. Most (58.6%) of these MTPs related to a need for patient immunizations or problems with patients’ medication adherence. With regards to actions taken for MTPs identified, a total of 162 actions were taken with patients and 69 with prescribers. Differences in the mean number of actions taken by pharmacists with patients and prescribers for MTPs identified with and without MTM alerts, by MTP type, are presented in Table 1. On average, MTPs resulted in more actions taken with patients than with prescribers. Similar patterns for actions were observed for MTPs identified both with or without an MTM alert.

Table 1.

Summary of medication therapy problems (MTPs) identified and mean number of associated actions taken by pharmacists with patients and prescribers by MTP type.

Types of MTPs Identified (total N = 86)a	MTPs congruent with alert warning (n = 34)a	MTPs different from alert warning (n = 6)a	Alerts with no MTPs (false alarms) (n = 29 alerts)b	MTPs identified without an MTM alert (n = 46)a	Actions taken with patients per MTPs identified (n = 162 actions)c mean (SD)		Actions taken with prescribers per MTPs identified (n = 69 actions)c mean (SD)
	n(%)	n(%)	n(%)	n(%)	With alerts n = 80c	Without alerts n = 82c	With alerts n = 31c	Without alerts n = 38c
Unnecessary medication therapy	2 (6)	0 (0)	11 (37.9)	4 (8.7)	1.5 (1.9)	1.8 (1.0)	0.8 (0.5)	0.8 (0.5)
Needs additional medication therapy	6 (18)	2 (33.3)	4 (13.8)	16 (34.8)	1.3 (0.8)	2.6 (2.0)	0.9 (0.4)	0.6 (0.6)
Needs different medication product	1 (3)	2 (33.3)	0 (0)	3 (6.5)	0.5 (0.6)	0.7 (0.6)	0	1.3 (0.6)
Dose too low	0 (0)	1 (16.7)	0 (0)	4 (8.7)	0	2.3 (1.3)	0	1.3 (0.5)
Adverse drug reaction	10 (29)	0 (0)	10 (34.5)	2 (4.3)	1.9 (0.7)	0	1.0 (0.6)	0.5 (0.7)
Dose too high	0 (0)	1 (16.7)	0 (0)	4 (8.7)	0	1.5 (0.6)	0	1.3 (0.5)
Non-adherence	15 (44)	0 (0)	4 (13.8)	13 (28.3)	2.1 (0.8)	1.4 (1.5)	0.3 (0.4)	1.1 (0.6)

^aNumber refers to the number of MTPs identified. Indication-related MTPs include: unnecessary medication therapy and needs additional medication therapy; Effectiveness-related MTPs include: needs different medication product and dose too low; Safety-related MTPs include: adverse drug reaction and dose too high; Adherence-related MTPs include: non-adherence.

^bNumber refers to the number of MTM alerts submitted.

^cNumber refers to the number of actions taken by pharmacist in response to the MTPs identified.

SUS scores

SUS scores across all pharmacists’ and MTM vendors was 70.8 (9.8) out of a maximum possible score of 100. According to Bangor et al., this result indicates an overall “OK to good” assessment⁴⁶ by the pharmacists regarding their satisfaction with the usability of the MTM vendor systems. SUS items with the lowest mean scores were “I found the system unnecessarily complex” (reverse coded) with a mean score of 1.7 (1.1) and “I think I would like to use this system frequently” with a mean score of 2.2 (1.3).

Naturalistic usability testing

After excluding usability sessions from two pharmacists where no MTM alerts were generated for the CMR being completed, a total of 7 usability testing sessions, each lasting approximately 34 min on average, were completed with three pharmacists. Each pharmacist participated in 2–3 usability sessions. Across the 7 sessions, two “use cases” for MTM alerts were observed: 1) pharmacists preparing for the patient’s medication therapy review and 2) after the medication therapy review, when pharmacists were documenting/billing the CMR. Across the 7 sessions, a total of 13 MTM alerts were encountered by pharmacists with a median (range) of 2 (1–4) MTM alerts encountered per session. Six (46.1%) alerts were resolved by the pharmacist by the end of the session. The remaining MTM alerts were either not addressed by the pharmacists or were unable to be fully resolved by the end of the session for various reasons, such as the pharmacist needing to follow up with a prescriber for more information or needed to discuss the alert content with the patient. Pharmacists spent a median of 2 min and 22 s per alert.

A total of 39 negative usability incidents were identified. The most common incidents pertained to challenges associated with workflow (11 incidents across 6 sessions and 3 pharmacists), alert display (8 incidents across 4 sessions and 3 pharmacists), and alert system reliability (faulty guidance; 6 incidents across 5 sessions and 2 pharmacists.) With regards to workflow, challenges were noted wherein the alert did not support the pharmacists’ preferred MTM workflow (e.g., the alert required clinical resolution on a screen different from which the pharmacist would have chosen to use.) During one session, a pharmacist stated, “I would like to … look at it [alert information] more closely, but it’s not letting me do it from here.” Display problems included challenges with drop-down menu options not matching the clinical circumstances of the CMR. As one pharmacist stated, “None of these [drop-down options on the MTM alert] is really appropriate.” Challenges with alert system reliability were observed when the guidance provided by alerts was perceived by pharmacists as not aligned with actual patient care needs. For example, during one session a pharmacist stated, “I know she [the patient] has already tried both of the [MTM alert’s] recommended medications, and they have not worked for her.”

Semi-structured interviews with pharmacists and triangulation across data sources

Pharmacist interviews each lasted approximately 30–60 min and provided rich end-user insights into MTM alert usability and usefulness. Main themes and representative quotations from interviews, and triangulation (i.e., convergence/divergence) of findings across MTM alert and usability testing data sources, are summarized in Table 2. Collectively, these triangulated findings revealed two overarching key challenges pertaining to MTM alert usability and four key challenges with MTM alert usefulness, informing a total of 15 actionable recommendations (Table 3) for improving the design of MTM alerts from a user-centered perspective.

Table 2.

Major findings from pharmacists’ semi-structured interviews triangulated across medication therapy management alert and naturalistic usability testing data sources. Factors in the left column were derived from the Russ prescriber-alert model.²³

			Triangulation Across Data Sources C= Convergence D = Divergence U= Unknown/not evaluated through data collected
Factor(s)	Interview Findings (n = 9)a	Representative Quotations from Interviews	MTM alert data submitted by pharmacists (n = 9)	Naturalistic usability testing (n = 3)
Logic/Reliability	Alerts can help pharmacists identify patients in need of intervention (i.e., with MTPs). However, many alerts are not applicable for patients’ clinical context, which reduces alerts’ reliability and pharmacists’ trust in alerts.	“Same thing with the adherence alerts … 75% of the time they are applicable, but 25% of the time they are not, because the patient was hospitalized or they paid cash for short fill so we could synch their medicine or they got a sample so they didn’t pick it up for awhile. Those things the insurance company doesn’t know and I don’t know how they would, but that does make the alert … not applicable a lot of the time.” “With certain medications that you really would only, you’d only want your people to be using them on an as needed basis … like for sleep for example, that you wouldn’t want someone to have to be using those every night, and … it [MTM alert] will show up as like a … something to get done, which then gets reported to the insurance that someone isn’t adherent, but it is really something that you wouldn’t want them to be super adherent to.	C	C
Logic/Reliability/Content	Lack of reliability has negative effects on pharmacists’ relationships with patients and prescribers. The clinical information provided by MTM alerts lacks patient-specificity or is insufficient for optimal patient care Oftentimes, the more experience pharmacists have providing MTM, the less useful they perceive alerts.	“… so you can’t use the [anticoagulant] that they want you to switch to for people with certain heart valves, if they are not indicated. So it [MTM alert] is kind of bad advice in that regard. I guess I would kind of feel bad for the pharmacy who blindly sends that request off to the doctor and the doctor says, no, that is not indicated. They probably would be unhappy about having their time be wasted …” “Their [alerting function of the MTM software] system is based totally off of medication history … But … It almost seems like these systems should to be able to negotiate a way to get lab data also available to pharmacist.” “I think that again it is kind of [a clinical] decision. I think sometimes it depends on who is doing the work, so sometimes if you have a pharmacy intern or something like that, maybe less knowledgeable about medications or kind of once in awhile when these [MTM] alerts come up, we can be talking to patients about things that mainly aren’t as big of a deal in the day-to-day as it is kind of in a textbook.”	U	C
Cognitive factors	Pharmacists express ethical concerns with MTM alerts, noting familiarity with the logic used to generate alerts but concern that alerts are not always in patients’ best interest	“it is not a better medication for the patient, it is not a worse medication either, but it is more about saving the insurance company money versus helping the patient.”	U	Cb
Redundancy	Some MTM alerts duplicate other pharmacy systems and usual pharmacy practice (e.g., alerts generated in dispensing systems) Some MTM alerts are displayed repeatedly, even after being addressed by the pharmacist	“… So, whenever I check any medication that we dispense, it [dispensing system] will pop up with a clinical alert, like I mentioned earlier, if there is anything in the system that looks fishy.” “A lot [of patients] don’t want to switch medications [even though they are recommended by the MTM alert]. I have had a bunch of those [alerts] where I have said, “No, patient is not interested or patient is unwilling, they are doing well on what they are on … I have denied those, they pop back up in maybe two to three months later. I’m like, well I am not going to bug the patient about it and ask them about it again.”	U	Cc
Pharmaceutical knowledge	Patients are the primary information source for pharmacists in the identification of MTPs, with many MTPs identified without MTM alerts	“Definitely patients I mean I think that is number 1. A lot of times like I said we do ask patients to sign a release of information so that we can access their medical records and that is helpful …. ”	C	U
Workflow/Medication management	Information fragmentation (i.e., lack of system integration and the need to access patient information from multiple disparate documentation systems) is required to resolve alerts • Workflow mismatch between MTM alert and pharmacists’ other responsibilities (e.g., dispensing)	“So I think transitioning between the different systems kind of impedes the workflow there.” “… Like in my situation, it would be nice if it [MTM software] was integrated with the actual pharmacy [dispensing] system that we use, so that whenever you are filling, an alert would come up right then rather than having the pharmacist call afterwards, because most of the time if we are working on something, it means that either the patient is there or they are coming in, and that is the best time to get this done …”	Dd	C
Display	Documenting alert resolution is challenging because alert drop down options do not always match the clinical context Documenting MTPs that pharmacists identify without alerts is often impossible	“… And then I have to choose from the radios that they have [on the MTM alerts], and the radios are usually like ‘the patient is in a long term care facility.’ Well, that’s not true. ‘Patient has different directions from the doctor.’ That is not necessarily true either. Or the doctor changed them or whatever. So none of the radios really fit with that, so then I just have to pick ‘other’ and then I explain.” “if you uncover something on your own, finding where to input that [into the MTM software] and document that appropriately can kind of be more of a struggle.”	U	C

^a:n refers to the number of pharmacists from which data for the specific source (interviews, MTM alerts, usability testing) was submitted.

^b:An additional ethics concern, “gaming” was noted during usability testing (i.e., starting an MTM case but leaving it as pending in order to prevent others from accessing the case).

^c:Redundant actions required for alert resolution (e.g., having to enter “educated patient” three times to resolve one alert) was noted during usability testing.

^d:During MTM alert data submission, pharmacists indicated that 89.6% of MTM alerts appeared at the ideal time in their MTM workflow.

Table 3.

Summary of key challenges with MTM alert usability and usefulness identified through triangulation and actionable recommendations for improving MTM alert design.

Key Challenges with MTM Alertsa	Actionable Recommendations for MTM Vendor Systems
Usability Challenges
Alert display does not always support pharmacists’ preferred MTM workflow and sometimes requires redundant data entry.	Ensure MTM alerts are generated at times which support different use cases (e.g., for pharmacists first accessing the MTM vendor systems after the CMR is complete, because they primarily document services in another system) Enable flexibility in how (e.g., from which screens) pharmacists can access alert information and initiate documentation Remove duplicative data entry fields (e.g., the need to re-type “educated patient” in response to an alert) Improve, reduce, and/or remove drop-down menus/options to better align data entry options with the full range of clinical contexts encountered during a CMR (e.g., a pharmacist’s inability to provide medication synchronization for a patient who receives medications at another pharmacy.)
A lack of system integration between MTM vendors and other pharmacy systems interrupts pharmacist workflow	MTM systems should fully integrate with pharmacy dispensing systems and electronic medical records
Usefulness Challenges
Redundant alerts are pervasive and reduce pharmacist efficiency	Enable systems to recognize pharmacists’ prior actions taken on alerts and generate/suppress alerts accordingly Ensure that MTM alerts are unique from alerts generated for the same patients during prescription fulfillment/drug utilization review
MTM alerts often falsely identify MTPs, many MTPs are identified without alerts, and usefulness of alerts varied with pharmacist experience	Examine alert sensitivity and specificity and make adjustments (e.g., re-word or remove alerts) accordingly to minimize false positives and negatives. Consider the development of new MTM alerts, particularly for immunizations and a re-design of existing alerts (i.e., those targeting non-adherence) to improve alert usefulness Except for safety alerts, enable users to suppress MTM alerts, by alert type, user group (e.g., a newly hired pharmacist vs. experienced pharmacist), and prevalence of MTP documentation without alerts (e.g., “turn on” alerts for MTPs currently found without alerts to support consistent and comprehensive MTP detection across users.) Update alert logic to ensure alignment with contemporary clinical practices (e.g. remove alerts that warn about the use of multiple medications for hypertension)
Alerts are perceived as more highly influenced by payer goals, rather than patients’ clinical and safety needs	Form advisory groups of patients, pharmacists, and prescribers to facilitate participatory design of MTM alerts that are informed by MTM stakeholders For alerts targeting MTPs with lower clinical priority (i.e., not likely to pose an imminent safety concern), enable users to select which alerts are generated or suppressed for their patient population
MTM alerts contain too little patient-specific information, positioning pharmacists to rely on patients for information which can be problematic	Re-design MTM alerts to provide pharmacists with more patient-specific information to inform decision-making, such as recent laboratory data Policies should be developed by CMS and other payers that incentivize and reward models for MTM delivery which utilize meaningful health information exchange for CMR completion with bidirectional data exchange between prescription dispensing systems, MTM vendor software systems, and electronic medical records

^a:Identified through triangulation of all three data sources (MTM alert submission, semi-structured interviews, usability testing).

Discussion

This research builds on the limited existing literature describing alerts for pharmacist-provided medication reviews^8-14 and provides insight on understudied contexts of medication review practice and policy, namely the provision of MTM by U.S. community pharmacists contracted with MTM vendors. There are many strengths of this research. To the authors’ knowledge, this is the first user-centered evaluation of MTM alerts used by community pharmacists. Our convergent, parallel mixed-methods design provided comprehensive insights into the experiences of community pharmacists with MTM alerts that we believe ultimately revealed more information about alert usability and usefulness than one method would have. The application of the Russ model also lends confidence that most, or all, important components of the pharmacist-alert experience were probed during interviews and usability testing. The final triangulated analyses described herein identified several overarching key challenges with MTM alert usability and usefulness; these challenges informed a total of 15 actionable recommendations (Table 3) that are anticipated to be useful to many MTM stakeholders, such as MTM vendors and payers, to improve pharmacist efficiency, patient and prescriber satisfaction with MTM, and patient outcomes. Future research should be conducted to examine these MTM outcomes pre- and post-incorporation of these recommendations into MTM alert design.

Many of these recommendations for MTM alerts align with results from Marcilly et al., ‘s 2015 review of usability “flaws” associated with medication-related alerts used in hospitals and primary care⁴⁷ and these similarities are noted. Specifically, Marcilly et al. identified usability challenges with suboptimal alert timing (recommendation 1), workflow concerns (recommendations 1, 2, and 4), redundancies in alert appearances and data entry (recommendations 3, 6, and 7), challenges with alerts falsely identifying MTPs (recommendations 8 and 11), system integration issues (recommendation 5), and a lack of patient-specific information (recommendations 14 and 15)⁴⁷. These overlapping findings are especially notable as the Marcilly review excluded papers applying a user-centered design, specifically those describing perceived usability or “feelings/opinions.“⁴⁷ Moreover, the current findings emphasize that MTM alert developers should incorporate human factors principles into alerts’ design.³⁸ The findings described herein identify specific ways in which recommendations from prior alert literature apply to the design of MTM alerts and also contribute novel ideas, particularly for optimizing MTM alert usefulness (specifically recommendations 8–15), as described below.

Although pharmacists in our study reported that MTM alerts were generally helpful and often assisted in MTP identification, opportunities were identified to improve MTM alert usefulness. For example, over 40% of alerts resulted in no actual MTPs identified and about the same number of MTPs were identified both with and without alerts. Moreover, pharmacists’ actions in this study did not appear to differ whether a MTP was identified with or without the assistance of a MTM alert. Recommendations 8 and 11 are made in an effort to reduce the likelihood of alerts falsely identifying MTPs.

Similar results suggesting medication-related alerts’ limited usefulness were found by Verdoon et al.¹⁰ They analyzed MTPs identified by Dutch pharmacists completing medication reviews pre- and post-implementation of alerts built into the medication review software. While pharmacists identified slightly more MTPs on average per patient post-implementation of the alert system (3.6 vs. 3.2, p < 0.01), the majority (59%) of MTPs were identified by pharmacists without assistance from the newly implemented alerts. Further, pharmacists resolved MTPs less often if they were identified via an alert (as opposed to MTPs identified without the assistance of an alert) and the overall resolution rate of all MTPs was lower after alerts were implemented.¹⁰ In addition, Kwint et al. examined MTPs identified by Dutch community pharmacists and evaluated whether MTPs were identified using clinical records or patient interviews. They determined that 27% of MTPs were identified during patient interviews and that these MTPs were more likely to result in a recommendation for a medication change.⁹ While these Dutch findings point to several alert limitations, in contrast, a team in Australia has described the success of a commercial decision support system for medication reviews that uses a knowledge-based design wherein information inputs from the users “teaches” the system to better identify MTPs.^11,12 An evaluation of the software found that it identified more MTPs than pharmacists.¹³ Notably, most MTPs identified by pharmacists (and not by the system) were related to medication adherence. Most recently, Kim et al. found that while the average number of MTPs identified during telephonic MTM was about the same when a clinical decision support tool was used with or without the incorporation of pharmacogenetic testing data, the latter helped pharmacists to identify more serious MTPs.¹⁴

In the current evaluation, most MTPs identified without alerts related to non-adherence (which aligns with the Australian team’s findings¹³) or a need for immunizations. With regards to adherence alerts, it is notable that both “false alarms” (false positives) and “false negatives” (i.e., MTPs found when no alert was generated) were documented. Similarly, Witry et al. studied community pharmacist non-adherence notes (either manually entered or acted on in response to an alert generated based proportion of days covered data) and found that only 2.2% of notes identified a specific adherence-related MTP. The majority of adherence alerts in their study were false alarms⁴⁸. Therefore, with regards to usefulness, the current design of MTM adherence alerts might not be optimal and needs further refinements and research (recommendations 8 and 9.)

With regards to immunizations, given that this evaluation included (on average) pharmacists with moderate MTM experience, more novice pharmacists might benefit if MTM vendors developed new alerts to assist with identifying immunization needs (recommendation 9). An ongoing evaluation is examining the implementation of an alert for the recombinant zoster vaccine in community pharmacy dispensing workflow to promote second dose receipt.⁴⁹ Similar alerts could be designed for MTM workflow.

Some of the findings of this evaluation aligns with the study team’s prior findings for MTM alerts: First, that users’ level of experience delivering MTM influences their perceived usefulness of MTM alerts (Table 2).¹⁷ The study team has conducted some additional preliminary work, interviewing small samples of community pharmacists and community pharmacy residents but did not identify notable differences in their decision-making processes; however, user-MTM alert interactions were not observed and over half of the pharmacists interviewed were new practitioners with less than 10 years of experience making them demographically similar in some ways to the residents.⁵⁰ Similarly, when compared to pharmacists with many years of experience providing medication reviews, Kwint et al. found that pharmacists inexperienced with medication reviews identified fewer MTPs; however, pharmacists with less experience identified a higher proportion of MTPs that were considered clinically relevant compared with their more seasoned peers.⁵¹ As such, more research is needed to fully elucidate how clinical decision making in MTM differs across users of varying experience levels and alerts should be customizable to the decision-making needs of different user groups, e.g., novices vs. experts (recommendation 10).

Secondly, the study team’s prior work found that users generally believe that MTM alerts are primarily designed to meet payers’ priorities (e.g., adherence alerts to drive performance on CMS Stars measures) and not necessarily patients’ needs.¹⁷ This was echoed in the current evaluation. Recommendations 12 and 13 are aimed at addressing these concerns.

Finally, this work confirms and extends the study team’s prior findings regarding the need for improved health information exchange (HIE) in the context of MTM.^17,50,52 Alerts should be designed so patient-specific information is accessible to community pharmacists (recommendations 14 and 15). Previous literature shows that community pharmacists regularly rely on patients as their primary source of health information.⁵⁰ However, while recent studies suggest that cognitively functional adults are fairly reliable in accurately reporting previous diagnoses, reports of more detailed medical and family history are less reliable and open to bias, with authors’ conclusions reinforcing the importance of utilizing medical records in conjunction with patients’ reports to resolve discrepancies and improve medical care.^53-56 To improve MTM alerts’ relevance and usefulness, patients’ specific health history (i.e. labs, care coordination notes, allergies, active and past medication lists) need to be available to pharmacists. Without access to patients’ health information, pharmacists can neither confirm nor deny the alert’s validity, interpret the alert’s relevance (i.e., MTP identification) nor optimally facilitate MTPs’ resolution. International literature’s findings align with this recommendation, as Kwint et al. determined that the extent of pharmacist-physician collaboration, such as shared medical records, is associated with an increased implementation rate of pharmacist recommendations.⁵⁷ Similarly, Mulder-Wildermors et al. interviewed a sample of Dutch community pharmacists about barriers and facilitators to implementing drug therapy changes suggested by alerts generated during prescription processing and identified a lack of patient-specific information in the alerts as a barrier.⁵⁸

There are limitations to this evaluation. First, pharmacists did not necessarily submit data for all MTM alerts generated for the CMRs, and some alerts (n = 8) submitted could not be evaluated for congruency with MTPs identified either because the alert was not targeting a Cipolle-defined MTP²⁴ or because insufficient information was submitted by the study participant. It is also possible that pharmacists were inconsistent with how they categorized MTPs identified. The study team attempted to address this through training and review/follow-up on submitted data. Moreover, alerts were categorized by 2 residency-trained pharmacists, but it is possible that other pharmacists or clinicians might have categorized the alerts differently. No data on MTP resolution were collected, so only actions, as self-reported by pharmacists, that were taken with patients and prescribers in response to alerts were summarized. In addition, data from 2 of the pharmacists completing usability testing were excluded from the evaluation and usability testing was only completed with one MTM software vendor platform due to MTM case availability at the time sessions were completed. Further, usability testing sessions only included a small, naturally-occurring sample of 13 MTM alerts from clinical practice whereas MTM vendor software systems generate many other MTM alerts to community pharmacists. Therefore, it is possible that more usability challenges would have been identified with additional testing and that some findings might not be transferable to all MTM alerts or MTM vendor software systems. Finally, it is important to highlight that the nature of this study design resulted in small samples for analyses; therefore, quantitative estimates from the MTM alert data should be studied further with larger samples.

Conclusion

This study is the first of its kind to explore community pharmacists’ experiences with MTM alerts in order to develop recommendations to improve alert usability and usefulness. Recommendations could be useful to many MTM stakeholders, such as MTM vendors and payers, to improve pharmacist efficiency, patient and prescriber satisfaction with MTM, and patient outcomes.