Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Jun 1.
Published in final edited form as: J Biomed Inform. 2015 May 7;55:249–259. doi: 10.1016/j.jbi.2015.04.011

Making Pharmacogenomic-based Prescribing Alerts More Effective: A Scenario-based Pilot Study with Physicians

Casey Lynnette Overby 1, Emily Beth Devine 2,3,4, Neil Abernethy 2,4, Jeannine S McCune 3, Peter Tarczy-Hornoch 2,5,6
PMCID: PMC4704108  NIHMSID: NIHMS692786  PMID: 25957826

Abstract

To facilitate personalized drug dosing (PDD), this pilot study explored the communication effectiveness and clinical impact of using a prototype clinical decision support (CDS) system embedded in an electronic health record (EHR) to deliver pharmacogenomic (PGx) information to physicians. We employed a conceptual framework and measurement model to access the impact of physician characteristics (previous experience, awareness, relative advantage, perceived usefulness), technology characteristics (methods of implementation-semi-active/active, actionability-low/high) and a task characteristic (drug prescribed) on communication effectiveness (usefulness, confidence in prescribing decision), and clinical impact (uptake, prescribing intent, change in drug dosing). Physicians performed prescribing tasks using five simulated clinical case scenarios, presented in random order within the prototype PGx-CDS system. Twenty-two physicians completed the study. The proportion of physicians that saw a relative advantage to using PGx-CDS was 83% at the start and 94% at the conclusion of our study. Physicians used semi-active alerts 74%-88% of the time. There was no association between previous experience with, awareness of, and belief in a relative advantage of using PGx-CDS and improved uptake. The proportion of physicians reporting confidence in their prescribing decisions decreased significantly after using the prototype PGx-CDS system (p=0.02). Despite decreases in confidence, physicians perceived a relative advantage to using PGx-CDS, viewed semi-active alerts on most occasions, and more frequently changed doses toward doses supported by published evidence. Specifically, sixty-five percent of physicians reduced their dosing, significantly for capecitabine (p=0.002) and mercaptopurine/thioguanine (p=0.03). These findings suggest a need to improve our prototype such that PGx CDS content is more useful and delivered in a way that improves physician's confidence in their prescribing decisions. The greatest increases in communication effectiveness and clinical impact of PGx-CDS are likely to be realized through continued focus on content, content delivery, and tailoring to physician characteristics.

Graphical Abstract

graphic file with name nihms-692786-f0001.jpg

1. Introduction

There is great hope for personalized medicine to improve drug safety and efficacy.(1, 2) However, performing personalized drug dosing (PDD) based on pharmacogenomic (PGx) associations (i.e., associations between constitutional genetics and the efficacy, toxicity, and/or pharmacokinetics of medications) is challenging for physicians. One challenge is that physicians have little exposure to PDD recommendations based on how individual genetic variations influence drug metabolism because few exist. Second, it is unclear how to integrate PGx associations with other factors that also can influence dosing, such as comorbidities and end organ function. Third, physicians perceive that existing resources are inadequate to guide PDD.(3) These challenges may be compounded by the scarcity of PGx education in medical schools.(4, 5) The integration of PGx into clinical decision support (CDS) systems may help overcome these challenges.

Several studies examine factors associated with user acceptance of CDS, factors such as time constraints, clinical importance, and patient refusal.(6-10) However, few studies assess how CDS content influences physician use.(11) One study provides a framework for assessing CDS content in terms of clinical appropriateness, defined as correct and current for the patient, according to expert review of the alerts.(12) There are, however, no frameworks for investigating the effective communication of CDS content. There are also few frameworks for investigating how different CDS implementation methods influence appropriate physician use. Potential methods include passive, semi-active, and active CDS. These three terms are synonymous with definitions for knowledge resources, information retrieval tools, and classic CDS, as described by the Agency for Healthcare Research Quality (AHRQ).(11) Passive CDS/knowledge resources use manual processes for both submitting information and generating recommendations; semi-active CDS/information retrieval use automated processes for submitting information and manual processes for generating recommendations; and active CDS/classic CDS use automated processes for both submitting information and generating recommendations. Active CDS also includes modal and non-modal alert messages. An alert presented within a modal dialog box prevents user interaction with the system outside of the dialog box, where as a non-modal dialog box allows interaction with other user interface controls.(13)

The influence of CDS implementation methods on prescribing error rates was investigated in a randomized study of physicians performing prescribing tasks in the context of simulated clinical cases.(14) Results revealed that modal alerts were over three times more effective than non-modal alerts. However, no similar studies investigate the clinical impact of semi-active versus active CDS, and very few studies assess the influence of physician characteristics on CDS use. A recent systematic review of 148 randomized controlled trials of the clinical effectiveness of CDS reported that only 36% of trials described the providers’ expertise in using CDS, even though expertise is variable. Finally, studies evaluating the association between CDS expertise and patient outcomes are few.(11)

In this work, we applied a novel framework and an associated measurement model to the PGx-CDS problem. We conducted a pilot study to assess the impact of physician, technology, and task characteristics on effective communication and the clinical impact of a prototype PGx-CDS system. This is the first study to investigate use of PGx-CDS using this framework; and is a critical first step in implementing PGx-CDS for PDD embedded in an EHR system.

2. Materials and Methods

2.1 Setting

We conducted simulations for our pilot study at the University of Washington (UW). Participants were recruited from a pool of approximately 30 cardiology and 30 oncology fellows practicing at UW. Program coordinators of each fellowship program each distributed four recruitment emails to all fellows. In addition, flyers were created and distributed to potential participants. Six oncology drugs and five cardiology drugs were selected based on their having an approved US Food and Drug Administration (FDA) label that includes evidence for adjusting dosing based on patient-specific biomarkers (that is, on having PGx information). With input from clinical experts, we developed hypothetical clinical case scenarios that prompted prescribing tasks, and revisions of scenarios that included presentation of PGx information. For oncology fellows, we developed one scenario each for capecitabine, irinotecan, mercaptopurine, nilotinib, tamoxifen, and thioguanine. For cardiology fellows, we developed one scenario each for carvedilol, clopidogrel, metoprolol, propafenone, and warfarin. Simulations were conducted between April and August 2011, using two environments for collecting data: (1) within an on-site setting where physicians interacted directly with a prototype PGx-CDS system, and (2) via a web-based survey instrument where PGx-CDS system screenshots were used. Our switch to the web-based survey instrument occurred after data collection for our parallel usability study was completed. That study assessed data from ten participants who completed on-site simulations. (15) In both settings, we presented the same clinical case scenarios with test patient data. The main difference in data collection between the two approaches was that usability measures described in Ref (15) were not collected with the web-based survey. Specifically, within the on-site setting two investigators were present, a facilitator (CLO) and an observer (EBD). The facilitator read instructions to study participants and facilitated completing tasks associated with hypothetical clinical case scenarios using the prototype PGx-CDS system. The observer recorded observations and statements made by each study participant. Additional details can be found in the companion study.(15)

We implemented our prototype CDS system in a test environment that incorporates different approaches for connecting and presenting PGx information, and we built it on the existing clinical infrastructure of the inpatient EHR at UW, including computerized provider order entry (CPOE) capabilities.(16). Using this prototype, we evaluated methods for effective communication of PGx-CDS information to physicians, and assessed the clinical impact of these methods within the prototype system. Note that our definition of effective communication was adapted from the Secretary's Advisory Committee on Genetics, Health, and Society,(17), defining ‘effective communication’ as a process by which PGx data are communicated with supportive information that promotes informed PDD decision-making by physicians.

2.2 Methods for PGx-CDS system implementation and four forms of content

In the prototype PGx-CDS system we presented content in two contexts: (1) reviewing genetic variant information prior to ordering a medication (the laboratory review context), and (2) ordering a medication using CPOE (the medication order entry context). We also implemented both semi-active and active CDS, and made a distinction between low and high actionable alert messages for each drug. Four forms of content were presented: genetic test results, gene specific resources (accessed from an “infobutton” located by the genetic test results), alert messages, and alert message evidence (accessed from an “EVIDENCE” button located in alert messages). Additional details on PGx-CDS implementation and content can be found in Ref (16, 18). All CDS content we used in this pilot is available in Ref (18) (Appendices 12 and 13, pages 337-334).

2.3 Pilot study design

Our approach builds upon previous efforts assessing characteristics of PGx content, assessing implementation needs for PGx-CDS, and implementing a prototype PGx-CDS system(16, 19, 20). Here we describe our pilot study to assess the effective communication and the clinical impact of PGx-CDS.

Using our prototype PGx-CDS system, we designed a randomized pilot study (Figure 1). Each participating fellow was presented with five clinical scenarios appropriate to their clinical specialty (oncology or cardiology). The first and second scenarios were the same for all physicians within the same specialty. The first scenario (tamoxifen, metoprolol) was used to orient the physician to the prototype PGx-CDS system. The second scenario (nilotinib, carvedilol) was used to collect data that would allow us to estimate the inter-physician variability to inform power calculations for subsequent studies. The third through fifth scenarios were presented in a pseudo-randomized fashion with respect to: (1) order, and (2) the low or high actionability of the alert message within each scenario. In our study, we considered evidence provided in the drug labels of medications listed on the FDA “Table of Pharmagogenomic Biomarkers in Drug Labeling”(21) to be actionable information. We classified information associated with a medication and genotype as “high actionable” if the FDA label recommended PDD. “Low actionable” messages did not include a recommendation regarding PDD, but included other forms of evidence such as clinical outcomes from studies and evidence of toxicity.(16) Oncology fellows received scenarios about capecitabine, irinotecan, and one of two purine antagonists, mercaptopurine or thioguanine (mercaptopurine and thioguanine are used to treat the same types of cancers and the doses are similar). Cardiology fellows received scenarios about clopidogrel, propafenone, and warfarin.

Figure 1.

Figure 1

Open in a new tab

Summary of randomized pilot study experiment. We collected pre-/post- experiment study data followed by an experiment with five clinical case scenarios. We used a repeated measures approach to collect measures after each encounter with the third through fifth clinical case scenarios. The main measurement instruments were: (1) a pre- experiment survey, (2) three experimental survey instruments, and (3) a post- experiment survey. “n” is the number of study participants and “m” is the number of times an experimental survey was presented to be completed during the pilot study. Please note that not all participants completed both pre- and post- surveys as shown here. Abbreviations: PGx=pharmacogenomics.

The main measurement instruments were: (1) a pre- experiment survey, (2) three experimental survey instruments, and (3) a post- experiment survey. The pre-experiment survey was completed before the study began. Experimental survey instruments were administered before and after physicians completed each simulated clinical case. These were: (1) a pre-PGx survey, (2) a post-PGx survey, and (3) a post-scenario survey rating the usefulness of various forms of PGx-CDS. Finally, the post-experiment survey was administered upon completions of the fifth clinical scenario (Figures 1 and 2). The study was approved by the UW Human Subjects Institutional Review Board (IRB) with a waiver of documentation of consent.

Figure 2.

Figure 2

Open in a new tab

Example simulated clinical cases presented to oncology (top) and cardiology (bottom) junior physician participants. (Abbreviations: ECOG=Eastern Cooperative Oncology Group, DPD=Dihydropyrimidine Dehydrogenase, DPYD=DPD gene, GERD=Gastroesophageal Reflux Disease, PCI= Percutaneous Coronary Intervention, CYP2C19=Cytochrome P450 2C19)

2.4 Conceptual framework to assess effective communication and clinical impact

To conduct the study, we adapted a conceptual framework (Figure 3) from the field of technology acceptance including measures based on two technology acceptance theories: (1) the diffusion of innovations (DOI) theory; (22, 23); and (2) the task technology fit (TTF) theory.(24). DOI seeks to explain the rate of technology adoption. It proposes five stages to the innovation-decision process: knowledge (exposure to the existence of the innovation and its functions), persuasion (forming a favorable attitude), decision (commitment to adoption), implementation, and confirmation of the decision to adopt. TTF theory suggests that task-technology fit is achieved by both the tasks the user performs and the characteristics of the technology used. Fit, in turn, leads to utilization and performance impact.

Figure 3.

Figure 3

Open in a new tab

Conceptual framework integrating concepts from diffusions of innovations theory (DOI, dotted lines) and task-technology fit theory (TTF, dashed lines). Bold boxes represent the domains we measure.

2.5 Measurement model

To create our measurement model (Figure 4), we drew from several constructs from the DOI and TTF theories. As a proxy for each construct, we identified domains of interest, and then created the survey instruments so that each question informed one of the domains. We used three constructs from DOI theory (prior conditions, knowledge, persuasion), and created survey questions that enabled us to elicit from physicians information about four domains that might predict their perceptions of the PGx-CDS system: (1) previous experience with genetics and CDS for PDD, (2) awareness of genetic tests and CDS for PDD, (3) relative advantage of using genetic tests and CDS for PDD, and (4) perceived usefulness of use of genetic tests and CDS for PDD. Taken together, these questions comprised physician characteristics, the predictors of interest in our experiment.

Figure 4.

Figure 4

Open in a new tab

Measurement model. Italicized text represents domains, bolded boxes are relevant constructs from diffusions of innovations (DOI) and task-technology fit (TTF) theories, and ovals represent predictor, pharmacogenomics clinical decision support (PGx-CDS) system, and outcome variables. See Appendix A for measurement details. Bolded lines and dashed lines are relationships we did and did not test (respectively) in this study. Non-bolded lines indicate relationships between domains, constructs and variables.

Two constructs from TTF theory (task characteristics, technology characteristics) characterized our intervention – the PGx-CDS system. Technology characteristics included the method of CDS alert implementation (semi-active, active) and CDS alert message actionability (low, high). Task characteristics were the prescribing tasks associated with each case scenario. Two additional constructs from the TTF [utilization (adoption, rejection), performance impact] were used to create one of our two outcomes of interest – clinical impact. To assess our second outcome of interest, effective communication, we elicited information about physicians’ confidence in their prescribing decisions and usefulness of the PGx-CDS system.

2.6 Domains, Survey Instruments and Research Questions

Appendix A provides an overview of each domain, the corresponding underlying theories and theoretical constructs, the survey instrument(s) used to assess each domain, the study time points at which each instrument was administered, the specific survey questions (items) used to capture information about each domain, and the response scale for each survey question.

Physician characteristics collected during the pre- and post- experiment surveys included previous experience, awareness, relative advantage, and perceived usefulness. There were two items each of previous experience, awareness, and relative advantage, one referring to the PGx-CDS laboratory review context and the other to the medication order entry context. There were four items measured for perceived usefulness, one for each of the four methods of PGx-CDS implementation (Table 1). For purposes of analysis, we considered previous experience and awareness immutable, in terms of participating in the study; relative advantage and perceived usefulness mutable.

Table I.

Analysis of pre/post study of the perceived usefulness of pharmacogenomics clinical decision support (PGx-CDS) content, and relative advantage of using PGx-CDS content among eighteen physicians who completed both pre- and post- experiment surveys.

Perceptions about PGx-CDS content (% useful, n=18) Odds Ratio (95% CI)
Genetic test results useful 0.40 (0.04, 2.44)
Gene specific resources useful 0.33 (0.03, 1.87)
Alert message useful 0.40 (0.04, 2.44)
Alert message evidence useful 0.43 (0.07, 1.88)
Relative advantage (% agree, n=18) Odds Ratio (95% CI)
Agreement that genetic tests should be used to adjust drug dose 1.0 (0.19, 5.37)
Agreement that decision support aids improve quality of prescribing decisions 0.00 (0.00, 5.32)
Open in a new tab

CI = confidence interval

Technology characteristics recorded were the type of CDS implementation (use of semi-active CDS) and alert message actionablity (presence of a low or high actionable alert message). Two items were used for assessing use of semi-active (optional) CDS, one referring to gene specific resources and the other to PGx-CDS alert message evidence. (Table 1) Alert message actionability for each hypothetical clinical case scenario was recorded for each study participant using survey log data. The task characteristic, that is each clinical scenario and the corresponding drug, was recorded in survey log data. The log included information about the order of presentation of each scenario, as determined by the pseudo-randomization process. Two items were used for measuring effective communication: changes in confidence in prescribing decisions and usefulness. We used the pre-PGx and post-PGx survey instruments to collect data on confidence in prescribing before and after having access to the PGx-CDS system. We used the post-scenario survey instrument to collect data on perceived usefulness of scenario-specific forms of PGx-CDS. Occurrences where physicians did not use PGx-CDS were evaluated as though they do not find it useful.

We assessed clinical impact by measuring prescribing uptake, prescribing intent, and change in PDD. We measured uptake of semi-active PGx-CDS (the optional gene specific resources and alert message evidence) content. We used the post-scenario survey instrument to assess uptake of these two forms of semi-active PGx-CDS. We also measured scenario-specific prescribing intent. The pre-PGx and post-PGx survey instruments were used to collect data on scenario-specific prescribing intent, specifically whether the physician cancelled, overrode, or modified the order for each drug prescribed after having access to PGx-CDS in each clinical case scenario. Change in PDD was assessed by whether he/she changed their preferred starting dose, when comparing before to after completing each scenario.

Using our Measurement Model, we explored eight research questions:

  1. Do the mutable physician characteristics (relative advantage and perceived usefulness) change, when comparing before to after use of the PGx-CDS system?;

  2. Is there an association between baseline physician characteristics (previous experience, awareness, relative advantage) and the clinical impact (uptake) of the PGx-CDS system, stratified by each type of semi-active PGx-CDS content (gene-specific resources; alert message evidence)?;

  3. Is the technology characteristic of actionability (low/high) associated with:
    • a)
      Effective communication (level of confidence in prescribing decision)?;
    • b)
      Effective communication (usefulness), stratified by each type of semi-active (gene-specific resources; alert message evidence) and active (genetic test results; alert messages) PGx-CDS content?;
    • c)
      Clinical impact (uptake), stratified by each type of semi-active PGx-CDS content?;
    • d)
      Clinical impact (prescribing intent)?;
  4. After use of the PGx-CDS system, as a whole, is there a change in:
    • a)
      Effective communication (change in confidence in prescribing decision)?;
    • b)
      Clinical impact (change in PDD)? Stratified by individual drug

2.7 Statistical Analyses

The data from the third through fifth scenarios were used for statistical analyses. We used McNemar's test to estimate whether scores for physician characteristics (relative advantage or perceived usefulness) changed after use of the PGx-CDS system (Question 1). We used generalized estimating equations (GEE) with an exchangeable correlation structure to account for within physician correlation across the three clinical scenarios when investigating: associations between physician characteristics and the uptake of the PGx-CDS system (Question 2); associations between the technology characteristic of actionability (low/high) and effective communication (level of confidence in prescribing decision or usefulness, Questions 3a and 3b); associations between the technology characteristic of actionability (low/high) and clinical impact (uptake or prescribing intake, Questions 3c and 3d); and to assess whether there was a change in effective communication (change in confidence in prescribing decision) after using the PGx-CDS system (Question 4a). For all GEE analyses, we report odds ratios (OR) and 95% confidence intervals (CIs). For Question 3a, we report frequencies stratified by actionability. For Questions 3b and 3c, we stratified results by type of semi-active or active alert and report frequencies. To investigate whether there was a change in clinical impact (change in PDD) after using the PGx-CDS system, we used the two-tailed Wilcoxon signed rank test, stratified by individual drug (Question 4b). We report Z and p-values, as well as the mean dose values prior to and after having access to the PGx-CDS system. All analyses were conducted in Stata 11.2 (Stata Corp, College Station, Tx). We used a significance level of 0.05 throughout.

3. Results

This study enrolled 22 physicians, 15 (68%) were oncology fellows, and 7 (32%) were cardiology fellows. All seven cardiologists and three of the oncologists completed the study in the simulation laboratory environment; the remaining twelve oncologists completed the web-based version. There was an equal distribution of three clinical case scenarios (for medications clopidogrel, propafenone and warfarin) presented to the cardiology fellows. 52% of alerts presented to cardiology fellows were of low-actionability; 48% of high actionability. These proportions were similar for oncology fellows.

3.1 Physician characteristics

When investigating whether using the PGx-CDS system as a whole was associated with changes in these two mutable physician characteristics, we found: the proportion of physicians that see a relative advantage of using genetic test results for PDD remained the same (61.0%); the proportion of physicians that see a relative advantage to using PGx-CDS content for PDD was 83% prior to and 94% following our experiment; and when assessing each of four forms of PGx-CDS content, the proportion of physicians who indicated a perceived usefulness was lower after use of the PGx-CDS system to a range of 27.8% to 50.0%, from 44.4% to 66.7% (Figure 5). These differences, however, are not statistically significant (Table I). When investigating the impact of three mutable physician characteristics (previous experience, awareness, relative advantage) on clinical impact (uptake of semi-active PGx-CDS) we found no significant associations (Table II). We did not stratify our results for physician characteristics by clinical specialty because of the small number of participants.

Figure 5.

Figure 5

Open in a new tab

Frequencies at which eighteen physicians who completed both pre- and post-experiment surveys agree in a relative advantage and in a perceived usefulness of using genetic test results and pharmacogenomics clinical decision support (PGx-CDS) content for personalized drug dosing (PDD).

Table II.

Impact of physician characteristics on clinical impact (uptake) of semi-active pharmacogenomics clinical decision support (PGx-CDS) among fifty-seven encounters with the hird through fifth clinical case scenarios.

Physician characteristic (n=57) Gene specific resources OpenInfobutton website Odds Ratio (95% CI) Alert message evidence OpenInfobutton website Odds Ratio (95% CI)
Awareness 1.21 (0.17,8.6) 1.14 (0.22-5.99)
Experience a 3.92 (0.42-36.23)
Relative advantage 5.59 (0.55, 56.93) 0.35 (0.07-1.63)
Open in a new tab
a

Odds ratio value is missing because the model did not converge; CI = confidence interval

3.2 Clinical impact

When combining across low and high actionable alerts, physicians used the gene specific resources 88% of the time and used the alert message evidence 74% of the time. When investigating if use of the PGx-CDS system as a whole was associated with physician changes in PDD, the results were case scenario and drug-specific (i.e., task characteristic-specific). Sixty-five percent of physicians changed the prescribed dose after using the PGx-CDS system, although a significant change (a decrease) was only observed for capecitabine (z=3.047, p=0.002) and mercaptopurine/thioguanine (z=2.168, p = 0.03). (Table III)(25-30)

Table III.

Impact of using the pharmacogenomics clinical decision support (PGx-CDS) system on quantity of drug prescribed (in mg) among fourty-five encounters with the third through fifth clinical case scenarios for which physicians completed both pre- and post-PGx surveys.

Prescribing task for each clinical case scenario Standardized guidelines (yes/no) Pre PGx-CDS system dosing (Mean +/− SD) Post PGx-CDS system dosing (Mean +/− SD) Z-value
Cardiology medications
Warfarin (n=7) Yes (Ref 25) 5 ± 1.4 4.1 ± 1.2 1.41
Clopidogrel (n=5) No# 75 ± 0 45 ± 67.1 1.00
Propafenone (n=6) No 228 ± 197 100 ± 77.5 1.71
Oncology medications
Irinotecan (n=8) No 59.4 ± 4.2 40.6 ± 25.7 1.98*
Capecitabine (n=12) No# 1050 ± 127.9 237.6 ± 444.2 3.05**
Mercaptopurine/Thioguanine (n=7) Yes (Ref 26, 27) 59.7 ± 22.78 24.1 ± 26.2 2.17*
Open in a new tab
*

p<0.05

**

p<0.01

#

No standardized guideline was available at the time this study was conducted (CPIC guidelines were first published for clopidogrel in August 2011 (28, 29), and for capecitabine in August 2013 (30))

3.3 Effective communication

When assessing use of the PGx-CDS system as a whole on the effectiveness of communication, the proportion of physicians reporting confidence in their prescribing decision decreased, when comparing before to after use of the PGx-CDS system (p=0.02). Further, those who were confident prior to using the system became less confident after, when compared to those who were less confident before (OR 0.18, 95% CI: 0.02, 0.83). Table IV summarizes the proportion of each PGx-CDS content type physicians found useful during the study, stratified by case scenario (a task characteristic).

Table IV.

Frequency at which physicians found pharmacogenomics clinical decision support (PGx-CDS) content presented during the study useful, very useful or extremely useful among fifty-seven encounters with the third through fifth clinical case scenarios.

Content type for each clinical case scenario Genetic test results (%) Gene specific resources (%) Alert message (%) Alert message evidence (%)
Cardiology medications
Warfarin (n=7) 71% 57% 86% 29%
Clopidogrel (n=7) 86% 71% 86% 86%
Propafenone (n=7) 71% 86% 71% 57%
Oncology medications
Irinotecan (n=11) 82% 82% 82% 64%
Capecitabine (n=13) 85% 77% 85% 62%
Mercaptopurine/Thioguanine (n=12) 83% 67% 83% 42%
Total (n=57) 81% 74% 82% 56%
Open in a new tab

3.4 Technology characteristics

When investigating the impact of a technology characteristic on the clinical impact of PGx-CDS, we detected no significant associations between level of alert message actionability (low/high) and uptake of semi-active PGx-CDS (Table V) or prescribing intent (OR, 0.50; 95% CI: 0.21, 1.16).

Table V.

Impact of technology characteristic (alert message actionabilitylow/high) on effective communication (perceived usefulness) and clinical impact (uptake) of pharmacogenomics clinical decision support (PGx-CDS) among fifty-seven encounters with the third through fifth clinical case scenarios.

Perceived usefulness of PGx-CDS with low and high alert message actionability (n=57) Odds Ratio (95% CI)
Genetic test results usefulness 0.74 (0.38, 0.97)
Gene specific resources usefulness 1.07 (0.63, 1.81)
Alert message usefulness 0.53 (0.18, 1.53)
Alert message evidence usefulness 1.50 (0.65, 3.39)
Uptake of semi-active PGx-CDS with low and high alert message actionability (n=57) Odds Ratio (95% CI)
Gene specific resources used 0.91 (0.21, 3.98)
Alert message evidence used 0.91 (0.26, 3.17)
Open in a new tab

CI = confidence interval

When investigating the impact of a technology characteristic, on the effectiveness of communication, physicians were confident in their prescribing decision with 45% of low actionable messages and 56% of high actionable messages. However, we detected no significant association between level of alert message actionability (low/high) and confidence in prescribing decision (OR 1.84, CI: 0.55, 6.20). Likewise, no significant association was found between alert message actionablility (low/high) and the usefulness of any of the four forms of PGx-CDS content. (Table V). 44 to 56% low actionable messages, and 44 to 56% of high actionable messages were found useful, depending on the form of content. Figure 6 illustrates the frequency at which physicians found each of the four forms of content useful with low and high actionable alerts.

Figure 6.

Figure 6

Open in a new tab

Frequencies at which physicians found pharmacogenomics clinical decision support (PGx-CDS) content useful with low and high actionable messages.

4. Discussion

This pilot study contributes to the growing body of literature about how to implement CDS for personalized drug dosing (PDD).(31-40) We explored questions to inform development of a prototype PGx-CDS system for PDD that could be embedded in the EHR. The results from this pilot study suggest, overall, that physicians will believe in a relative advantage to using a PGx-CDS system, but that improvements over the current prototype will be needed for PGx-CDS content to be communicated effectively. In summary, physicians perceived a relative advantage to using genetic test results and PGx-CDS content for PDD (Section 3.1. Physician characteristics). Physician characteristics were not associated with the clinical impact of the PGx-CDS system (Table II). When assessing effective communication, use of the PGx-CDS system was associated with lowered confidence in prescribing decisions (Section 3.3. Effective communication), and is reflected in the proportion of physicians who changed doses toward doses supported by published evidence after all scenarios combined (Table III). Interestingly, the variability in doses increased after use of PGx-CDS, and decreases in dose were significant only for two oncology drugs. Technology characteristics (alert message actionability) was not associated with clinical impact or effective communication of PGx-CDS (Table V).

Exploring each of these findings, in turn, we find that both prescribing intent and usefulness may be influenced by the design of the prototype PGx-CDS system. The majority (>80%) of physicians found that PGx CDS content improved the quality of prescribing decisions, but only 61% believed it was advantageous to use genetic test results for PDD. Even though genetic test results must exist before CDS content can be utilized, this finding could reflect that without associated CDS, the genetic test results are less often perceived as providing an advantage. Despite the majority of physicians believing in a relative advantage to using PGx CDS content, we also found that the perceived usefulness of PGx-CDS was lower, ranging from 28% of physicians finding the alert messages useful, to 50% finding the genetic test results useful (Figure 5, post-experiment survey). Given that we presented genetic test results within each simulated clinical case in the laboratory review context, and prior to the physician viewing the medication order entry context, it is possible that (s)he made prescribing decisions prior to ordering the medication. Given this order of presentation of PGx-CDS content, the physician might be less likely to find the alert message useful, or to change dosing because their prescribing decision was already influenced by the information in the laboratory review context. These findings highlight the need to better understand how the timing with which PGx-CDS is provided influences PDD decision-making. It is well-established that providing decision support at the time and location of decision-making is important for successful CDS implementation (11, 41). In future work, it may be appropriate to deliver PGx-CDS in the medication order entry context alone, rather than in two contexts as we did in this pilot study. Alternatively, prescribing intent may be measured and assessed for both contexts (i.e., after viewing genetic test results, and after ordering a medication). That kind of analysis would facilitate better understanding use of CDS provided a multiple points in the PDD process.

To our knowledge, our study is the first to evaluate the influence of physician characteristics on the clinical impact of PGx-CDS. Other studies have reported that physician characteristics, including attitudes and behaviors, predict adoption of the EHR (42, 43) and e-prescribing.(44) Here we assess the influence of physician characteristics, specifically previous experience, awareness, and relative advantage, on the adoption of PGx-CDS and did not find a statistically significant association. This question, however, may be worth investigating further given that physicians were generally more likely to use semi-active PGx-CDS when, prior to study participation, they were aware of genetic testing, had prior experience using genetic testing, or agreed in a relative advantage to using genetic tests (Table II). The one exception was that physicians were less likely to use the alert message evidence when they had agreed that genetic testing should be used for PDD prior to study participation. Future studies with larger sample sizes will be needed to improve our understanding of how physician characteristics influence clinical impact (uptake/adoption) of PGx-CDS.

Our results suggest that limiting PGx-CDS to include only high actionable alert messages with supportive evidence in the medication order entry context might increase the effective communication (usefulness) of PGx-CDS (Figure 6). These findings are expected, given nearly half of the alert messages were high actionable and thus provided a recommendation. Providing a recommendation is associated with successful CDS implementation (11, 41) in other settings, but more research is needed for the PGx-CDS system.

This study found that task characteristics (hypothetical clinical case with corresponding drug) may influence the clinical impact of PGx-CDS on PDD. Specifically, we found that the availability of PGx-CDS significantly influenced chemotherapy PDD, but not cardiology PDD (Table III). One plausible explanation is that oncologists seek to maintain medication dose intensity with the goal of maximizing survival. Presenting alerts to oncologists may lower confidence in their prescribing decisions, thus providing more “opportunities” for them to lower drug doses when they are presented with information suggesting they do so. We also found that variability in doses increased for all medications, suggesting that some physicians made relatively large changes in the dose while others did not. These findings highlight the need to know the characteristics of intended users and the tasks they perform, so that the clinical impact of these characteristics can be determined. Understanding differences in physician specialties is also needed, because the frequency of medication dosing errors differs among specialties.(45) Future work should explore whether individualizing alerts by physician specialty ensures an acceptable number and quality of alerts for a user.(46) Further, more investigation is needed to understand the influence of confidence in prescribing decision on PGx-CDS implementation success, and to understand how task and physician characteristics influence the clinical impact of PGx-CDS.

The strengths of this pilot study are in the use of a novel conceptual framework and a measurement model to assess the effectiveness of communication and the clinical impact of PGx-CDS for PDD. The limitations include small sample size and use of simulated clinical cases. Only 22 participants could be recruited, despite planned recruitment strategies. Also, physicians who participated may be inherently more accepting of PGx-CDS than those who did not participate; indeed, all participants had previous experience in using EHRs and CPOE systems, plus the recruitment pool consisted of junior physicians who were accustomed to using computers. Finally, it is difficult to know if physicians will attend to PGx-CDS content during a busy clinical practice in the same way that they attend to PGx-CDS content during simulated exercises.

This work highlights four lines of inquiry for future research in the effective communication and clinical impact of PGx-CDS for PDD: (1) content, (2) content delivery, (3) types of physician recipients, and (4) enhancing the usefulness of PGx-CDS. First, we need to better understand what kinds of PGx-CDS content are most useful for PDD and when that content should be delivered. In a parallel study, we asked physicians to think aloud as they interacted with the PGx-CDS prototype system.(15) Results from that study suggested that providing dosing recommendations in each alert, rather than summarizing evidence at the point of prescribing decisions, might improve the effective communication of PGx-CDS. Future work will investigate this hypothesis and aim to understand how effective communication influences the clinical impact of PGx-CDS. Second, research is needed to understand how to tailor PGx-CDS to different types of clinical users. Lastly, future investigation is needed into when and how semi-active CDS (and knowledge resources) are useful. Improving the effective communication of PGx-CDS is important to achieve the ultimate goal of improving patient safety with drug dosing tailored to the individual patient.

5. Conclusion

Clinical decision support holds promise to overcome current challenges to implementing personalized drug dosing based on pharmacogenomic associations. We provide a novel framework to study the effective communication and the clinical impact of a prototype PGx-CDS system. Our main findings from this pilot study were: physicians perceived a relative advantage to using PGx-CDS content, although not to using genetic test results specifically; physicians perceived all four types of PGx-CDS content moderately useful, and they used gene-specific resources and alert message evidence frequently; physician characteristics were not associated with the clinical impact of the PGx-CDS system; technology characteristics (implementations and alert message actionability) were not associated with effective communication or clinical impact of PGx-CDS; use of the PGx-CDS system was associated with lowered confidence in prescribing decisions and is reflected in the proportion of physicians who reduced PDD after all scenarios combined. Interestingly, these decreases were significant only for two oncology drugs. Overall, this study suggests that physicians will see a relative advantage to using the system, but also identifies potential improvements in content, content delivery, and tailoring of physician characteristics to enhance communication effectiveness and the clinical impact of PGx-CDS.

Supplementary Material

Highlights.

  • Clinical decision support (CDS) for pharmacogenomics (PGx) holds promise

  • A novel conceptual framework and measurement model was used to assess pilot PGx-CDS

  • This study suggests that physicians see a relative advantage to using a pilot PGx-CDS system

  • We identify potential improvements to enhance communication effectiveness of PGx-CDS

  • We identify potential improvements to enhance the clinical impact of PGx-CDS

Acknowledgements

The authors would like to thank: Dr. Guilherme Del Fiol for assisting with the development of OpenInfobutton websites; Dr. Joe Smith for assisting with the development of Cerner Discern Expert® triggered alert messages; Dr. Lingtak-Neander Chan for assisting with developing cardiology scenarios; Dr. Cathy Yeung for assisting with developing pharmacogenetics and metabolism resources specific to this study; Drs. Isabelle Ragueneau-Majlessi, Cathy Yeung, and Sophie Argon, for providing input on the interpretation of pharmacogenomic information and for allowing us to use the e-PKgene resource in this study; Drs. Daniel Capurro, Bernardo Goulart, and Veena Shankaran for pilot testing our simulated cases; and Dr. David Fenstermacher who provided feedback on manuscript content. This paper is largely based on Chapter 7 titled “Evaluating the utility of the pharmacogenomics clinical decision support model implementation” from the PhD dissertation of CLO titled “A clinical decision support model for incorporating pharmacogenomics knowledge into electronic health records for drug therapy individualization: a microcosm of personalized medicine.” This research was supported by the following grants: NIH NHHRI #T32 HG000035, NIH NLM #T15 LM07442, and NIH NCRR UL 1RR 025014 (Overby); and AHRQ 5K08 HS014739 (PI: Devine). The funders played no role in the study.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of Interest Statement

Casey Lynnette Overby – has no conflicts to declare

Emily Beth Devine – has not conflicts to declare

Neil Abernethy – has no conflicts to declare

Jeannine S. McCune – has not conflicts to declare

Peter Tarczy-Hornoch – has no conflicts to declare

References

  • 1.Woodcock J, Lesko LJ. Pharmacogenetics--tailoring treatment for the outliers. N Engl J Med. 2009 Feb 19;360(8):811–3. doi: 10.1056/NEJMe0810630. [DOI] [PubMed] [Google Scholar]
  • 2.Giacomini KM, Krauss RM, Roden DM, Eichelbaum M, Hayden MR, Nakamura Y. When good drugs go bad. Nature. 2007 Apr 26;446(7139):975–7. doi: 10.1038/446975a. [DOI] [PubMed] [Google Scholar]
  • 3.Barrett JS, Narayan M, Patel D, Zuppa AF, Adamson PC. Prescribing habits and caregiver satisfaction with resources for dosing children: rationale for more informative dosing guidance. BMC Pediatr. 2011;11:25. doi: 10.1186/1471-2431-11-25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Menasha J, Schechter C, Willner J. Genetic testing: a physician's perspective. Mt Sinai J Med. 2000 Mar;67(2):144–51. [PubMed] [Google Scholar]
  • 5.Baars MJ, Henneman L, Ten Kate LP. Deficiency of knowledge of genetics and genetic tests among general practitioners, gynecologists, and pediatricians: a global problem. Genet Med. 2005 Nov-Dec;7(9):605–10. doi: 10.1097/01.gim.0000182895.28432.c7. [DOI] [PubMed] [Google Scholar]
  • 6.Fihn SD, McDonell MB, Vermes D, et al. A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin. National Consortium of Anticoagulation Clinics. J Gen Intern Med. 1994 Mar;9(3):131–9. doi: 10.1007/BF02600026. [DOI] [PubMed] [Google Scholar]
  • 7.Sundaram V, Lazzeroni LC, Douglass LR, Sanders GD, Tempio P, Owens DK. A randomized trial of computer-based reminders and audit and feedback to improve HIV screening in a primary care setting. Int J STD AIDS. 2009 Aug;20(8):527–33. doi: 10.1258/ijsa.2008.008423. [DOI] [PubMed] [Google Scholar]
  • 8.Tamblyn R, Huang A, Perreault R, et al. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. Cmaj. 2003 Sep 16;169(6):549–56. [PMC free article] [PubMed] [Google Scholar]
  • 9.Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med. 1997 Nov;12(11):672–8. doi: 10.1046/j.1525-1497.1997.07140.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Goud R, de Keizer NF, ter Riet G, et al. Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation. Bmj. 2009;338:b1440. doi: 10.1136/bmj.b1440. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Lobach DF, Sanders GD, Bright TJ, et al. Enabling Health Care Decisionmaking Through Clinical Decision Support and Knowledge Management. Agency for Healthcare Research and Quality (US); Apr, 2012. 2012. [PMC free article] [PubMed] [Google Scholar]
  • 12.McCoy AB, Waitman LR, Lewis JB, et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc. 2012 May-Jun;19(3):346–52. doi: 10.1136/amiajnl-2011-000185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Olsen D. Building Interactive Systems: Principles for Human-Computer Interaction. Course Technology Press; Boston: 2009. [Google Scholar]
  • 14.Scott GP, Shah P, Wyatt JC, Makubate B, Cross FW. Making electronic prescribing alerts more effective: scenario-based experimental study in junior doctors. J Am Med Inform Assoc. 2011 Nov-Dec;18(6):789–98. doi: 10.1136/amiajnl-2011-000199. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Devine EB, Lee CJ, Overby CL, et al. Usability evaluation of pharmacogenomics clinical decision support aids and clinical knowledge resources in a computerized provider order entry system: a mixed methods approach. Int J Med Inform. 2014 Jul;83(7):473–83. doi: 10.1016/j.ijmedinf.2014.04.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Overby CL, Tarczy-Hornoch P, Kalet IJ, et al. Developing a Prototype System for Integrating Pharmacogenomics Findings into Clinical Practice. Journal of personalized medicine. 2012;2(4):241–56. doi: 10.3390/jpm2040241. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Secretary's Advisory Committee on Genetics Health and Society (SACGHS) U.S. System of Oversight of Genetic Testing: A Response to the Charge of the Secretary of Health and Human Services. 2008.
  • 18.Overby CL. A Clinical Decision Support Model for Incorporating Pharmacogenomics Knowledge Into Electronic Health Records for Drug Therapy Individualization: A Microcosm of Personalized Medicine [Ph.D.] University of Washington; Ann Arbor: 2011. [Google Scholar]
  • 19.Overby CL, Tarczy-Hornoch P, Hoath JI, Kalet IJ, Veenstra DL. Feasibility of incorporating genomic knowledge into electronic medical records for pharmacogenomic clinical decision support. BMC Bioinformatics. 2010;11(Suppl 9):S10. doi: 10.1186/1471-2105-11-S9-S10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Overby CL, Tarczy-Hornoch P, Hoath J, Smith JW, Devine EB, Fenstermacher D. An Evaluation of Functional and User Interface Requirements for Pharmacogenomic Clinical Decision Support. 2011:134–41. [Google Scholar]
  • 21.US Food and Drug Administration [02/24/2015];Table of Pharmacogenomic Biomarkers in Drug Labels. Last Updated 08/18/2014; Available from: http://www.fda.gov/drugs/scienceresearch/researchareas/pharmacogenetics/ucm083378.htm.
  • 22.Moore GC, Benbasat I. Development of an instrument to measure the perceptions of adopting an information technology innovation. Information systems research. 1991;2(3):192–222. [Google Scholar]
  • 23.Rogers EM. Diffusion of innovations. The Free Press; 1962. [Google Scholar]
  • 24.Goodhue DL, Thompson RL. Task-technology fit and individual performance. Mis Quarterly. 1995:213–36. [Google Scholar]
  • 25.Johnson JA, Gong L, Whirl-Carrillo M, et al. Clinical Pharmacogenetics Implementation Consortium Guidelines for CYP2C9 and VKORC1 genotypes and warfarin dosing. Clin Pharmacol Ther. 2011 Oct;90(4):625–9. doi: 10.1038/clpt.2011.185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Relling MV, Gardner EE, Sandborn WJ, et al. Clinical pharmacogenetics implementation consortium guidelines for thiopurine methyltransferase genotype and thiopurine dosing: 2013 update. Clin Pharmacol Ther. 2013 Apr;93(4):324–5. doi: 10.1038/clpt.2013.4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Relling MV, Gardner EE, Sandborn WJ, et al. Clinical Pharmacogenetics Implementation Consortium guidelines for thiopurine methyltransferase genotype and thiopurine dosing. Clin Pharmacol Ther. 2011 Mar;89(3):387–91. doi: 10.1038/clpt.2010.320. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Scott SA, Sangkuhl K, Gardner EE, et al. Clinical Pharmacogenetics Implementation Consortium guidelines for cytochrome P450-2C19 (CYP2C19) genotype and clopidogrel therapy. Clin Pharmacol Ther. 2011 Aug;90(2):328–32. doi: 10.1038/clpt.2011.132. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Scott SA, Sangkuhl K, Stein CM, et al. Clinical Pharmacogenetics Implementation Consortium guidelines for CYP2C19 genotype and clopidogrel therapy: 2013 update. Clin Pharmacol Ther. 2013;94(3):317–23. doi: 10.1038/clpt.2013.105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Caudle KE, Thorn CF, Klein TE, et al. Clinical Pharmacogenetics Implementation Consortium guidelines for dihydropyrimidine dehydrogenase genotype and fluoropyrimidine dosing. Clin Pharmacol Ther. 2013 Dec;94(6):640–5. doi: 10.1038/clpt.2013.172. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Bell GC, Crews KR, Wilkinson MR, et al. Development and use of active clinical decision support for preemptive pharmacogenomics. J Am Med Inform Assoc. 2014 Feb;21(e1):e93–9. doi: 10.1136/amiajnl-2013-001993. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Bielinski SJ, Olson JE, Pathak J, et al. Preemptive genotyping for personalized medicine: design of the right drug, right dose, right time-using genomic data to individualize treatment protocol. Mayo Clin Proc. 2014 Jan;89(1):25–33. doi: 10.1016/j.mayocp.2013.10.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Fusaro VA, Brownstein C, Wolf W, et al. Development of a scalable pharmacogenomic clinical decision support service. AMIA Jt Summits Transl Sci Proc. 2013;2013:60. [PubMed] [Google Scholar]
  • 34.Goldspiel BR, Flegel WA, DiPatrizio G, et al. Integrating pharmacogenetic information and clinical decision support into the electronic health record. J Am Med Inform Assoc. 2014 May-Jun;21(3):522–8. doi: 10.1136/amiajnl-2013-001873. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Gottesman O, Scott SA, Ellis SB, et al. The CLIPMERGE PGx Program: clinical implementation of personalized medicine through electronic health records and genomicspharmacogenomics. Clin Pharmacol Ther. 2013 Aug;94(2):214–7. doi: 10.1038/clpt.2013.72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Hoffman JM, Haidar CE, Wilkinson MR, et al. PG4KDS: a model for the clinical implementation of pre-emptive pharmacogenetics. Am J Med Genet C Semin Med Genet. 2014 Mar;166C(1):45–55. doi: 10.1002/ajmg.c.31391. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.O'Donnell PH, Danahey K, Jacobs M, et al. Adoption of a clinical pharmacogenomics implementation program during outpatient care--initial results of the University of Chicago “1,200 Patients Project”. Am J Med Genet C Semin Med Genet. 2014 Mar;166C(1):68–75. doi: 10.1002/ajmg.c.31385. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Pulley JM, Denny JC, Peterson JF, et al. Operational implementation of prospective genotyping for personalized medicine: the design of the Vanderbilt PREDICT project. Clin Pharmacol Ther. 2012 Jul;92(1):87–95. doi: 10.1038/clpt.2011.371. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Shuldiner AR, Palmer K, Pakyz RE, et al. Implementation of pharmacogenetics: the University of Maryland Personalized Anti-platelet Pharmacogenetics Program. Am J Med Genet C Semin Med Genet. 2014 Mar;166C(1):76–84. doi: 10.1002/ajmg.c.31396. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Van Driest SL, Shi Y, Bowton EA, et al. Clinically actionable genotypes among 10,000 patients with preemptive pharmacogenomic testing. Clinical Pharmacology & Therapeutics. 2013;95(4):423–31. doi: 10.1038/clpt.2013.229. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. Bmj. 2005 Apr 2;330(7494):765. doi: 10.1136/bmj.38398.500764.8F. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Wakefield DS, Halbesleben JR, Ward MM, Qiu Q, Brokel J, Crandall D. Development of a measure of clinical information systems expectations and experiences. Med Care. 2007 Sep;45(9):884–90. doi: 10.1097/MLR.0b013e3180653625. [DOI] [PubMed] [Google Scholar]
  • 43.Schectman JM, Schorling JB, Nadkarni MM, Voss JD. Determinants of physician use of an ambulatory prescription expert system. Int J Med Inform. 2005 Sep;74(9):711–7. doi: 10.1016/j.ijmedinf.2005.05.011. [DOI] [PubMed] [Google Scholar]
  • 44.Devine EB, Patel R, Dixon DR, Sullivan SD. Assessing attitudes toward electronic prescribing adoption in primary care: a survey of prescribers and staff. Inform Prim Care. 2010;18(3):177–87. doi: 10.14236/jhi.v18i3.770. [DOI] [PubMed] [Google Scholar]
  • 45.Fijn R, Van den Bemt PM, Chow M, De Blaey CJ, De Jong-Van den Berg LT, Brouwers JR. Hospital prescribing errors: epidemiological assessment of predictors. Br J Clin Pharmacol. 2002 Mar;53(3):326–31. doi: 10.1046/j.0306-5251.2001.bjcp1558.doc.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Krall MA, Sittig DF. Clinician's assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp. 2002:400–4. [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

NIHMS692786-supplement.docx (46.8KB, docx)

RESOURCES