Abstract
Importance
Medication computerised provider order entry (CPOE) has been shown to decrease errors and is being widely adopted. However, CPOE also has potential for introducing or contributing to errors.
Objectives
The objectives of this study are to (a) analyse medication error reports where CPOE was reported as a ‘contributing cause’ and (b) develop ‘use cases’ based on these reports to test vulnerability of current CPOE systems to these errors.
Methods
A review of medication errors reported to United States Pharmacopeia MEDMARX reporting system was made, and a taxonomy was developed for CPOE-related errors. For each error we evaluated what went wrong and why and identified potential prevention strategies and recurring error scenarios. These scenarios were then used to test vulnerability of leading CPOE systems, asking typical users to enter these erroneous orders to assess the degree to which these problematic orders could be entered.
Results
Between 2003 and 2010, 1.04 million medication errors were reported to MEDMARX, of which 63 040 were reported as CPOE related. A review of 10 060 CPOE-related cases was used to derive 101 codes describing what went wrong, 67 codes describing reasons why errors occurred, 73 codes describing potential prevention strategies and 21 codes describing recurring error scenarios. Ability to enter these erroneous order scenarios was tested on 13 CPOE systems at 16 sites. Overall, 298 (79.5%) of the erroneous orders were able to be entered including 100 (28.0%) being ‘easily’ placed, another 101 (28.3%) with only minor workarounds and no warnings.
Conclusions and relevance
Medication error reports provide valuable information for understanding CPOE-related errors. Reports were useful for developing taxonomy and identifying recurring errors to which current CPOE systems are vulnerable. Enhanced monitoring, reporting and testing of CPOE systems are important to improve CPOE safety.
Keywords: Patient safety; Decision support, computerized; Human error; Information technology; Medication safety
Computerised provider order entry (CPOE) has long been considered and demonstrated to be a high-leverage tool for preventing medication errors, and incentives are being provided to accelerate its adoption.1–3 However, there is a growing awareness and increasing documentation of concerns that CPOE can also introduce or facilitate new errors.4–6 The Institute of Medicine Committee report ‘Health IT and Patient Safety: Building Safer Systems for Better Care’ recognised that Health Information Technology (HIT) is part of a complex sociotechnical system and recommended investing in efforts to uncover and understand the vulnerabilities of HIT systems to errors and unintended consequences.7 More recently, the US Food and Drug Administration Safety and Innovation Act (FDASIA) similarly recommended developing similar approaches for reporting with a key recommendation advocating compilation of reports of errors across multiple systems.8
In 1999, the United States Pharmacopeia (USP) launched a pioneering online medication error reporting system that has now collected more than two million medication errors.9 In 2003, in response to the growing number of reports suggesting that CPOE was playing a role in the medication errors being reported, USP added a coded field for reporters to check off ‘CPOE’ as a contributing cause of the error. Shortly thereafter, the USP's MEDMARX annual report stated that computer entry and CPOE errors had become the third leading contributing cause being checked off in medication error submissions.10 However, since that initial report there has been no detailed investigation or analysis of CPOE-related MEDMARX error reports. Furthermore, the report narratives have not been assessed previously. To better understand how and why the errors were occurring, as well as ways they could have been prevented, we undertook a study of the USP MEDMARX CPOE medication error reports, subjecting these reports to a detailed review as well as testing the vulnerability of current CPOE systems to the types of errors identified.
The aims of this study are to (1) analyse the USP MEDMARX medication error reports where CPOE was checked off as a ‘contributing cause’ of the error by performing in-depth review of 10 000 of the error report narratives to understand details of each error and develop a new taxonomy for CPOE-related errors and (2) develop and test ‘use cases’ based on these actual error reports and assess the vulnerability of leading CPOE systems to these errors.
Methods
Phase I: MEDMARX data analysis—taxonomy development and coding
We queried the USP MEDMARX (now part of Quantros Safety and Risk Management suite) for all medication error reports from January 2003 to April 2010 that were coded by the error reporters as having ‘CPOE’ as one of the ‘contributing causes’ of the errors. These spontaneous error reports were submitted from institutions subscribing to the MEDMARX medication error reporting system. Reporters typically included a mix of centralised quality assurance staff (who collected reports from front-line staff and then entered reports for their institution into MEDMARX) and in a minority of institutions, front-line staff directly entering reports.
We identified a total of 63 040 medical error reports having CPOE checked off as a contributing cause in 1.04 million total reports. These reports served as the data for analysis of CPOE errors and taxonomy development. A total of 10 060 error reports were then manually reviewed, representing all 191 of the reports categorised as National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) outcome categories E–I (ie, categories where patient harm occurred, which we designated as highest priority to review in detail) plus a random sample of the remaining A–E category reports.11 Each of these 10 060 error reports was reviewed and coded by one of three clinical pharmacists (MGA, JJB and ACS) with an emphasis on detailed review of the free text narrative description of the error. For cases where there were questions, or consideration of new codes, the cases were re-reviewed in detail by the entire team of three pharmacists and a general internist (GDS). These codes grounded in the data and served as the basis for our taxonomy development (grounded qualitative analysis).12
The coding was done using a customised qualitative-coding software tool developed in Microsoft Access with codes progressively developed or added based on the error report narratives and iteratively refined via weekly meetings of the clinical review team. Each report was coded to categorise three elements of the error: (a) what happened, (b) why it happened and (c) potential prevention strategies. Pharmacist reviewers were instructed to code what and why exclusively based on information in the narrative and accompanying report. For potential prevention strategies, pharmacist investigators were encouraged to suggest ways the error could have been prevented based on the report and their knowledge of medication safety and information technology. To ensure conservative coding, when reports lacked sufficient information to determine a what, why and prevention classification, reviewers were instructed to assign them as ‘unknown’. Each case could be assigned one or more codes in each category. Inter-reviewer reliability was assessed using a 1% random sample of reviews to calculate a kappa statistic. Once the coding was completed, the codes were re-reviewed and reorganised using several card sorting and iterative team consensus exercises to group and refine the final taxonomy.
Phase II: CPOE vulnerability testing based on reported error scenarios
During the above qualitative review of the error reports, reviewers were instructed to flag cases that might serve as representative ‘test cases’ to assess whether errors identified could be replicated in current CPOE systems. A weighted scoring system based on error frequency, severity, generalisability and testability was used to narrow this list down to key error scenarios for testing. Based on this prioritisation, 21 test scenarios were chosen. Scenarios included erroneous or problematic CPOE orders related to wrong units, major overdoses, drug allergies, order element omission errors, wrong frequency and drug–disease contraindications, as well as three ‘correct’ but ‘complex’ test orders (eg, prednisone tapers, alternate-day dosing, non-formulary drug) that reports often suggested led to problem-prone workarounds such as potentially confusing free text comments in the drug order (see online supplementary appendix 1 for list of test case scenarios).
We identified a convenience sample of leading vendor and homegrown CPOE system test sites and obtained institutional and institutional review board permission to enter these problematic orders on test patients at these sites. To test each of these error scenarios, we recruited one to two typical users (mostly medical residents or primary care attending physicians) with at least 1-year experience with the CPOE system (range 1–8) and instructed them to enter the erroneous orders. Users understood that these orders were problematic but were instructed to proceed with placing these orders as they typically would, using any methods they might routinely use to enter a desired order. Outcomes of whether orders were successfully entered and behaviours of medical doctors and CPOE systems were recorded by physicians or pharmacists (GDS, TE, MGA and ACS) and research assistant (DLW) observers who rated the ease or difficulty of entering the erroneous or complex orders using predefined operational definitions (table 1).
Table 1.
Operational definitions used to classify ease of entry of ‘error scenario’ test orders
How easy was it to place the order? | ||||
1 | 2 | 3 | 4 | 5 |
Easy | Minor workarounds | Some protections | Difficult | Impossible |
| ||||
| ||||
| ||||
| ||||
|
Results
Phase I: qualitative review
Of 1.04 million reported errors, 63 040 (6.1%) were classified by the reporters as CPOE related. Our pharmacists reviewed and coded 10 060 (15.7% sample) of these 63 040 reports and derived a taxonomy that included 101 codes describing what occurred, 67 codes describing why errors occurred as well as 73 codes describing potential prevention strategies (see online supplementary appendix 2 for full taxonomy). Tables 2–4 summarise findings for the top 25 most frequent codes assigned for the what, why and prevention codes. Many reports lacked sufficient detail describing the error to permit adequate coding, particularly to classify why the error occurred. Although all of these reports had ‘CPOE’ checked off by the reporter as a contributing cause, our reviewers could determine the role of the CPOE system in only 5004 (49.8%) of the reports based on report content alone. Pharmacists’ inter-rater agreement rates and kappa scores for the taxonomy coding of what occurred and why the error occurred were 66%, kappa 0.56 (95% CI 0.39 to 0.72) and 64%, kappa 0.58 (95% CI 0.42 to 0.73), respectively.
Table 2.
Top 25 what happened? Codes
Code | N |
---|---|
Missing or incorrect directions/patient instructions | 2088 |
Ordered wrong dose or strength | 877 |
Missing quantity or wrong number ordered | 877 |
Unknown | 680 |
Wrong schedule entered | 566 |
Duplicate order: same exact drug | 510 |
Overdose or potential overdose | 376 |
Ordered wrong formulation/dosage form | 363 |
Order not processed/delayed | 361 |
Extra dose potential | 337 |
Ordered wrong drug | 302 |
Routing issue | 275 |
Comment field issue | 267 |
Missed does potential | 256 |
Nursing administration issues | 240 |
Wrong time selected | 234 |
Ordered/entered for wrong patient | 229 |
Discontinuation issues | 216 |
Not processed/delayed: order confusing/needed clarification | 203 |
Patient missed dose | 203 |
Omitted drug | 167 |
Ordered wrong PO formulation, (eg, ER, XR, etc) | 160 |
Patient given extra dose | 152 |
Telephonic/verbal order issues | 139 |
Correct drug ordered/wrong drug processed | 138 |
ER, extended release; PO, by mouth; XR, extended release.
Table 3.
Top 25 why did it happen? Codes
Code | N |
---|---|
Unknown | 5326 |
Multiple systems (two or more electronic systems) | 1211 |
Use of system or SIG abbreviations | 494 |
Failure to follow established procedures or protocol | 480 |
Profiling Issues: failure to perform or use correctly | 443 |
Inexperienced end user | 415 |
Lack of computer training/system knowledge | 325 |
Typing error | 206 |
Hybrid system (electronic and paper) | 205 |
Communication issues | 200 |
System limitations/inadequacy: routing/mapping issue | 186 |
Lack of clinical knowledge | 186 |
Medication reconciliation issue | 184 |
Alert ignored/overridden | 153 |
Nursing administration issues | 149 |
Pharmacy order entry problems/issues | 134 |
Transcriptions (copy/paste) | 133 |
Comments field free text confusing/confusion | 121 |
eMAR/MAR issues | 119 |
Order set/template/protocol issues | 113 |
Drug dictionary miscode/out-of-date drug information | 109 |
Patient identification issue | 95 |
Initial vs continuing order issue | 93 |
Patient transferred (within hospital) | 91 |
Misinterpretation of order(s) | 72 |
eMAR, electronic medication administration record; MAR, medication administration record; SIG, sig code for directions on how to take a medication.
Table 4.
Top 25 prevention codes
Code | N |
---|---|
Unknown | 3021 |
Systems integration | 1520 |
Standardised constructs for dosing regimens | 1142 |
Enhanced education/training | 791 |
Standardised SIGs | 775 |
Autocalculation for prescription quantities | 604 |
Duplicate order checking/support | 403 |
Default dosing selections | 318 |
Drug database improvements/enhancements | 232 |
Dose range checking | 212 |
Individual dosing calculations | 207 |
Improved design/functionality | 201 |
Medication reconciliation support | 200 |
Scheduling feedback | 169 |
Blank field checking | 141 |
Standardised constructs for dose form-route | 136 |
System for reconciling new/now with continuing dosing | 116 |
Enhanced allergy entry for drugs not included in allergy list | 109 |
Direct order entry to minimise verbal/telephonic issues | 106 |
Route-formulation checking | 93 |
Include time in checklist (12, 24, etc) | 91 |
Duplicate therapy checklist/support | 89 |
Formulary status and restrictions warnings | 89 |
Medication handoff/transfer standardisation | 86 |
Better testing of order sets/updates | 69 |
SIG, sig code for directions on how to take a medication.
Phase II results: CPOE vulnerability testing
Our pharmacist reviewers identified 338 error reports as potential candidate scenarios for testing the vulnerability of current systems. This list was narrowed to 21 scenarios by combining similar scenario types (ie, various drug overdosages; orders for drug to which patient was allergic) and prioritised based on preselected criteria of (a) frequency, (b) seriousness and (c) testability. These scenarios included five inpatient-only scenarios (eg, intravenous orders) that were not tested on outpatient systems, and three ‘complex’ orders (not errors per se, but designed based on reports that repeatedly arose from similar potentially error-prone orders reported). We recruited a convenience sample of 13 representative systems (four homegrown, one open source, eight commercial, including each of the leading inpatient and outpatient vendors) at 16 test sites. Not all tests could be performed on all systems (because of formulary and other design limitations). Excluding these scenarios, we entered a total of 375 erroneous orders during 24 testing sessions on 13 systems at 16 test sites.
Overall, 298 (79.5%) erroneous orders were able to be placed, including 100 (28.0%) being ‘easily’ placed (order simply accepted with no extra steps or warnings), with another 101 (28.3%) placed with only ‘minor workarounds’ (eg, adjusting default dosage, with no warnings). Thus, 201 (56.3%) of the errors could be relatively easily replicated (entered easily or with minor workarounds) with no warning or blocking of potentially dangerous orders. Table 5 lists the frequencies of how often erroneous orders were prevented versus went through easily or with some difficulty. Only 26.6% of orders generated specific warnings related to the erroneous order. Of these, 69% were passive alerts (information display only or easily over-ridden and/or ignored). Another 29% required workarounds but nonetheless, could still be entered. Notable failures included erroneous orders for pioglitazone accepted for patients with congestive heart failure in 87.5%, orders for insulin 60 ‘mL’ (rather than ‘units’) going through in 75.0% and no specific warnings for a 1000-fold levothyroxine overdose in 37.5% of attempts. Figure 1 illustrates a breakdown of which of the erroneous order scenarios had greater protection (ie, were more difficult to enter; higher mean scores overall) versus those where systems were generally more vulnerable (lower mean scores overall). Finally, for 40 of 72 (55.6%) of the error-prone more complex (eg, variable daily warfarin, prednisone taper) test orders, prescribers encountered problems or ordering difficulties, creating potentials for error-prone workarounds.
Table 5.
Frequency distribution of erroneous orders going through, ease with which they went through, and whether there was a warning
N | Per cent | |
---|---|---|
Did order go through? | ||
Yes | 298 | 79.5 |
No | 59 | 15.7 |
Untestable | 18 | 4.8 |
Likert scale: difficulty | ||
Easy | 100 | 28.0 |
Minor workarounds | 101 | 28.3 |
Some protections | 69 | 19.3 |
Difficult | 28 | 7.8 |
Impossible | 59 | 16.5 |
Warnings? | ||
Yes | 95 | 26.6 |
No | 216 | 60.5 |
Irrelevant warnings only | 44 | 12.3 |
Uncertain/maybe | 2 | 0.6 |
Figure 1.
Radar plot showing mean score for each test scenario across all tested computerised provider order entries in difficulty of entering erroneous orders. To maximise safety, the plot ideally should occupy the most outer grid (score 5); that is, impossible to enter the erroneous orders. For example, greatest protection was against 1000-fold overdose of levothyroxine; however, drug–disease contraindication checking had the lowest mean score indicating least protection, hence making it easier to enter this erroneous order.
Discussion
We analysed a large medication error database for errors that were reported as being related to CPOE and developed a new taxonomy of the types, causes and prevention strategies that we could identify in these reports, resulting in a number of useful insights regarding the frequency of specific error types. We then performed vulnerability testing to examine whether these errors could be replicated in current CPOE systems with the worrisome finding that the majority (overall, 56.3%) of the selected erroneous orders could be readily entered.
Report narratives provided both rich details of the types of errors that occur in CPOE systems and served as the basis for the development of a taxonomy that facilitated classification of the types. Leading CPOE-related errors included missing or erroneous sig (label directions) or patient instructions, wrong dose or strength, problems with wrong quantity or strength, scheduling problems (particularly related to inpatient orders and timing of stat vs continuing orders), delays in medication processing or administration due to confusing orders and wrong drug or wrong patient errors.
Many of these problems are not unique to CPOE and could also occur with handwritten ordering, although some, in theory (eg, drug overdosages), should be preventable with properly designed electronic systems.2 13–15 Reasons for these errors were discernible for roughly half of the error narrative reports and included problems with miscommunication between multiple electronic or hybrid paper-electronic systems, user issues such as failure to follow established protocols, inexperience or lack of training in using the CPOE system, typing and pull-down menu errors, medication reconciliation issues, ignoring or over-riding alerts and confusion related to or arising from comments fields.
Although it may be argued that many errors were isolated occurrences or perhaps based on the vulnerability of older systems, when we tested current systems, we found that current systems had high degrees of susceptibility to many of these errors. Nearly 80% of the erroneous orders could be entered, with more than half entered with little or no difficulty or warnings. More than a quarter (28.0%) of the orders were easily entered (in the words of our test physicians and research pharmacists, ‘sailed right through’), with no warnings or additional efforts on the part of the ordering physicians.
Systems that overalert or frustrate busy physicians attempting to enter appropriate orders are also an important problem and can lead to the so-called ‘alert fatigue’. Thus, systems need to balance ease of ordering with appropriate protections. Our study was not designed to determine the best balance, although better designed systems have the potential to achieve both better efficiency (eg, well-designed order sets) and improved signal-to-noise ratio (better ratio of appropriate to nuisance alerts).15–18 Shockingly, one of our test systems had zero warning alert fire in response to our erroneous test orders; we discovered that all alerts had been turned off for a system upgrade several months earlier and it was not until we performed our testing that it was discovered they had not been reinstated. We documented a high degree of variability of vulnerability and alerting from system to system. We even observed variations in different implementations of the same system or even different users entering orders in different ways at the same site. This is similar to the findings when the Leapfrog tool was used to test potential errors, revealing wide variations in detection of test orders that would cause adverse events with varied local implementations, even of the same vendor's systems.19
From the policy perspective, one approach would be to regulate electronic health records and/or clinical decision support. Another would be to allow vendors to continue the current approach in which there is relatively little regulation but to improve postmarket surveillance. The FDASIA committee has recommended the latter approach to the Food and Drug Administration (FDA), Office of the National Coordinator and Federal Communication Commission.8 The data from our study suggest that it is possible to aggregate large numbers of reports across multiple vendors and draw useful conclusions.
Our study was limited by the fact that the reported CPOE-related errors were based on spontaneous self-reporting of medication errors. Thus, no conclusions can be drawn about the actual incidence or relative frequency of these errors and problems. In addition to the well-known problem of under-reporting from spontaneous reporting systems, there are problems with the quality and non-verifiability of the reports we studied.20 21 Many reports were incomplete, lacked details of the role CPOE played in the error (ie, only that they had the box checked that CPOE was contributing factor). Thus, as instructed, our reviewers conservatively coded these reports as ‘unknown’, for what happened and why, making this a leading category. Despite these limitations of the quantity and quality of these spontaneous reports, the errors speak for themselves as noteworthy and likely (particularly based on our vulnerability testing results) representative CPOE safety issues. This points to the need to improve the quality of such reports and perhaps standardise collection of contributing causal factors. Additional potential limitations are that, reports and subscribers to MEDMARX may not be representative of all prescribing systems, and outpatient reports were under-represented as subscribers were mainly hospitals and hospital systems. Many of the reports were nearly a decade old, although our efforts to replicate these errors demonstrate that vulnerabilities also exist in current systems. Our study did not examine the likelihood that the erroneous orders placed would be intercepted by pharmacists or other staff and hence, not cause harm. Nonetheless, the MEDMARX reports contained many examples of where errors did reach or harm patients. Finally, our qualitative pharmacist coding and rating of the error scenarios was based on subjective reviewer judgement. To offset this, we assigned clear operational definitions to the codes as they were developed, had a consensus process for adjudicating questions or disagreement and achieved reasonably good inter-reviewer reliability scores in assessment of the reports. We have continued to refine this taxonomy for a white paper to be published by the US FDA that can help guide future research as well as organisations analysing CPOE-HIT error reports in the future.
In conclusion, we reviewed error reports to identify patterns of CPOE-related errors and used them to develop a new taxonomy and recurring error scenarios. We then tested current systems and found areas of noteworthy vulnerability. Developers and users need to be aware of this potential for error and should build in protection strategies at multiple levels to learn from and protect patients by continuously improving the safety of CPOE systems.22–25 Efforts that permit both better reporting and awareness of medication errors as well as testing the vulnerabilities of local CPOE systems are needed; such efforts are crucial to safe prescribing, ongoing postimplementation monitoring and improvement of CPOE systems.
Supplementary Material
Acknowledgments
We gratefully acknowledge the National Patient Safety Foundation (NPSF) for funding for this study, as well as United States Pharmacopeia and Quantros for providing the MEDMARX error report data. We also acknowledge additional assistance received from Dr Sarah Slight in reviewing the data and taxonomy.
Footnotes
Twitter: Follow David Bates at @dbatessafety
Contributors: GDS, MGA and TE: contributed to conception and design, analysis and interpretation of the data and results, drafting and writing the article, revising it critically for important intellectual content and final approval of the version to be published. JJB, AW, RK, AHR, RBE and ACS: contributed to supervision of the paper, developed hypotheses, analysis and interpretation of the data and results, as well as drafting and writing the article, revising it critically for important intellectual content and final approval of the version to be published. DLW, T-TT and DWB: contributed to interpretation of the data and revising the paper critically for important intellectual and content. None of the sponsors had a role in the design and conduct of the study, collection management, analysis and interpretation of the data; preparation, review or approval of the manuscript and decision to submit the manuscript for publication.
Competing interests: GDS is currently working on an FDA-funded grant investigating CPOE systems and medication safety. RK works with a company, Wearable Intelligence that is bringing Google Glass to healthcare, specifically for use in rounding, with Emergency Medical Systems crew, for handoffs, for use in the operating room to display what the surgeon is viewing and for alerting clinicians. There is no direct competition and it does not affect the issues discussed in our article, but it is a role in an HIT effort. DWB is a coinventor on patent no. 6029138 held by Brigham and Women's Hospital on the use of decision support software for medical management, licensed to the Medicalis Corporation. He holds a minority equity position in the privately held company Medicalis which develops web-based decision support for radiology test ordering. He consults for EarlySense, which makes patient safety monitoring systems.
Provenance and peer review: Not commissioned; internally peer reviewed.
References
- 1.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280:1311–16. 10.1001/jama.280.15.1311 [DOI] [PubMed] [Google Scholar]
- 2.Schiff GD, Rucker TD. Computerized prescribing: building the electronic infrastructure for better medication usage. JAMA 1998;279:1024–9. 10.1001/jama.279.13.1024 [DOI] [PubMed] [Google Scholar]
- 3.Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010;363:501–4. 10.1056/NEJMp1006114 [DOI] [PubMed] [Google Scholar]
- 4.Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197–203. 10.1001/jama.293.10.1197 [DOI] [PubMed] [Google Scholar]
- 5.Ash JS, Sittig DF, Dykstra R, et al. The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inf 2009;78(Suppl 1):S69–76. 10.1016/j.ijmedinf.2008.07.015 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010;170:1578–83. 10.1001/archinternmed.2010.324 [DOI] [PubMed] [Google Scholar]
- 7.IOM (Institute of Medicine). Health IT and patient safety: building safer systems for better care. Washington DC: The National Academies Press, 2012. [PubMed] [Google Scholar]
- 8.Bates DW. Draft FDASIA Committee Report. 2013. http://www.healthit.gov/FACAS/sites/faca/files/FDASIARecommendationsDraft030913_v2.pdf (accessed 7 Sep 2014).
- 9.Santell JP, Hicks RW, McMeekin J, et al. Medication errors: experience of the United States Pharmacopeia (USP) MEDMARX reporting system. J Clin Pharmacol 2003;43:760–7. 10.1177/0091270003254831 [DOI] [PubMed] [Google Scholar]
- 10.Hicks RSJ, Cousins DD, Williams RL. MEDMARX 5th Anniversary data report: a Chartbook of 2003 Findings and Trends 1999–2003. Rockville, MD: United States Pharmacopeia Center for the Advancement of Patient Safety, 2004. [Google Scholar]
- 11.National Coordinating Council for Medication Error Reporting and Prevention. Types of Medication Errors, 2001. http://www.nccmerp.org/medErrorCatIndex.html (accessed 4 Oct 2013).
- 12.Strauss A, Corbin JM. Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, California: Sage Publications, 1990. [Google Scholar]
- 13.Abramson EL, Kaushal R. Computerized provider order entry and patient safety. Pediatr Clin N Am 2012; 59:1247–55. 10.1016/j.pcl.2012.08.001 [DOI] [PubMed] [Google Scholar]
- 14.Gandhi TK, Weingart SN, Seger AC, et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med 2005;20:837–41. 10.1111/j.1525-1497.2005.0194.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13:5–11. 10.1197/jamia.M1868 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Phansalkar S, Edworthy J, Hellier E, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010;17:493–501. 10.1136/jamia.2010.005264 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Wright A, Sittig DF, Carpenter JD, et al. Order sets in computerized physician order entry systems: an analysis of seven sites. AMIA Annu Symp Proc 2010;2010:892–6. [PMC free article] [PubMed] [Google Scholar]
- 18.Wright A, Phansalkar S, Bloomrosen M, et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform 2010;1:331–45. 10.4338/ACI-2010-05-RA-0031 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Metzger J, Welebob E, Bates DW, et al. Mixed results in the safety performance of computerized physician order entry. Health Aff (Millwood) 2010;29:655–63. 10.1377/hlthaff.2010.0160 [DOI] [PubMed] [Google Scholar]
- 20.Hazell L, Shakir SA. Under-reporting of adverse drug reactions : a systematic review. Drug Saf 2006;29:385–96. 10.2165/00002018-200629050-00003 [DOI] [PubMed] [Google Scholar]
- 21.Pal SN, Duncombe C, Falzon D, et al. WHO strategy for collecting safety data in public health programmes: complementing spontaneous reporting systems. Drug Saf 2013;36:75–81. 10.1007/s40264-012-0014-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Sittig DF, Singh H. Electronic health records and national patient-safety goals. N Engl J Med 2012;367:1854–60. 10.1056/NEJMsb1205420 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Horsky J, Phansalkar S, Desai A, et al. Design of decision support interventions for medication prescribing. Int J Med Inf 2013;82:492–503. 10.1016/j.ijmedinf.2013.02.003 [DOI] [PubMed] [Google Scholar]
- 24.Horsky J, Schiff GD, Johnston D, et al. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012;45:1202–16. 10.1016/j.jbi.2012.09.002 [DOI] [PubMed] [Google Scholar]
- 25.McKibbon KA, Lokker C, Handler SM, et al. The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials. J Am Med Inform Assoc 2012;19:22–30. 10.1136/amiajnl-2011-000304 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.