Evaluation and Certification of Computerized Provider Order Entry Systems

Abstract

Computerized physician order entry (CPOE) is an application that is used to electronically write physician orders either in the hospital or in the outpatient setting. It is used in about 15% of U.S. Hospitals and a smaller percentage of ambulatory clinics. It is linked with clinical decision support, which provides much of the value of implementing it. A number of studies have assessed the impact of CPOE with respect to a variety of parameters, including costs of care, medication safety, use of guidelines or protocols, and other measures of the effectiveness or quality of care. Most of these studies have been undertaken at CPOE exemplar sites with homegrown clinical information systems. With the increasing implementation of commercial CPOE systems in various settings of care has come evidence that some implementation approaches may not achieve previously published results or may actually cause new errors or even harm. This has lead to new initiatives to evaluate CPOE systems, which have been undertaken by both vendors and other groups who evaluate vendors, focused on CPOE vendor capabilities and effective approaches to implementation that can achieve benefits seen in published studies. In addition, an electronic health record (EHR) vendor certification process is ongoing under the province of the Certification Commission for Health Information Technology (CCHIT) (which includes CPOE) that will affect the purchase and use of these applications by hospitals and clinics and their participation in public and private health insurance programs. Large employers have also joined this focus by developing flight simulation tools to evaluate the capabilities of these CPOE systems once implemented, potentially linking the results of such programs to reimbursement through pay for performance programs. The increasing role of CPOE systems in health care has invited much more scrutiny about the effectiveness of these systems in actual practice which has the potential to improve their ultimate performance.

Background

Computerized physician order entry (CPOE) is an application that can be used to write orders either in the hospital or in the outpatient setting. It is linked with clinical decision support, which provides much of the value of implementing it. ^1,2 CPOE can be implemented either with or without a full electronic health record, although implementing a clinical data repository with results reporting is essentially a prerequisite. In the inpatient setting, it is often implemented before full electronic charting, because the value of doing so is higher.

Most recent estimates suggest that about 15% of hospitals in the United States are using CPOE, ^3,4 although a definitive study has yet to be done, and the proportion continues to rise. In many hospitals, while a CPOE application is in place, the proportion of physicians actually using it is low.

Outside U.S. hospitals, approximately 15% of U.S. providers appear to be using electronic health records. ⁵ Computerized ordering is potentially one of the greatest benefits of implementing an EHR, so that typically computerized prescribing is commonly a part of ambulatory EHR implementations.

Empiric CPOE Evaluations

A number of studies have assessed the impact of CPOE with respect to a variety of parameters, including costs of care, ⁶ medication safety, ^7,8,9 and quality. ^10,11 A landmark controlled trial by Tierney et al. at Regenstrief, found that CPOE was associated with a decrease in total costs of $887 per admission and in length of stay of 0.89 days. ⁶ At the time, the application included some decision support, including suggestions about specific sets of orders for certain diagnoses, and feedback about the costs of laboratory and radiology tests. The authors were not able to tease apart which specific interventions resulted in savings, as they were all implemented simultaneously.

A series of studies has demonstrated that computerization of drug ordering has improved medication safety. ^7,8,9 Early studies demonstrated that with even limited decision support, the serious medication error rate fell by 55% after the introduction of CPOE, and the overall medication error rate fell by over 80%. Subsequent studies have found that specific other types of decision support such as implementation of renal dosing ¹² and geriatric dosing ¹³ result in improvement as well. In patients with renal insufficiency, the percent of orders with an appropriate dose and frequency increased from 30% to 51% after implementation of a renal function-specific dosing decision support system and in addition length of stay fell by 0.5 days. ¹² Evaluation of suggestions about dosing of psychotropic medications in geriatric patients found a 34% improvement in agreement with recommended dosing guidelines and also a decrease in the in-hospital fall rate. ¹³

Other studies have evaluated the impact of CPOE on quality-related measures. One initial study suggested that reminders to deliver immunizations for pneumonia and influenza in hospitalized patients were ineffective, but in a follow-up study at the same institution a major benefit was found when immunizations were given unless the physician opted out. ¹¹ Another study evaluated the impact of a computerized guideline around prescribing of vancomycin, which is a drug that can be overused, leading to antimicrobial resistance, ¹⁴ and this study found that the guideline resulted in fewer vancomycin-days per provider.

Organizational Implementation Studies

To date, the majority of studies addressing issues related to the implementation of CPOE have been done at institutions that are organizational leaders in health information technology (HIT). These organizations include Partners/Brigham and Women’s Hospital, the Regenstrief Institute, LDS Hospital, and Vanderbilt. ¹⁵ These sites have provided the bulk of the original data on implementations of HIT including CPOE. However, these studies are somewhat limited in their generalizations by several factors: their CPOE systems are all home grown and not commercially available; these systems have been implemented and evolved over many years, and they have been implemented almost exclusively in academic teaching hospitals with little or no experience in community hospitals. In addition these organizations all used different approaches to evaluate their CPOE system implementations. ¹⁵

However, CPOE use is now growing rapidly in the non-teaching hospital setting, primarily in community hospitals. These hospitals are virtually all implementing commercial CPOE systems and several studies of CPOE implementations in this environment have been undertaken. ¹⁶ Initially these CPOE studies focused on various barriers to adoption of CPOE ¹⁷ and on factors associated with successful CPOE implementations. ¹⁸ The complexity, expense, and risks of failed CPOE implementation have been made clear by the experience at Cedar Sinai Hospital, although most CPOE implementation failures are not published. ^19,20 Given the increasing measurement of impact of these systems by healthcare organizations, there will be an increasing literature on the effect of CPOE implementation in community hospitals using commercial CPOE systems.

CPOE Benefit Realization Approaches

Several studies have emerged on the costs and potential benefits associated with the implementation of HIT systems including CPOE. ²¹ Many organizations initially assumed the benefits of CPOE would automatically occur with implementation and did not routinely measure the impact of CPOE beyond preliminary return on investment calculations. ²⁰ However, because of the greater understanding of the financial and safety risks of CPOE and the possibility of a CPOE implementation mishap or even failure ^22–25 organizations are more often routinely measuring clinical and financial benefits as part of their CPOE implementations. This is frequently done not only to assure that these potential benefits are actually achieved, but also as a means to further the effective adoption of HIT system applications such as CPOE by demonstrating positive impacts during the implementation process. 15,21,26–28 To wit, one organization in Canada attempted to manage the holistic impact of CPOE on the health care institution by developing a methodology that utilizes ongoing feedback to guide the CPOE implementation towards the satisfaction of stakeholder objectives. Stakeholders jointly defined quantitative and qualitative metrics for their objectives, established target value vectors for the metrics that represent acceptable implementation outcomes and specified evaluation milestones. These were used to compare pre- and post-CPOE implementation clinical performance, enabling a socio-technical feedback-improvement cycle. ²⁹

Based on the increasing experience with CPOE implementations outlined above it seem prudent for an organization implementing CPOE to consider measuring a number of parameters during a routine implementation. 20,21,27–29 These measures at a minimum would include:

• 1 Easily available metrics such as mortality rate and length of stay in areas in which implementation is done.

• 2 Performance on any quality measures targeted by CPOE such as delivery of pneumococcal vaccine, and other Hospital CORE Measures targeted by the Joint Commission on Accreditation of Healthcare Organizations.

• 3 Some measures of efficiency such as medication turn-around time or time to first dose of antibiotics in community acquired pneumonia

• 4 How many warnings or alerts go off of various types in medication ordering including allergy, drug–drug, and drug–laboratory, and how often they were heeded.

CPOE Vendor Studies

Because of increasing healthcare delivery organizational interest in achieving demonstrable benefits from electronic health record (EHR) implementations, HIT vendors have begun either funding studies of EHR benefits ²¹ or building their own database of potential and actual benefits realized from the implementation of their own proprietary EHR systems. For the past several years vendors have been collecting and recording provider-reported information about the benefits of their electronic medical record and practice management systems.

These benefits include those from CPOE as well as other applications within EHRs. Vendor-supplied benefits data are useful in helping prospective purchasers identify, estimate, and prioritize EHR benefits. The combination of consistently reported, quantitative benefits data from multiple vendor client sites and analysis of the factors influencing those benefits can help purchasers who are evaluating or implementing EHR systems. The benefits reported in these vendor databases are typically “basic”; they represent what has been seen and measured in relatively early use of the EHR products. They include numerous measures primarily focused on cost savings and other efficiencies as outlined in ▶. Much of this EHR vendor collected information is unsolicited and presumed to be anecdotal. It is compiled from numerous sources, and measured by the providers themselves for various purposes (usually not for publication), and may be of variable quality. Much of the information seems to be “low hanging fruit” that is easy to measure, as opposed to results of disciplined studies. ³⁰

Table 1.

Table 1. Sample EHR Vendor Database Benefit Measures

1.	Overall return on investment for EHRs
2.	Increased revenues associated with EHRs
3.	Cost reductions with EHR implementations in the areas of:
	a. Paper chart cost savings
	b. Transcription cost savings
	c. Billing cost savings
4.	Total medication use cost savings
5.	Procedure cost savings
6.	Referral cost savings
7.	Provider support cost savings
8.	Provider productivity enhancements
9.	Improved staff efficiency
10.	Data mining cost savings
11.	Physical space savings
12.	Improved quality of care process and outcome measures
13.	Reduction in medical errors
14.	Reduction in liability expenses

More significant improvements in clinical and safety outcomes are expected as providers become more familiar with these tools and as more decision support is introduced; given the recent problems with CPOE implementations ^31–33 vendor collection of benefits related to CPOE will likely increase. Despite their inherent limitations these vendor databases offer one opportunity to evaluate CPOE systems.

CPOE HIT Vendor Comparisons

In contrast to the vendor collected and reported benefits from the use of their proprietary EHR systems, there are several organizations that externally evaluate these commercial vendor EHR systems. These comparisons are often proprietary and made available to EHR purchasers and others for significant fees. ^16,34–36 The information in these databases is usually collected directly from organizations where these proprietary vendor products have been implemented, with little or no input from the vendors themselves. The methodologies used to conduct these evaluations are varied and not fully transparent. These approaches usually include off-site evaluations conducted with a variety of surveys, telephone interviews, and conference calls to collect information from clients about their vendor product implementations. On-site evaluations are not usually part of the approach used in these vendor evaluations and this lack of on-site validation may limit the accuracy of the data in these vendor comparative databases. These evaluations focus heavily on vendor product installations and usage, and organizational satisfaction with the vendor products and implementations. One of the most extensive evaluations of CPOE implementations has been conducted by KLAS ¹⁶ in its CPOE digest reports. These reports have evaluated the extent of CPOE usage across 10 different commercial vendor CPOE products across 237 inpatient sites and 83 outpatient sites. Beyond detailed usage statistics by vendor product, these reports track some process measures including: usage of decision support alerts, usage of bar-coding, system response time, system reliability, and user satisfaction among other measures. However, the impact of these vendor CPOE systems on clinical quality and safety process measures, as well as patient outcome measures, is not reported. Another example of CPOE specific evaluations of vendor products is the Medication Safety Tool Report by Five Rights Consulting that covers CPOE as well as bar-coding and other medication safety technologies. ³⁵

Finally, there are other databases of vendor HIT implementation experiences such as the Dornfest/Health Information Management System Society (HIMSS) Analytics database. ³⁶ Like the other vendor evaluations the primary assessment measures relate to utilizations of vendor products with limited tracking of clinical or safety outcomes associated with usage of CPOE or EHR products. Because of the variability of these vendor-specific EHR clinical and safety benefit evaluations, HIMSS has formed a task force to begin collecting and standardizing benefits associated with EHR products including CPOE. ³⁰ This effort may lead to increasing standardization of EHR benefits and assessment and greater collection of safety and quality information as a part of the process of these CPOE vendor evaluations.

CPOE/EHR Certification Approaches

Certification Commission for Health Information Technology (CCHIT)

A broad range of healthcare payers, from the government to the private sector, is now offering, or preparing to offer, financial incentives for EHR adoption. In particular, several pay-for-performance initiatives currently include incentives for the use of EHR and CPOE. But there is serious concern that a mechanism does not currently exist to ensure that commercially available EHRs and related technologies are robust enough to deliver the anticipated and published benefits. Physician specialty associations in particular have devoted much effort to highlighting the uncertainty faced by their members as EHR buyers; outlining the problems with EHR product suitability, quality, interoperability, and data portability and highlighting the fact that these capabilities can often be very difficult to judge. Certification of commercial vendor EHR products has the potential to open up the flow of HIT incentives and simultaneously reduce the risk for HIT purchasers, acting as a doubly powerful catalyst to accelerate adoption. At the urging of the National Coordinator for Health Information Technology, AHIMA, HIMSS, and The Alliance (formerly NAHIT), have joined forces to launch The Certification Commission for Healthcare Information Technology, a private sector effort to bring product certification to the EHR marketplace. ³⁷ The purpose of The Commission is to create an efficient, credible, sustainable mechanism for the certification of healthcare information technology products. The goals of product certification are: to reduce the risk of HIT investment by providers, to ensure interoperability of HIT products with emerging local and national health information infrastructures, to enhance the availability of HIT incentives from public and private purchasers/payers, and to accelerate the adoption of robust, interoperable HIT throughout the U.S. healthcare system. CCHIT began its work in 2004 with start-up funding from the three organizations and in September 2005 the CCHIT was awarded $7.5M over 3 years by the Department of Health and Human Services to develop and assess EHR and network certification criteria and inspection process. The commission timeline for vendor product certification as currently stated is to: develop, pilot test, and assess certification of EHR products for ambulatory care settings by September 2006, develop, pilot test, and assess certification of EHR products for inpatient care settings by September 2007, and to develop, pilot test, and assess certification of infrastructure or network components through which EHRs interoperate by September 2008. ³⁷

To date the CCHIT has taken the EHR DSTU (Draft Standard For Trial Use) from Health Level Seven (HL7) as the basic framework and created a set of certification criteria for 2006 and a roadmap of likely criteria for certification in 2007 and 2008. For testing CCHIT has developed a series of test scenarios incorporating the criteria (formerly called Use Cases) which describe realistic clinical situations around which an observed demonstration is structured and a series of test scripts, which detail step-by-step procedures that will be followed during the test. CCHIT piloted certification of ambulatory EHRs in March–April 2006 and launched actual certification in May 2006. The first group of vendor products deemed certified was announced on July 18, 2006. The pilot included a careful examination of both inter-rater reliability and validity, on the basis of which procedures, test scenarios, and test scripts were revised as necessary. ³⁷

Inspection of actual vendor products for compliance with CCHIT criteria occurs in a series of three steps. In the first step vendors self-attest by supplying documentation of their system and formally signing an attestation as to accuracy. The second step involves jury-observed demonstrations of the vendor EHR products, according to the test scenarios and scripts, running at vendor facility with jurors and proctors observing via simultaneous Web conference/audio conference. Each vendor sets up a test environment that replicates the live environment of its EHR system, and provides appropriate personnel during the observed demonstration portion of pilot testing to execute all the procedural steps in the published test scripts, as well as for review of elements contained in the technical testing portion. In the third and last step independent technical tests of vendor products are performed using off-site laboratories under the oversight of independent testing organizations with the test scripts outlined above. ³⁷

Product certification will play an important role in pushing vendors to enhance EHR products and become an important factor purchasers take into account. Payers and other groups who offer incentives and/or publish information about performance and/or IT infrastructure used by providers are very likely to incorporate certification in their programs. However, certification only applies to EMR products—the tool providers can use—but does not address how it is used to improve performance.

Leapfrog Group and National Quality Forum

The 1999 Institute of Medicine report “To Err Is Human” raised public awareness of the frequency and cost of adverse drug events in medicine. In response, in November 2000 a coalition of health care purchasers announced the formation of the Leapfrog Group, an organization dedicated to making “great leaps forward” in the safety and quality of health care in America. Their first target—computerized physician order entry—was selected specifically for its potential to reduce harm to patients from medications. ^38,39 The Leapfrog Group has subsequently incorporated into their standards the Hospital Safe Practice Survey, a nationwide hospital survey of 30 safe practices identified by the National Quality Forum, ⁴⁰ one of which is inpatient CPOE. This safe practice standard requires that hospital prescribers should enter hospital medication orders using an automated information management system that:

• • Is linked to prescribing error prevention software;

• • Enables the review of all new orders by a pharmacist before administration of the first dose of the medication;

• • Permits the notation of all pertinent clinical information about the patient, including allergies, in one place;

• • Categorizes medications into families (e.g., penicillin and its derivatives) to facilitate the checking of medications within classes and retains the information over time;

• • Internally and automatically checks the performance of the information system;

• • Requires prescribers to document the reasons for any override of an error prevention notice;

• • Performs dose range checks to prevent excessive doses from being inadvertently ordered; and

• • Distinguishes between different doses of the same medication used for multiple indications, including off-label uses.

As opposed to the EHR vendor certification process conducted by CCHIT, this approach certifies CPOE as an implemented safe practice in individual hospitals or healthcare delivery organizations. So far more than 1000 hospitals have responded to the Leapfrog Hospital Safe Practices survey including the CPOE safe practice and a total of 76 hospitals have claimed to meet this NQF CPOE certification standard. ⁴⁰ This NQF CPOE safe practice standard is currently undergoing revision to update the standard and enhance the criteria for compliance with effective CPOE especially in the area of clinical decision capabilities, and to link it more closely to the Leapfrog CPOE inpatient standard. ⁴⁰

The Leapfrog Approach to CPOE Testing

Because the decision support in an individual organization CPOE installation can be highly variable, the need for an independent evaluation process has been apparent for some time. In addition, reports highlighting the potential for CPOE to introduce significant errors, and thereby impairing patient safety, make the need for such evaluation more pressing than ever. ²² The mechanisms by which computerized physician order entry can improve the safety, quality, and efficiency of care have been discussed extensively in the literature. ²

The Leapfrog inpatient CPOE standard included a requirement that the organization operating CPOE demonstrate via a test that their inpatient CPOE system can alert physicians to at least 50% of common serious prescribing errors. The Leapfrog Group desired that the CPOE evaluation methodology promote industry development and adoption of functions to improve safety and quality; and serve as a quality improvement tool for hospitals as well as a method of certification to a standard. The methodology should test for sophisticated, leading edge clinical decision support as well as basic, commonly available decision support. It should provide feedback to hospitals about their system’s clinical decision support capabilities and performance, including excessive alerts which may result in “alert fatigue,” causing clinicians to ignore decision support or press for its deactivation, thereby decreasing the overall effectiveness of the system. Systems should also be evaluated for functions that promote efficiency and reduce waste, such as duplicate order checking.

In developing the Leapfrog CPOE test a framework was developed to include twelve different categories of CPOE-based decision support that could prevent prescribing errors that lead to adverse drug events. A scoring system was developed based on the known frequency and severity of adverse drug events. ³⁸ Simulated test patients and accompanying simulated test medication orders were developed to evaluate a CPOE system’s ability to intercept prescribing errors in all twelve decision support categories. Order set development began with an initial set provided by the Institute for Safe Medication Practices, previously used by them for evaluating pharmacy information systems. This set was modified extensively to adapt it to the types of decision support appropriate for CPOE as shown by industry experience and literature on the kinds of medication ordering errors most likely to result in ADEs. ^6,42,43 Another set of test patients and orders was developed for pediatrics based on the literature ⁴⁴ and expert opinion. The resulting master order set consisted of over 130 adult and over 50 pediatric test orders addressing nine categories of erroneous medication orders plus three order types that evaluate system efficiency: nuisance alerts, cost of care, and corollary orders.

The orders and test scenarios downloaded by a hospital taking the test represent a subset of the orders from the master order set in each decision support category. Selecting these randomly “on the fly” from the master order set makes it unlikely that a given site will be able to anticipate the specific orders that will be tested, and restricts the proportion of test patients and orders that are released publicly at any given time and location, further protecting the content of the test material. In addition the order set will be periodically reviewed and revised and modified, and new orders and scenarios introduced to maintain the validity and currency of the test.

The scoring system interprets the raw test results reported by hospitals that reflect the relative importance of each type of decision support for prevention of harm to patients. To achieve this, scores needed to reflect the elements of both severity of a potential ADE not intercepted by the system, and its likely frequency. Thus an event that happens rarely but is catastrophic should have a high score attached; an event that is less severe but likely to happen often might similarly deserve a high score. Likely frequency of ADEs that would result from specific ordering errors was determined from several large published and unpublished studies performed by automated ADE surveillance, a method superior to voluntary reporting for detecting ADEs. ^42,45,46 Frequency was scored on a three point scale (most frequent, less frequent, least frequent). Severity determination was based on expert opinion among our advisors, and described as life threatening, severe, significant, or not significant. A matrix was designed to determine summary scores from the attributes of severity and frequency. ³⁸

The Leapfrog CPOE evaluation methodology simulates different clinical scenarios using a wide variety of test patients and orders to evaluate how a hospital’s computerized physician order entry system responds to unsafe medication ordering and clinical situations. A Web-based application was developed to allow hospitals to self-administer the evaluation. ³⁸ The hospital taking the evaluation downloads a list of test patients with various demographic characteristics, medical conditions, and medication regimens and programs them into their CPOE testing environment. They then download a series of test orders to be entered against the test patients. The CPOE system’s response to the entered order is then noted and reported through the online evaluation system. There is a specific amount of time that is allotted between these steps to prevent “just-in-time” programming of the system to improve performance on the test. At the conclusion of testing, the hospital receives an overall score, and scores describing performance in specific clinical decision support categories (▶). This feedback assists the hospital in selecting areas for new implementation of decision support or improvement of their current CPOE system, yet also provides Leapfrog and other purchasers with an objective evaluation of the safety performance of the hospital’s CPOE system.

Table 2.

Table 2. Leapfrog CPOE Evaluation Test Clinical Decision Support Categories

Therapeutic Duplication	Therapeutic overlap with another new or active order; may be same drug, same drug class, or components of combination products.
Single and Cumulative Dose Limits	Specified dose that exceeds recommended dose ranges; will result in a cumulative dose that exceeds recommended ranges; can also include does limits for each component of a combination product.
Allergies and Cross Allergies	Allergy has been documented or allergy to other drug in same category exists.
Contraindicated Route of Administration	Order specifying a route of administration that is not appropriate for the identified medication.
Drug–Drug and Drug–Food Interactions	Results in known, dangerous interaction when administered together with a different medication or results in an interaction in combination with a drug or food group.
Contraindications/Dose Limits Based on Patient Diagnosis	Contraindication based on patient diagnosis or diagnosis affects recommended dosing.
Contraindications/Dose Limits Based on Patient Age or Weight	Contraindication based on age or weight.
Contraindications/Dose Limits Based on Laboratory Studies	Contraindication based on laboratory studies or for which laboratory studies must be considered for dosing.
Contraindications/Dose Limits Based on Radiology Studies	Contraindication for this patient based on interaction with contrast medium (in ordered radiology study).
Corollary	Intervention that requires an associated or secondary order to meet the standard of care (prompt to order drug levels during medication ordering).
Cost of Care	Test that duplicates a service within a timeframe in which there is typically minimal benefits from repeating the test.
Nuisance	Order with such a slight or inconsequential interaction that clinicians typically ignore the advice/prompt.

Future Recommendations

The Leapfrog CPOE evaluation methodology will complement the National Quality Forum’s hospital safe practices survey questionnaire, and its implementation will complete the evaluation component of the initial Leapfrog CPOE standard. It will be made publicly available with the official release of the updated Leapfrog Safety Surveys on March 1, 2007, any hospital or ambulatory clinic will be able to use this test to evaluate specific decision support capabilities within its implemented CPOE systems. While this is the first test developed to certify Electronic Health Record applications in actual use, it is likely to be followed by tests of other EHR applications. While the use of simulation to evaluate EHR systems is clearly new, it has been used to design and develop EHR systems ^47,48 but only in limited formats and settings to date. It may offer the potential for other testing approaches to evaluate many aspects of EHRs in actual use rather than software on the shelf. Nonetheless this approach can complement efforts that are already underway to certify electronic health record products at the vendor level by CCHIT. Certification of EHR products will help ensure that systems deliver the benefits that providers, payers, purchasers and government officials seek and expect. A certification process will provide a clear definition of product capabilities and compatibilities. It will also ensure interoperability of these products with emerging local and national health information infrastructure. Hopefully, this certification process will reduce information technology investment risk for providers, and encourage payers/purchasers to offer incentives for investment in information technology. Both the CCHIT certification and Leapfrog CPOE testing programs should be considered works in progress, both used empiric approaches to develop each of their unique methods and the degree to which these criteria will actually impact safer more reliable care is unclear. It is essential that these certification approaches be tested and evaluated with respect to their impact on EHR systems, their impact on patient care, and their impact on patient outcomes. The results of these evaluations should be used to further refine and improve these novel approaches to evaluation and certification of CPOE and EHR systems.

In addition to purchasing a certified EHR product and periodically reassessing it, organizations will need approaches to be able to assess how well their applications are functioning in an ongoing way, as well as tools for updating them when appropriate, although the degree to which this will be done by the organizations versus by the vendor is uncertain. This must be done much more frequently for the decision support associated with CPOE than is needed for most other applications. For example, at a minimum, we believe that organizations should routinely be able to assess how often each type of alert is going off, and what actions clinicians are taking in response to these alerts. These data must be available both at the aggregate and line-list levels, and organizations should be able to identify individuals with unusually high override patterns, especially for certain specific alerts. To achieve these goals, organizations will have to change their focus from implementation of EHR systems to ongoing monitoring and optimization of these EHR systems. This will require not only a change in thinking but also a new allocation of resources and effort which most organizations have neither considered nor allocated to date.

Conclusion

With the increasing understanding of the complexity, variability, and inherent risk in the implementation of complex EHR systems with interventional applications such as CPOE, has come greater interest in evaluating these expensive and poorly studied systems. The recent report of a CPOE system whose implementation may have actually increased mortality in a children’s hospital ²⁴ will only increase demands that these systems be held accountable. To date, much of the evaluation these CPOE systems has been done at leading medical informatics research organizations using locally developed and controlled software. This has been extremely valuable, and will continue to be useful in the future for development and testing of new applications and decision support. However, most organizations will implement commercial software, and more studies of the impact of this software especially in community hospitals are essential, both with respect to basic safety and effectiveness, and implementation. This has led to a diverse array of new methods to evaluate these systems from local organizational implementation studies to more formal benefit measurement programs. Vendors have responded by building their own evaluation databases and a whole industry has developed to compare vendor products. This has been supported by a large federal effort to certify EHR products, but also by various pay for performance initiatives that are developing their own certification approach for CPOE and EHR capabilities. One issue is that to date these approaches have targeted primarily vendor “on-the-shelf” products, not what is actually installed, and it will be important to broaden this focus over time. All organizations will need to perform ongoing evaluation of their CPOE applications and their EHR if the potential benefits of these technologies are to be actually realized. ⁴¹