Functional Characteristics of Commercial Ambulatory Electronic Prescribing Systems: A Field Study

Abstract

Objective: To compare the functional capabilities being offered by commercial ambulatory electronic prescribing systems with a set of expert panel recommendations.

Design: A descriptive field study of ten commercially available ambulatory electronic prescribing systems, each of which had established a significant market presence. Data were collected from vendors by telephone interview and at sites where the systems were functioning through direct observation of the systems and through personal interviews with prescribers and technical staff.

Measurements: The capabilities of electronic prescribing systems were compared with 60 expert panel recommendations for capabilities that would improve patient safety, health outcomes, or patients' costs. Each recommended capability was judged as having been implemented fully, partially, or not at all by each system to which the recommendation applied. Vendors' claims about capabilities were compared with the capabilities found in the site visits.

Results: On average, the systems fully implemented 50% of the recommended capabilities, with individual systems ranging from 26% to 64% implementation. Only 15% of the recommended capabilities were not implemented by any system. Prescribing systems that were part of electronic health records (EHRs) tended to implement more recommendations. Vendors' claims about their systems' capabilities had a 96% sensitivity and a 72% specificity when site visit findings were considered the gold standard.

Conclusions: The commercial electronic prescribing marketplace may not be selecting for capabilities that would most benefit patients. Electronic prescribing standards should include minimal functional capabilities, and certification of adherence to standards may need to take place where systems are installed and operating.

Introduction, Background, and Hypothesis

Ambulatory electronic prescribing is expected to improve both the safety and the efficiency of pharmaceutical use.^1,2 However, the ability of electronic prescribing systems to achieve these goals will depend largely on their functional capabilities and on how those capabilities are integrated into clinical practice.³ Several studies have demonstrated significant benefits from electronic prescribing,^4,5,6,7 but emerging evidence also shows that electronic prescribing systems can have unintended effects.^7,8 Although the evidence for electronic prescribing has come largely from “home-grown,” customized systems installed at academic medical centers,⁹ one series of studies showed that a commercial electronic health record (EHR) system, which included electronic prescribing, contained more complete medication lists than did comparable paper records,¹⁰ and it also delivered highly effective immunization reminders.¹¹ In addition, one commercially available system was included in a case series that compared the capabilities of five inpatient computerized physician order entry systems.¹² Overall, however, little is known about the capabilities and the potential value that may be available from the growing number of ambulatory electronic prescribing systems that are now available commercially. Since policies are now being shaped to promote the widespread adoption of ambulatory electronic prescribing in the United States,^1,13 it is important to expand the knowledge base about commercial electronic prescribing systems.

In light of the limited evidence available for evaluating electronic prescribing systems,^3,14 we previously conducted a modified Delphi expert panel process to develop recommendations for the capabilities that electronic prescribing systems should have if they are to improve patients' health outcomes and reduce their costs.¹⁵ This process resulted in a set of 60 specific functional recommendations for electronic prescribing systems. The panel expected 52 of the recommended features to have significant positive effects on patient safety and health outcomes, and 18 of the recommended features to have significant positive effects on patients' ability to manage their costs. (Ten of the recommendations were expected to have positive effects on both dimensions of care.)

In this article, we report the results of a field study aimed at determining the extent to which these 60 recommendations have been implemented among a purposive sample of commercially available outpatient electronic prescribing systems. Our research questions were as follows: (1) How much do commercial electronic prescribing systems differ from one another in their implementation of capabilities that would be important for advancing patients' interests? (2) Which potentially important electronic prescribing capabilities have not been implemented among commercial systems? We hypothesized that the range of capabilities among systems might be relatively narrow if the commercial marketplace was inherently selecting for systems designed to promote patient safety and cost management, and, conversely, that the range would be relatively larger if systems could gain significant adoption without many of these features. We expected that the potentially important capabilities that have not been implemented among commercial systems would help to identify technical and organizational challenges in need of further informatics research and systems redesign.

Methods

Selection of Electronic Prescribing Systems

Using a published directory,¹⁶ press releases, and other sources, we compiled a list of 129 candidate vendors of electronic prescribing systems. Two or more reviewers examined each company's Web site for evidence of original electronic prescribing products. When necessary, we telephoned companies to confirm or clarify our interpretation of their Web site. This screening process identified a total of 58 distinct, commercially available electronic prescribing products, supplied by 51 vendor companies (▶).

Figure 1.

Results of the site selection process. Purposive sampling¹⁷ involved selecting example products to represent each of the major product types, as described in Methods section. EHR = electronic health record; e-Rx = electronic prescribing; IT = information technology.

For each electronic prescribing product, we conducted a telephone interview with vendor representatives to determine the product's basic features, technical architecture, and intended practice environments; the number of sites where the product was installed; and the total number of current users. Based on these data, we classified each product according to the characteristics shown in ▶. Of the 58 electronic prescribing products identified, 29 products (from 26 vendors) satisfied the study's inclusion criteria of being used in outpatient settings, at a minimum of 50 sites, or by a total of 1,000 users.

Table 1.

Characteristics of Electronic Prescribing Systems Screened, Eligible, and Selected for Site Visits

Characteristics	All Candidate Products (N = 58)	Outpatient Products Having >50 Sites or 1,000 Users (N = 29)	Products Selected for Site Visits (N = 10)
Full EHR systems (vs. non-EHR systems)	42 (72%)	24 (83%)	5 (50%)
ASP model architecture (vs. client-server)	19 (33%)	8 (28%)	4 (40%)
Supports handheld user interface	25 (43%)*	10 (34%)	5 (50%)
Integrates laboratory data	31 (57%)†	20 (69%)	6 (60%)
Provides safety alerts	43 (75%)*	22 (76%)	8 (80%)
Provides formulary information	46 (81%)*	25 (86%)	10 (100%)
Suggests medications based on diagnosis	16 (29%)‡	11 (38%)	2 (20%)
Output options
Print	56 (98%)*	29 (100%)	10 (100%)
e-Fax	49 (86%)*	27 (93%)	9 (90%)
Electronic data transmission	33 (58%)*	18 (62%)	5 (50%)

EHR = electronic health record; ASP = application service provider, a Web-based service rather than locally installed client-server software.

Percentages are calculated based on reports of vendor representatives in initial telephone interviews.

^*Data not available for one product.

^†Data not available for four products.

^‡Data not available for two products.

From the 29 eligible products, we selected a sample of ten for site visits, using purposive sampling¹⁷ to ensure that examples of each major product category were included. The features used for categorization were EHR systems versus freestanding, non-EHR systems; Web-based application service providers versus client-server systems; and applications available on a handheld platform versus those available only on a desktop platform (▶). In each of the eight possible subcategories defined by these features (for example, EHR systems with a client-server architecture plus a handheld user interface), we selected the system or systems that appeared to have achieved the broadest adoption.

Assessment of Electronic Prescribing Systems

We reinterviewed the vendors of the ten selected systems regarding their implementation of each expert panel recommendation.¹⁵ Vendors were also asked to identify an outpatient office where their system was in active use. Each of these sites was contacted and a site visit was scheduled to include interviews with a professional who was using the system to prescribe and with a technical staff person who was responsible for maintaining the system. We expected vendors to identify their best sites, making the results reflect the best possible rather than the average for each electronic prescribing system.

Each site visit was conducted by at least two of the four investigators with relevant professional experience (CJW, RSM, RCM, and DSB), following a structured interview guide. Interviews began with open-ended questions about the office's goals for electronic prescribing and their satisfaction with the system to date. Technical staff people were asked about the system's architecture and support, including privacy and security features. Prescribers were asked to demonstrate the system's major features, including the creation of prescriptions for two standard clinical scenarios. Interviewers then asked additional questions and made additional observations as necessary to determine each system's implementation of the expert panel recommendations. Interviewers independently recorded their observations and additional notes on the preprinted interview guides. Five of the site visits occurred before the complete set of recommendations was finalized. For these systems, we conducted additional telephone interviews with the original subjects to assess their implementation of the 11 additional recommendations that were generated in the final round of the expert panel process. For these five systems, we did not conduct additional vendor interviews; thus, all data on vendor claims were collected before the site visit took place for each system. Site visits were completed between July 2002 and July 2003. We paid physicians $100 and technical staff $75 for participating. The Institutional Review Boards at RAND and UCLA approved the study.

Analysis

For each of the 60 expert panel recommendations, we developed written conformance criteria. These criteria specified the observations necessary to categorize an electronic prescribing system as having implemented the recommendation fully, partially, or not at all. In addition, some recommendations could be categorized as inapplicable for certain systems, for example, the recommendation that systems prioritize safety alerts (no. 31 in the Appendix) was not applicable for systems that lack any safety alerts. The four investigators with relevant professional experience (CJW, RSM, RCM, and DSB) met after each site visit to compare observations recorded in the interviews with the conformance criteria and to reach a consensus decision regarding the system's implementation of each recommendation. We used a two-tailed Wilcoxon rank-sum test to compare implementation levels of different system types. For the five vendor interviews that addressed all 60 of the recommendations, the investigators counted the discrepancies between the vendors' claims for each system and the features that were observed in the related site visit. Finally, the four interviewers reviewed the collected notes taken for all open-ended questions to identify the sites' perceptions about the process of implementing electronic prescribing and their achievement of the goals that they set out for electronic prescribing.

Results

Characteristics of Electronic Prescribing Products

▶ compares characteristics of the included and excluded systems as well as characteristics of the final site visit sample and the products not sampled. During the site visit period, five of the 29 eligible products were withdrawn from the market because the vendor either ceased operation or ceased to offer the product. One of these products was among the ten in the final site visit sample. Complete data had been collected for this product, and these data were included in the subsequent analyses.

Implementation of Electronic Prescribing Recommendations

The Appendix shows the extent to which each of the 60 electronic prescribing recommendations was implemented among the ten selected systems. Twenty-two recommendations were inapplicable to at least one system. For example, the nine recommendations regarding specific features for alerts and messages (recommendations 29–37) were not applicable to the two systems that did not produce any alerts or messages to prescribers. Overall, 49.8% of the recommendations were fully implemented when they were applicable. Another 12.6% of the recommendations were partially implemented, when applicable.

Nine recommendations were not implemented by any of the systems, even partially. These included recommendations that would require electronic prescribing systems to handle prescription fulfillment data (recommendations 10, 47, and 48), to use more complex drug benefit data (recommendation 22), and to use more advanced drug knowledge bases (recommendations 26 and 49). Eleven recommendations were fully implemented by all the systems for which they were applicable. These tended to be less complex recommendations, such as allowing medications to be prescribed without forcing the entry of a diagnosis (recommendation 14). The remaining 40 recommendations were implemented by a portion of the systems. For example, five of the ten systems provided a current medication list (recommendation 7), and five of the ten systems provided medication suggestions based on the patient's diagnosis (recommendation 13). Although eight of the ten systems provided basic safety alerts based on patient allergies or current medications, only one system fully supported the alerting recommendation (recommendation 27) by also providing interaction alerts based on medical conditions and laboratory findings.

The expert panel that developed the recommendations used in this study also produced quantitative ratings for each recommendation's expected effects.¹⁵ We found no relationship between these ratings and the implementation of the recommendations—the recommendations expected to have greater benefits for patient safety and health outcomes were not implemented more frequently than those expected to have lesser effects.

Variability in Capabilities Implemented by Individual Systems

▶ shows the extent to which each of the ten systems implemented the subset of 52 recommendations that were rated by the expert panel as clearly beneficial for improving patient safety and health outcomes. System A in ▶ had implemented the most recommendations, having full support for 67% and partial support for another 12% of the applicable capabilities. System J had implemented the fewest recommendations, having full support for 29% and partial support for an additional 3% of the recommendations. Considering all 60 of the recommended capabilities, system A had implemented 64% fully and another 13% partially, and system I had implemented 26% fully and another 16% partially and system J had implemented 28% fully and another 5% partially.

Figure 2.

Full and partial implementation of patient safety and health outcomes recommendations by individual products. The ten systems that were assessed in site visits are each represented by a bar, lettered A to J. The height of each bar represents the proportion of applicable recommendations that each system implemented, among the subset of 52 recommendations for improving patient safety and health outcomes.

Implementation of Recommended Capabilities by Electronic Health Record Versus Stand-alone Systems

Prescribing systems that were part of EHR systems implemented more recommendations than did stand-alone non-EHR systems (▶). Considering all 60 recommendations, the median EHR-based system fully implemented 60%, whereas the median non-EHR system fully implemented 35% (p = 0.09). Including partial and full support together, median implementation levels were 72% for EHR systems and 46% for non-EHR systems (p = 0.06). Considering just the subset of 52 recommendations expected to improve patient safety and health outcomes, median implementation levels were also higher for EHR systems, both for full (EHR median = 63%; non-EHR median = 37%; p = 0.06) and for full + partial implementation (EHR median = 74%; non-EHR median = 46%; p = 0.04). No significant differences were found when comparing Application Service Provider with client-server systems and systems available on handheld platforms with those not available on handhelds. Systems that integrate laboratory data were, with one exception, collinear with the EHR systems.

Table 2.

Percentage of Recommendations Implemented by EHR and Non-EHR Electronic Prescribing Products

	Full Implementation, %		Full and Partial Implementation, %
Recommendation Group (No. of Recommendations in Group)	EHR (N = 5)	Non-EHR (N = 5)	EHR (N = 5)	Non-EHR (N = 5)
Overall (60)	60	35	72	48
Patient safety and health outcomes subset (52)	63	37	74	46
Helping patients manage costs subset (18)	53	13	67	20
Reducing underuse subset (15)	47	18	60	18
Functional categories
Patient identification (4)	75	50	100	75
Access to patient historical data (8)	38	14	86	29
Medication selection (14)	57	33	62	50
Alerts and other messages (12)	55	30	73	60
Patient education (2)	100	0	100	0
Data transmission and storage (7)	67	60	75	67
Monitoring and renewals (5)	60	20	60	20
Transparency and accountability (2)	100	0	100	50
Prescriber-level feedback (2)	50	0	50	0
Security and confidentiality (4)	100	75	100	100

EHR = electronic health record.

We calculated the percentage of recommendations that each system implemented within each of the subsets shown. The values shown are the medians of the within-subset percentages among systems. For example, in the functional category of patient identification (containing four recommendations), the median EHR system fully implemented 75% of the recommendations and the median non-EHR system fully implemented 50% of the recommendations.

▶ also shows that the 18 recommendations rated by the panel as clearly beneficial for patients' costs were implemented somewhat less often than the “patient safety and health outcomes” recommendations. Another subset of 15 recommendations that would improve health specifically by reducing the “underuse” of beneficial medications were also implemented less often than other recommendations. In the Appendix, the recommendations belonging to these subsets are indicated with ^† and ^‡ symbols.

During the expert panel process, ten functional categories were used to organize the 60 electronic prescribing recommendations (as shown in the Appendix). ▶ shows the median levels of “full” and “full plus partial” implementation of the recommendations within each of these categories. The highest levels of support were in security and confidentiality, and the lowest levels of support were in access to patient historical data, prescriber-level feedback, and medication selection. Higher levels of implementation were observed among EHR systems across all categories.

Accuracy of Vendor Claims

Vendor interviews had addressed all 60 of the recommendations for five electronic prescribing systems. On average, these vendors claimed to support 63% of the recommendations, either fully or partially. Of these positive claims, 21% were contradicted in the site visits. In some cases, these discrepancies were the result of a decision at the site not to implement a feature. For example, two sites chose not to install formulary features even though the products offered this feature. However, most false-positive discrepancies were not due to the sites' choice. For the 37% of recommendations that vendors did not claim to support, site visits revealed that the systems actually had implemented the recommendation in 7% of cases. Some of these discrepancies were due to the local site's having added new functions to the vendor system. For example, one site altered the vendor's system to make it display the average wholesale prices of medications (recommendation 21). Another site engineered diagnosis-based medication lists (recommendation 13) into a commercial system that did not offer the feature. Treating the site visit findings as a gold standard, the vendor interviews had a sensitivity of 96% and a specificity of 72% for assessing a recommendation's implementation.

Qualitative Findings

All the practices that we visited had completed installation of a functioning system. Implementation had typically taken three to four months; large organizations and those needing to integrate with an existing legacy computer system tended to take longer. For most of the sites, the decision to purchase their particular system was based on information gathered at conferences and seminars with subsequent sales follow-up. A few clinics had served as beta-testing sites for the vendor. One site had hired a consulting firm to evaluate systems and to implement the chosen system.

All the sites viewed electronic prescribing as replacing “pencil and paper” prescribing, but beyond that, their concepts about the role of electronic prescribing varied. Some practices saw electronic prescribing mainly as a system for improving interactions with pharmacies. For these sites, the goal of electronic prescribing was to save labor by reducing pharmacy callbacks for illegible prescriptions and renewal requests. A few practices saw electronic prescribing as part of a broader attempt to automate multiple aspects of practice, including clinical documentation. The goals of electronic prescribing for these sites were to improve the safety and appropriateness of prescriptions based on comprehensive information about the patient, including laboratory data, medical history, and drug allergies. Better efficiency was generally seen as a by-product of these improvements.

In most cases, prescribers and technical staff believed that they did save labor by improving interactions with pharmacies, but few sites had explicitly documented any savings. All users of electronic prescribing believed that the system had increased patient safety and quality of care, even though most had not implemented it for that purpose. One site reported receiving a discount on their malpractice insurance premiums because of their electronic prescribing system.

Discussion and Conclusions

Advancing the adoption of interoperable health information technology (HIT) is a major federal policy objective, expected to increase the quality of health care while also reducing its costs.^18,19 To this end, the National Coordinator for Health Information Technology has created a detailed strategy for federal actions to advance the adoption of HIT.¹ The strategy involves bringing stakeholders together to develop HIT standards, fostering private certification organizations to enforce those standards, and then instituting performance accountability for provider organizations that would require data from standardized, interoperable HIT systems. Electronic prescribing is seen as an important first step toward more comprehensive HIT, and federal policy makers expect to drive its adoption by requiring Medicare Prescription Drug Plan Sponsors to offer electronic prescribing.^1,13,20

This study shows that available electronic prescribing systems fail to offer many functional capabilities that could have significant benefits for improving patients' health and reducing their costs. More importantly, these deficiencies varied a great deal among the systems studied. Given that 51 of the 60 recommendations were implemented by at least one of the sites that we visited, it seems that technical feasibility was probably not the major factor in the variable implementation that we found for most of the recommendations. We also found no relationship between the importance of a recommendation, as judged by our panel, and its likelihood of being implemented among the systems studied. Thus, we conclude that the commercial market for electronic prescribing products may not be selecting for the features that would be most important for helping patients.

Several factors may underlie the large variation we observed among systems. On the one hand, vendors and providers may be uncertain about the features that would most improve care. In this case, the variation among systems should tend to narrow over time as more experience and research accumulate to guide the electronic prescribing market. On the other hand, our qualitative results would indicate that different vendors may be responding to different goals for electronic prescribing among providers, with some being narrowly focused on office efficiency and others being more oriented toward improving the quality of care. A recent study found similar variance among providers in their attitudes toward the adoption of EHRs, with some being willing to bear substantially greater costs to implement more advanced systems.^21,22 Over time, it is possible that providers who are currently focused on office efficiency will move toward greater interest in quality. However, increasing financial pressures on physicians may work against this evolution.^23,24 Furthermore, while the costs of implementing electronic prescribing are borne largely by providers, the economic benefits from improved patient safety and formulary adherence accrue largely to insurers and patients.²

Pending federal actions will address some of the misaligned incentives associated with electronic prescribing. The Medicare Modernization Act (MMA) provides for limited federal funding of electronic prescribing and provides a safe harbor to enable funding of electronic prescribing by third party organizations.²⁵ Third party funding will likely enhance physician adoption by internalizing some of the benefits that third parties would reap, but it could also shift competition among electronic prescribing systems further away from patients' interests. Our panel's recommendations included several related to transparency and accountability, which are intended to prevent third parties from introducing prescribing biases that would not benefit patients. Although we found only a few instances in which systems did not comply with these recommendations, vendors have the capability to substantially influence prescribing decisions, and third party support for electronic prescribing is currently just beginning.

The MMA also requires the development of standards for electronic prescribing systems that will go into effect under the new Medicare program.²⁵ The National Committee on Vital and Health Statistics has now released initial recommendations for these standards, focusing primarily on issues related to data interchange.²⁶ Such technically oriented standards may enable systems to achieve more of the features that our expert panel recommended, but they would not directly affect the incentives that providers and vendors have to implement the electronic prescribing features that would most benefit patients. Overall, our findings indicate that federal standards for electronic prescribing could best advance patient safety, health outcomes, and health care efficiency by including a minimal set of functional capabilities along with the more technical standards for system interoperability.

The study also found substantial discrepancies between the capabilities that vendors claimed for their products and the capabilities that were actually identified in site visits. Some of these discrepancies were attributable to decisions made at the practice site not to implement features that were actually available from the vendor. These findings highlight the fact that vendors may not be fully aware of the details about how their systems are implemented. Thus, the certification processes envisioned in the federal strategic plan should not be based solely on vendor reports about their products or on demonstrations by vendors outside of an actual practice setting. Furthermore, certification may need to take place for individual provider organizations in addition to taking place at the vendor level.

Our finding that EHR-based systems implemented more recommended capabilities might be attributable to the more comprehensive patient information available in EHR systems as a by-product of the other functions that they offer. However, one non-EHR system implemented almost as many recommendations as any of the EHR systems, demonstrating that substantial improvements in care might be achievable without the more comprehensive process changes that an EHR system would necessitate.

Modeling studies have suggested that ambulatory electronic prescribing could yield substantial savings for provider organizations.²⁷ In our study, most sites were satisfied that electronic prescribing had improved their efficiency, but we found little explicit substantiation of this perception. Some sites reported savings in labor from reduced telephone interactions with pharmacies, and one reported savings on malpractice premiums. However, none of the sites had formally assessed their return on investment.

Finally, several of the recommendations that were not implemented by any system point to important areas for further informatics research and development. Three of these recommendations would require electronic prescribing systems to accept and process prescription fulfillment data to help providers identify and solve patient adherence problems. Further progress toward achieving these capabilities may require the development of standard protocols for communicating pharmacy claims data back to the prescribing system that originated the prescription. Further research may also be needed on methods for analyzing and presenting fulfillment data in ways that enable productive conversations about adherence between patients and providers. Another recommendation that was not implemented at all would require electronic prescribing systems to obtain data on the amount that a patient has currently spent under a fixed drug benefit cap. This recommendation will grow in importance when the new Medicare prescription drug benefit begins in 2006. Since benefit cap information is currently available in the systems that pharmacies use to adjudicate claims at the point of sale, it should be possible to also supply this information to electronic prescribing systems. However, further research and development may be needed into methods for obtaining benefits data that are accurate at the time of prescribing.

Our study has several limitations. Vendors probably tried to identify their best sites for our data collection, so our findings may underrepresent the deficiencies of an average electronic prescribing installation. On the other hand, if vendors have been adding features since we collected data in 2002–2003, then our findings could overrepresent the actual deficiencies in today's market. We assessed implementation of the 11 recommendations generated in the last round of the panel process by telephone interview rather than by site visit for five of ten systems. Our assessments of these recommendations (denoted by # in the Appendix) may therefore be somewhat less accurate. However, these specific recommendations were generally not difficult to assess by telephone. Finally, we studied a purposive sample rather than a random sample of the 29 systems that met our eligibility criteria, but a random sample of such a small and heterogeneous population could not be considered representative. More importantly, our study was designed to identify the range of features available from commercial products, not to estimate the average set of features being offered. Furthermore, even a highly precise estimate of the average features available in commercial electronic prescribing systems would have limited usefulness because of the substantial ongoing turnover among vendors, with some products being withdrawn from the market every year and almost as many new, potentially less mature, products entering. Since the incentives driving the adoption of electronic prescribing probably have not changed, it is likely that the variation among the systems available today remains large.

In conclusion, our findings suggest that standards for electronic prescribing should include a set of minimal functional capabilities that would guarantee a standard, minimal level of support for patient safety and provide protection against biases that could be introduced on behalf of third party interests. As new electronic prescribing features are developed in the future, a process will also be needed to keep functional standards updated. Finally, more fundamental informatics research is needed on methods for using prescription fulfillment data and drug benefit data in electronic prescribing systems and on increasing providers' incentives for pursuing quality improvement through HIT.

Appendix 1. Implementation of Each Recommendation Among the Final Sample of Ten systems

	Recommendation	Fully Implemented	Partially Implemented	Not Implemented	Not Applicable
	Patient identification and data access
1*	A minimal set of patient-identifying information (name, gender, and date of birth or age) should be visible in the user interface throughout the process of creating a prescription.	4	4	2	0
2*	The system should have the capability of importing patient identification and demographic data from an electronic health record (EHR) or practice management system (PMS) used by the health care organization.	7	1	2	0
3*	The system should provide a method for manual entry of patient identification and demographic data when importing this information from an EHR or PMS is not possible.	7	0	3	0
4*	The system should permit records created under separate identities for the same patient to be merged or treated as one patient by a system administrator.	6	1	3	0
	Current medications/medication history
5*†‡	The system should be capable of extracting patient data for decision support from external sources including pharmacy, hospital, laboratory, and EHR systems.	0	5	5	0
6*§	The system should indicate (to the prescriber) when an external interface that provides data for (automated) decision support is not operational.	0	1	1	8
7*†‡	Prescribers with care responsibility for the patient should be able to review the patient's complete current medication list, based on open prescriptions from all other clinicians.	5	2	3	0
8*†	Prescribers should be able to review all nonprescription medications that the patient is currently taking, including over-the-counter and alternative medications.	7	0	3	0
9*	The system should provide a means for entering medications that the patient is currently taking that have not been prescribed through the system and are not available through external interfaces.	8	0	2	0
10*†‡	Prescribers should be able to review a summary by drug class of the patient's medication history including drug names, dosages, start and end dates, and adherence gaps.	0	0	10	0
11*‡	Prescribers should be able to retrieve details of individual past prescriptions including dosages, prescribing dates, and dispensing dates.	0	10	0	0
12*§	The system should support orders to discontinue currently open prescriptions with a message sent to notify the original prescriber of the discontinuation.	1	7	2	0
	Medication selection
13*†‡	The system should allow viewing a list of medications appropriate to the diagnosis when a diagnosis is entered.	5	0	5	0
14*	The system should allow efficient prescribing without the entry of a diagnosis and with the entry of speculative or tentative diagnoses.	10	0	0	0
15	The system should provide a method for prescribers to create customized menus of medication options.	6	3	1	0
16*†	The display of medication options should not be influenced by promotional considerations.	10	0	0	0
17*†	The meaning of any symbols or special fonts should be immediately available during the prescribing process.	4	0	5	1
18*†‡	Prescribers should have immediate access to the rationale for any medication choice that the system displays as being recommended or preferred for the current patient.	4	0	1	5
19*†	The system should omit from suggested medication menus options that would be medically contraindicated for the patient.	0	0	6	4
20*	The system should allow prescribing by name search from a complete list of medications, bypassing any restricted medication menus.	10	0	0	0
21†	The system should enable providers to determine the accurate formulary status and the actual cost to the patient for each medication option based on the patient's prescription insurance coverage.	1	5	4	0
22†	The system should provide access to the current amount remaining on the patient's prescription drug benefit cap, if one exists.	0	0	10	0
23*	Prescribers should be able to access brief summaries of medication effectiveness and potential harms based on the evidence used for U.S. Food and Drug Administration actions and on evidence from peer-reviewed, randomized clinical trials or peer-reviewed meta-analyses.	3	0	7	0
24*	The systems should provide selection from the dosages and forms that are available and appropriate for a given medication.	9	0	1	0
25*	The system should provide prescribers with access to assistance with dosing calculations based on body size and age, when such adjustments are indicated.	1	2	7	0
26*	The system should provide prescribers with access to assistance with dosing calculations based on kidney and liver function, when such adjustments are indicated.	0	0	10	0
	Alerts and other messages to prescribers
27*	The system should alert the prescriber when a medication is selected that has a contraindication or significant precaution based on the patient's allergies, current medications, medical conditions, and laboratory findings.	1	7	2	0
28*‡	The system should remind the prescriber when a prescription for a new medication might be indicated based on patient data (e.g., medical history, laboratory tests) in conjunction with rules from practice guidelines that are based on thorough analyses of the peer-reviewed literature.	1	0	9	0
29*	For every message, the system should provide immediate access to an explanation of its rationale, including disclosure of all guidelines and financial support used in its development.	5	0	3	2
30*	Prescribers should be able to clearly distinguish alerts and messages based on patient safety and health outcome concerns from those based on formulary adherence and other considerations.	8	0	0	2
31*	The system should prioritize safety alerts based on clinical importance (e.g., the frequency, severity, and certainty of the possible adverse consequences).	6	2	0	2
32*‡§	The system should allow low-priority safety alerts to be suppressed, either by the prescriber or at the time of installation.	3	0	4	3
33‡§	The system should allow the prescriber to suppress alerts and messages that are not based on patient safety.	5	0	1	4
34†§	Any sponsorship of a message to prescribers should be indicated in the message.	1	0	0	9
35‡	Prescribers should be able to proceed with their intended order, overriding any alert, with clinical justification required for the most severe potential adverse events.	1	6	0	3
36*	Alerts and messages should display the date that the underlying decision support rules were last updated.	1	0	7	2
37*	Prescribers should have access to patient data that were used to trigger alerts or tailor medication menus.	8	0	0	2
38*§	Prescribers should be able to correct or flag patient information that they believe to be erroneous.	6	1	3	0
	Patient education
39*†‡	The system should provide information for patients on how to take the medications prescribed and why they should be taken.	5	0	5	0
40*†‡§	The system should print a complete current medication list for patients.	6	1	3	0
	Data transmission and storage
41*‡§	The prescriber should be able to transmit prescriptions to the patient's pharmacy of choice (mail order or retail).	9	1	0	0
42*	Transmission of clinical data between systems should conform to the most recent versions of standards from Health Level 7 and/or the National Council for Prescription Drug Programs.	7	0	0	3
43*	Systems should use universal provider identifiers when they become available.	0	0	0	10
44*	Systems should use universal patient identifiers if they become available.	0	0	0	10
45*†§	Systems should disclose any promotional considerations that have influenced the display of pharmacy fulfillment options.	1	0	0	9
46*‡	Prescribers should be notified of transmission failure of a prescription to a pharmacy.	6	2	1	1
47‡	Systems should be able to receive and store notification from the pharmacy when each prescription is delivered to patient.	0	0	10	0
	Monitoring and renewals
48*‡§	The system should notify the prescriber when a prescription or prescription refill is not dispensed and delivered to the patient within a time interval specified by the provider.	0	0	10	0
49*	The system should remind the clinician to place orders for follow-up laboratory tests recommended by the manufacturer for monitoring.	0	0	10	0
50*	The system should alert the prescriber when the results of laboratory monitoring tests require action.	3	0	7	0
51*	The system should provide access to the results of laboratory monitoring tests.	4	0	6	0
52*§	Prescription renewals entered by nonprescribing office staff should be clearly attributable to both an individual staff person and an authorizing prescriber.	8	0	0	2
	Transparency and accountability
53*†	The system should display notification of corporate sponsorships and critical business relationships that could represent conflicts of interest, and vendors should completely and transparently disclose details of these relationships in publicly available documents.	1	1	1	7
54*†	Prescribing system vendors should provide access to complete and transparent disclosure of the sources and methods used to develop clinical decision support rules, including the triggers for alerts and other messages.	4	1	3	2
	Prescriber-level feedback
55*†‡	Prescribers should be able to review profiles of their own prescribing patterns.	5	0	5	0
56*	The system should be able to profile prescribers' history of overriding alerts.	3	0	4	3
	Security and confidentiality
57*	Systems should support compliance with the most current Health Insurance Portability and Accountability Act standards for privacy and security.	10	0	0	0
58	User activities should be recorded in a reliable audit trail that is accessible only to authorized personnel responsible for enforcing data privacy and security.	8	1	1	0
59*	Each user should be individually identified in the system and have role-based access privileges.	10	0	0	0
60*	Systems should support a method for checking the integrity of stored or transmitted data.	8	0	2	0

Recommendations were classified as not applicable when they were refining a feature that the system did not have. For example, the recommendation that systems prioritize safety alerts (#31) was not applicable for systems that lack any safety alerts.

^*Rated by the expert panel as clearly beneficial for patient safety and health outcomes.

^†Rated by the expert panel as clearly beneficial for helping patients manage their costs.

^‡Classified by the investigators as a recommendation that would address the underuse of medications with proven benefits.

^§Recommendation added in the final round of the panel process; assessed by telephone interview for five of the ten systems (see Methods).