Evaluating the Technical Adequacy of Electronic Prescribing Standards: Results of an Expert Panel Process

Abstract

Objective:

To support more informed prescribing decisions, e-prescribing systems need data on patients’ medication histories and their drug-specific insurance coverage. We used an expert panel process to evaluate the technical adequacy of two standards for delivering this information, the Medication History function of the NCPDP SCRIPT Standard and the NCPDP Formulary and Benefit Standard.

Methods:

We convened a panel representing 14 organizations that had experience with these standards. Experts within each organization submitted narrative responses and ratings assessing the standards in 6 domains, including data quality, completeness, usability, and interoperability. Areas of disagreement were discussed in recorded teleconferences. Narrative was analyzed using a grounded-theory approach.

Results:

Panelists agreed that the structure of the Medication History Standard was adequate for delivering accurate and complete information but implementation problems made the data difficult to use for decision support. The panel also agreed that the Formulary and Benefit Standard was adequate to deliver formulary status lists, but other parts of the standard were not used consistently and group-level variations in coverage were not represented. A common problem for both standards was the lack of unambiguous drug identifiers; panelists agreed that RxNorm deserves further evaluation as a solution to this problem.

Conclusions:

A panel of industry experts found the basic structure of these two standards to be technically adequate, but to enable benefits for patient care, improvements are needed in the standards’ implementation.

Introduction

Electronic prescribing (e-prescribing) is expected to reduce medication errors and lower medication costs, but to achieve these goals, e-prescribing systems must deliver safety and cost information to prescribers.¹ Often, this information must be obtained from outside of the provider organization; thus standards are needed for the interoperable exchange of this data.² Standards are emerging as a cornerstone for the role of informatics in health care policy,^{3, 4} but few studies have formally evaluated standards themselves.

We sought to evaluate two standards from the National Council for Prescription Drug Programs (NCPDP) that were under consideration as Initial Standards for e-prescribing under Medicare.⁵ The Medication History (RxH) function of the NCPDP SCRIPT standard, v. 8.1, is intended to give prescribers information about a patient’s current and past medications by listing their past pharmacy claims. The NCPDP Formulary and Benefit (F&B) Standard, v. 1.0, is intended to give information about patients’ prescription drug coverage by listing data about the coverage provide by specific drug insurance plans (rather then data about individual patients). This standard includes several types of drug coverage information, including formulary status lists (FSL), alternative suggestions (ALT), coverage limitations (COV), and patient copay information (COP). F&B data is downloaded in a “batch” fashion, an approach that is necessary to enable the display of coverage information for each medication in the pick-lists that prescribers use to make initial medication choices.

We evaluated these standards within a conceptual framework¹ for projecting the effects of the data they provide (or should provide) on clinical processes and, ultimately, outcomes Adequate RxH data should provide accurate information about medications prescribed by other physicians, thus enabling better alerts for potentially-harmful or potentially-beneficial but omitted medications. It could also help physicians to detect and address patient non-adherence to critically important medications. F&B information could support the prescribing of more-affordable medication options, which could improve patient adherence and also prevent call-backs from pharmacies for drug coverage problems.

Methods

We convened a panel of technical experts representing organizations that have direct experience in implementing e-prescribing standards. To encompass expertise with the origination, routing, and processing of each standard transaction, we included point-of-care (POC) system vendors, e-prescribing intermediary companies, drug knowledge content providers, pharmacies, and prescription benefit management companies (PBMs). Companies were nominated for recruitment in these categories based on their identification by the two largest intermediary companies, RxHub and SureScripts, as having substantial experience with the standards as of January, 2006. Of 15 companies approached, 14 participated. At an initial meeting of the Expert Panel, agreement was reached on the process the panel would use to evaluate each standard (Table 1). This process was followed separately for each standard.

Table 1.

Steps in the Expert Panel Process

1.	Draft survey to elicit experts’ narrative feedback and ratings for each technical domain
2.	Distribute draft survey to panel; incorporate their feedback in revising the final survey
3.	Distribute final survey to panel, giving panelists about 2 weeks to complete and return them
4.	Email individual panelists for clarification and elaboration on specific responses
5.	Collate, de-identify findings, and perform initial analysis to identify key themes
6.	Distribute de-identified findings to participating panelists
7.	Conduct a conference call (audiotaped) to discuss areas of disagreement and elicit more detailed information focused on key themes
8.	Qualitative analysis of narrative responses and transcripts using Atlas.ti (v5.0)
9.	Follow-up questions to clarify areas of ambiguity

Each stage of the process was organized around a survey, consisting of both narrative and rating-scale questions that were designed to elicit feedback regarding the evaluation domains shown in Table 2. Initial draft surveys were revised based on feedback from the panel. Final surveys were then distributed to the primary contact person for each organization, who was responsible for distributing the questions among appropriate experts within his or her company and for gathering and collating their answers. In this way, each participating company acted as a single expert on the panel. Study staff collected, aggregated, and de-identified responses and distributed these back to the panel. An audio-recorded conference call was then conducted with the panelists to discuss areas of disagreement and to elicit more detailed information on emergent themes. The panelists’ written, narrative responses as well as a transcript of the conference call were entered into the ATLAS.ti Knowledge Workbench (Scientific Software, Berlin, Germany) for qualitative analysis. Narrative data was coded using a grounded theory approach⁶ and emerging themes were revised based on discussion among all investigators. As needed, additional questions were directed to individual panelists and their responses were added to the primary documents for analysis.

Table 2.

Technical evaluation domains

Domain	Question content
Data quality	Erroneous data, erroneous use of data types, missing data needed (whether “optional” or not)
Completeness of the standard	Types of patients/drugs/etc. that can’t be handled, workarounds used, data elements needed
Data usability	Difficulties using the data (e.g. identifiers), misunderstandings of the standard, variability among implementations
Interoperability	Translation of data among related standards and external systems
Systems architecture	Coordination of data exchange among multiple evolving systems
Overall functioning	Delivery of information that can improve Rx decision-making

To provide collateral information on the industry’s use of different F&B files, RxHub also counted the monthly number of downloads that e-prescribing vendors completed for each type of F&B list in 2006.

Results

Medication History Standard

Of the 14 panelist companies, 10 participated in the RxH evaluation round based on their having specific experience with the standard. We analyzed 35 single-spaced pages of narrative data, using 41 codes to categorize 267 specific quotations.

In assessing the quality of data provided by the standard, panelists observed that outright errors in RxH data are rare, but some point of care (POC) e-prescribing vendors expressed difficulty in using RxH data because many important fields, including the prescriber’s identity, Sig, quantity dispensed, and dispensing pharmacy, are optional and are often left empty. The lack of this information hinders reconciliation with prescriptions that the POC system has generated, and reconciliation was seen as necessary for using RxH records in automated alerting without generating large numbers of false alerts.

A major usability problem that panelists cited was the lack of an adequate drug identifier. RxH records generally use the dispensed drug’s NDC code, but because there is no single, accurate source of NDC codes,⁷ these sometimes cannot be accurately mapped to the drug compendia that e-prescribing systems use internally, causing further difficulties with reconciliation. Some POC vendors said they had given up on reconciling RxH data and drive alerts only from prescription data originated on their software. Others used more complex reconciliation based on string matching when NDC matching fails, but with inconsistent success. The panel enthusiastically supported further development of RxNorm to improve drug representation, one saying “If RxNorm becomes a reality and this value is stored on the history, it will make the drug alert checking that much better.”

Another usability problem arose from the potential to obtain duplicate RxH records for what was actually the same prescription. This could occur when multiple requests are made for the patient over time, especially after changes in insurance coverage. Duplicate history records can lead to erroneous safety alerts such as duplicate therapy warnings.

All POC vendors and one intermediary agreed that an XML representation of the NCPDP SCRIPT standard would improve its usability.

Panelists also pointed out that retrieving RxH relies on the patient’s being identified through a successful X12N 270/271 eligibility check and, in many practices, half or more of eligibility checks fail for reasons that include local health plans’ non-participation with RxHub, differences between the provider and the health plan in patient identifying data (e.g. DOB, zip code errors), and some patients being uninsured. One POC vendor said “In order for medication history to be used effectively, it should be available in a consistent manner for the majority of the patients being managed by a provider or practice. In areas of scarce PBM coverage, for example, providers do not find this information useful even when available.”

Panelists also mentioned other scenarios in which a patient’s medication history may not be returned even if data is available. These reasons include inability to accurately identify a patient when they are one of many family members all covered under a family level plan, federal and state regulations that require filtering of sensitive prescription records, the inability to support compound drugs, and the lack of important over-the-counter drugs among prescription claims.

With regard to interoperability, RxH records that are created from pharmacy claims data rely upon the data transmitted from the dispensing pharmacy to the PBM during the claim submission process using the NCPDP Telecommunications 5.1 standard. The panelists generally agreed that a unique identifier could be carried from the original electronic prescription through existing fields in the NCPDP Telecom standard to be returned in medication history records. Claims databases would also need to support the storage of this identifier, but its presence would largely solve any reconciliation challenges.

Panelists reported overall satisfaction with the interoperability of the RxH standard with HL7-based prescription orders, though representatives of one intermediary reported minor mapping issues.

Several panelists reported that even though medication histories may be available for a patient, physicians have not integrated the use of this data into their workflow. Because of this, actual usage of the RxH data was less than expected. One POC vendor said “We get a sense from our providers that it’s valuable to have that information, but not a good sense for where it fits in their workflow.”

Formulary and Benefit Standard

One panelist company withdrew from participation prior to the F&B evaluation round, citing time constraints. Of the 13 remaining, 10 had specific experience with the F&B standard and participated in its evaluation process. We analyzed 39 single-spaced pages of narrative resulting from this process, assigning 166 quotes to 32 codes.

Panelists identified several problems that, in aggregate, lead to F&B data being absent for many patients. First, the standard assumes that the patient’s current drug insurance plan is identified through a successful eligibility check, but these checks often fail, for reasons described previously. One panelist observed that the usefulness of F&B data “is directly related to the number of successful eligibility transaction matches in terms of identifying a patient. Even though the PBMs [affiliated with RxHub] combined probably cover 50% to 70% of covered lives in the country, our experience has been around 30%. In many cases, the number is lower and can vary from practice to practice even within the same geography.” Another POC system vendor panelist observed that “there is a need for expansion of eligibility and formulary to additional payors.” In the U.S. overall, RxHub estimated that its coverage rates range from 15% to 96% among different Metropolitan Statistical Areas. Conversely, panelists also noted that some patients have eligibility with more than one plan and the Standard does not provide guidance on how to select the primary coverage or deal with differences. Finally, POC vendors criticized the absence of recognizable health plan names being provided in the standard’s “cross-reference list” file, preventing its use for looking up patients’ coverage when the eligibility transaction fails.

Panelists also identified problems with the accuracy of F&B data arising from the use of representative NDC codes as drug identifiers. “Because of the potential differences in NDC number, items provided in the formulary & alternatives files may not result in a one-to-one match on the vendor side. The impact of this is that when the physician chooses a drug, he or she may get erroneous formulary messages or no message at all.” Panelists enthusiastically supported the development of RxNorm as the standard drug identifier for F&B. “RxNorm would be great if fully implemented across the board, including OTCs,” observed one panelist.

Differences in coverage among different employer-level groups within individual health plans is another major source of inaccuracy in the F&B data presented to clinicians. “Current process is at a representative level, so member-specific exceptions and other variances are not accurately reflected in the F&B display,” explained one PBM Panelist. “Representative level means at a health plan level. For example, one health plan may have several groups which have varying F&B information; however, only one representative F&B is displayed,” clarified another.

Another usability problem with the standard is the variance in its use among health plans and PBMs. As one panelist summarized, “One payer/PBM may support all F&B lists while another supports only one. One payer/PBM may provide optional data elements while another doesn’t.” This creates difficulties in presenting clinicians with consistent coverage information. One POC system vendor panelist noted that “different F&B data providers have different requirements on the presentation of the data … that require great effort to accommodate. Multiple eligibility occurrences lead to [further] difficulties in deciding on appropriate information presentation.” Only one F&B file, the formulary status list (FSL), was used by all participating PBMs.

In the last half of 2006, RxHub was providing average downloads per month of 728 for FSL, 89 for ALT, 21 for COV, and 2 for COP files. Each F&B list is provided by a single PBM but they sometimes contain data for several of the PBM's health plan clients. Thus the download count depends on the number of distinct lists that PBMs are publishing, the number of POC vendors downloading each, and the frequency of refreshing the lists.

Discussion

Overall, our evaluation indicates that the structure of the RxH transaction is adequate to deliver valuable information about the medications that patients are taking, but its potential value is being undermined by the data’s inconsistent availability and by usability problems that make it difficult or impossible to reconcile the data it provides. This challenge to the value of RxH was corroborated by findings of a separate physician survey that we conducted, in which users had positive overall perceptions of e-prescribing, but most were not familiar with the medication history feature and they did not perceive that the systems helped them to learn what other providers had prescribed.⁸

Our evaluation of the NCPDP F&B standard found that its “formulary status list” component has been successfully implemented among many e-prescribing partners. However, technical and implementation issues, such as the failure to represent “group-level” variations in coverage within the same health plan, are leading to errors and omissions in the coverage information presented to prescribers as compared with the coverage that patients actually experience when they present to the pharmacy.

Our finding that F&B components other than the formulary status list are being used less consistently suggests that more research is needed on conveying prescription drug coverage to clinicians. Each PBM tends to use a different subset of the standard’s functionality and to enforce different ways of displaying this information. Overall, the copay file appears to be particularly underused, given that professionalism should lead prescribers to make the patient’s costs their primary cost consideration. Patients who have more than one source of coverage highlight the challenge of inconsistent F&B file use. Our panelists called for guidance on this situation but did not comment on how they handle it.

One limitation of our study was that, at the time of our expert panel, the only intermediary supplying RxH data was RxHub, so insurance claims were the only source of medication history data. Since then, other intermediaries such as SureScripts have begun supplying RxH data, using aggregated pharmacy sales as the source. However, integrating this additional RxH data source with data from insurance claims is likely to to make reconciliation more rather than less challenging for e-prescribing system vendors. Our evaluation of F&B data was also limited to participants using RxHub as a data source. Thus, our study does not reflect the readiness of industry participants that do not currently work with RxHub, including some state Medicaid health plans.

Our finding of ongoing difficulties with drug identifiers suggests that further development of RxNorm to serve as the preferred drug identifier could have the most immediate benefits. The universal and unambiguous identifiers that RxNorm could provide would improve the accuracy of reconciliation for RxH data and the accuracy of matching drug choices to F&B information. Furthermore, RxNorm would likely reduce the work needed to maintain F&B files, making it more feasible for health plans to deliver more complete F&B information. Both standards currently have fields to support the use of RxNorm, so no changes would be needed in standards’ structure. Additional pilot studies are needed, however, to demonstrate RxNorm’s coverage and to evaluate its fit especially within each of the different F&B files.

More research and development is also needed to establish a “real time benefit check” transaction that could confirm a specific patient’s coverage for a specific drug and dose that has been selected by a prescriber. RxHub has developed such a transaction but it is currently being used by only one of its PBM clients. Further studies should investigate the degree to which this transaction could increase the cost-effectiveness of prescribing decisions and reduce the rework associated with the coverage exceptions that continue to arise due to the inability of the F&B batch standard to represent individual-level coverage.

It currently appears likely that economic pressures and prescribing safety concerns will lead to a mandate for U.S. physicians to prescribe electronically in the relatively near future, using a set of interoperability standards that are likely to include the RxH and the F&B standards that we evaluated.^{9, 10} The results of our study indicate that these standards are being used with some success on a technical level, but that further research and coordination among e-prescribing industry participants may be necessary for e-prescribing interoperability to deliver the benefits that are envisioned by policymakers.