Evaluation of a User Guidance Reminder to Improve the Quality of Electronic Prescription Messages

Summary

Background

Prescribers’ inappropriate use of the free-text Notes field in new electronic prescriptions can create confusion and workflow disruptions at receiving pharmacies that often necessitates contact with prescribers for clarification. The inclusion of inappropriate patient direction (Sig) information in the Notes field is particularly problematic.

Objective

We evaluated the effect of a targeted watermark, an embedded overlay, reminder statement in the Notes field of an EHR-based e-prescribing application on the incidence of inappropriate patient directions (Sig) in the Notes field.

Methods

E-prescriptions issued by the same exact cohort of 97 prescribers were collected over three time periods: baseline, three months after implementation of the reminder, and 15 months post implementation. Three certified and experienced pharmacy technicians independently reviewed all e-prescriptions for inappropriate Sig-related information in the Notes field. A physician reviewer served as the final adjudicator for e-prescriptions where the three reviewers could not reach a consensus. ANOVA and post hoc Tukey HSD tests were performed on group comparisons where statistical significance was evaluated at p<0.05

Results

The incidence of inappropriate Sig-related information in the Notes field decreased from a baseline of 2.8% to 1.8% three months post-implementation and remained stable after 15 months. In addition, prescribers’ use of the Notes decreased by 22% after 3 months and had stabilized at 18.7% below baseline after 15 months.

Conclusion

Insertion of a targeted watermark reminder statement in the Notes field of an e-prescribing application significantly reduced the incidence of inappropriate Sig-related information in Notes and decreased prescribers’ use of this field.

Background and Significance

In 2012, 69% of all office-based physicians and 94% of the pharmacies in the United States were actively e-prescribing [1, 2]. As a key component in the health information technology infrastructure, e-prescribing has demonstrated improved safety, quality and cost effectiveness of patient care [3, 4, 5, 6]. However, as with any technology, the outcomes derived from e-prescribing are largely dependent on the manner in which it is implemented and previous research has demonstrated that e-prescribing systems can elevate the risks of some errors and may even introduce new sources of error [7, 8, 9, 10, 11].

A particular case in point is the Notes field in the National Council for Prescription Drug Programs’ (NCPDP) SCRIPT e-prescribing standard. Technically referred to as “free-text,” the 210-character Notes field is listed as an optional field in the SCRIPT standard. The field is intended to allow prescribers to include additional patient-specific information that is related to, but not technically part of, the prescription [12].

Importantly, the Notes field should not be used for prescription information that has a designated, standard field [12, 13]. In practice, however, the Notes field is often populated with information that should have been included in another designated field within the e-prescription message. This is because prescribers often feel the need to modify, clarify or reinforce information on e-prescriptions and the easy availability and flexibility of the free text Notes field provides a readily available opportunity [10, 14]. Among the most common violations is the inclusion of patient directions (Sig) information [15, 16]. This often creates workflow disruptions or confusion at receiving pharmacies which in turn often necessitates contact with prescribers for clarification thereby disrupting their workflow, too [16, 17, 18].

A number of strategies including clear labeling of the data, specific end user training, etc. have been suggested to reduce the inappropriate use of the Notes field and a task group within NCPDP is currently working to codify the Notes field while still allowing prescribers to enter free text, where needed and appropriate [12, 13, 22]. At present, however, inappropriate use of the free text Notes field continues to represent a particularly vexing problem [10, 14, 15].

Computerized reminders have been promoted to improve care and one possible remedy to reduce inappropriate use of the Notes field is to alert prescribers and provide them with targeted point-of-use guidance on proper use [19, 20, 21]. The objective of this study was to evaluate the effect of a visible embedded point-of-use reminder in the Notes field (labeled “Comments”) of an electronic health record (EHR) based e-prescribing application on the incidence of inappropriate inclusion of patient direction (Sig) information in the Notes field of e-prescriptions issued by prescribers.

Methods

In the e-prescribing application investigated in this study, the Notes field (designated by NCPDP in the SCRIPT e-prescribing standard) was labeled “Comments” by the software vendor. User guidance on appropriate use of the Notes field was implemented as a preformatted watermark reminder statement that appeared in the Notes field of the e-prescribing application (▶ Figure 1). The watermark reminder statement is a visible embedded overlay in the Notes field with the following text message:

Fig. 1.

Watermark Reminder in Notes Field of E-prescribing Application

“This field is for nonclinical comments to the pharmacist. Any additional clinical instructions for this prescription should be added using the “Additional Instructions” segment of the Sig Builder.”

This specific language was derived through a multidisciplinary internal consensus process by the EHR vendor to provide the most concise guidance to their end-users.

Prescribers in the study were clinicians that practiced in ambulatory practices affiliated with large multi-state non-profit health care organization that has deployed a commercially available Meaningful Use and Surescripts-certified out-patient EHR featuring advanced e-prescribing functionality. On average, each site has 5 to 10 practicing prescribers and operates independently in hiring and delivering care services. While there is centralized training and management of the EHR software, there is variation among the prescribers and sites in their use of the software. No end-user training was performed relative to the addition of the watermark reminder statement. Most study prescribers were physicians (n=89) with the remainder being equally split between nurse practitioners (n=4) and physician assistants (n=4).

To assess the effect of including the watermark reminder statement in the Notes field, e-prescriptions were collected from the same exact cohort of prescribers over three separate time periods: prior to the inclusion of the watermark reminder statement, three months post intervention, and again 15 months post intervention. New e-prescriptions issued by these prescribers were collected and analyzed initially for any Notes-specific Quality Related Events (QREs) and then for all Sig-related QREs in Notes using Surescripts’ E-prescribing Quality Guidelines [12]. A Notes-specific QRE was defined as any information contained in the Notes field for which a standardized and designated messaging field exists elsewhere in the SCRIPT standard. A Sig-related QRE was defined as any patient direction (Sig) information contained in the Notes field.

The cohort of prescribers selected for analysis included all those who generated at least 100 e-prescriptions during each of the three data collection periods. All new e-prescriptions issued by study prescribers that were successfully delivered to receiving pharmacies through the Surescripts Electronic Prescription Network during the three data collection periods were included in the study analysis. E-prescriptions issued between December 26, 2010 and February 6, 2011 (43 days) served as the pre-intervention baseline. On February 20, 2011, the watermark reminder statement was added to the Notes field as part of the regular scheduled version upgrade process. Following a three-month washout period, e-prescriptions issued by the study cohort were sampled again between May 20, 2011 and July 1, 2011 (43 days) to assess the immediate impact of the intervention on prescribers’ behavior. To assess the extent to which observed changes in prescribing were sustained over time, e-prescriptions generated by study prescribers were sampled again one year later between May 20, 2012 and July 1, 2012 (43 days). Before analysis, e-prescription data collected were de-identified by an independent expert and certified to meet the requirements for de-identification, as defined by the HIPAA Privacy Rule at 45 C.F.R. §164.514.

Three Pharmacy Technician Certification Board (PTCB) certified pharmacy technicians each having more than three years of experience processing e-prescriptions in the community pharmacy setting from the Surescripts Clinical Quality Team independently reviewed the Notes content of each e-prescription and characterized it as either appropriate or inappropriate use of the Notes field. If the e-prescription was considered to contain one or more violations of Surescripts Quality Guidelines it was considered to be a Notes-specific QRE. Reviewers were then asked to further identify all Notes-specific QREs that were Sig related and label them as Sig-related QREs see examples in ▶ Table 1. All technicians had completed prior training under the supervision of a pharmacist to confirm subject matter proficiency. To ensure consistency, classification rules were established complete with examples to aid during the review process. In cases where consensus could not be reached by the three reviewers, the e-prescriptions were discussed as a group to attempt to reconcile differences. A physician with 18 years of experience as a practicing family medicine practitioner and board-certification in clinical informatics served as the final adjudicator for e-prescriptions where the three technician reviewers could not reach a consensus after discussion. Statistical analysis was performed using the ANOVA procedure to calculate the incidence of Sig-related QREs in study e-prescriptions. Post hoc analyses were conducted using Tukey HSD tests to assess the statistical significance of pair-wise differences while controlling for experiment-wise error. All analyses was performed using SAS statistical software, version 9.4.

Table 1.

Examples of inappropriate patient directions (Sig) information in the Notes field.

Drug Description	Patient Directions (Sig)	Notes (labeled as Comments)
Valtrex 500 mg Tab	take 1 tablet (500MG) by oral route every day	begin after finishing 1G tabs
Benicar 20 mg Tab	take 1 tablet (20MG) by ORAL route every day	one tab qd
Flexeril 10 mg Tab	take 1 tablet (10MG) by ORAL route qhs PRN as needed	1/2 to 1 tab at hs PRN
Celexa 20 mg Tab	take 1 tablet (20MG) by oral route every day	1/2 tab for 5 days then full tab

Results

A total of 97 prescribers met the eligibility criterion of at least 100 e-prescriptions issued during each of the three data collection periods.

As illustrated in ▶ Table 2, at baseline 9.1% of e-prescriptions in the sample contained some text in the Notes field. Of these, 739 (2.8% of the total) contained a Sig-related QRE. Three months following implementation of the reminder the incidence of e-prescriptions with any text in the Notes field had dropped to 7.1%. Of these, 555 (1.8% of the total) contained a Sig-related QRE. After 15 months post implementation of the reminder, prescribers’ use of Notes had remained approximately constant at 7.4% as had the incidence of Sig-related QREs at 1.8%.

Table 2.

Effect of Watermark Reminder Statement on the Incidence of Sig-Related Quality Related Events (QREs) in Notes Field

	BASELINE N (%)	3-MONTHS N (%)	15-MONTHS N (%)
E-prescriptions in Sample	26,674 (100)	31,061 (100)	39,618 (100)
E-prescriptions with Notes	2,395 (9.1)a, b	2,212 (7.1)a	2,920 (7.4)b
Sig-related QREs in Notes	739 (2.8)c, d	555 (1.8)c	698 (1.8)d

^ap<0.05

^bp<0.05

^cp<0.05

^dp<0.05

The Kappa coefficient of agreement between each pair of the three quality specialists who independently reviewed e-prescriptions for Sig-related QREs was 0.90, 0.84, and 0.82, demonstrating reasonably good agreement among the three judges.

To assess the effect of the watermark reminder statement on the prescribing behavior of individual clinicians, the 64 prescribers who exhibited at least one Sig-related QRE at baseline were analyzed. As illustrated in ▶ Table 3, among prescribers who had at least one Sig-related QRE at baseline, the mean Sig-related QRE incidence rate decreased from 4.1% at baseline to 2.5% at the end of 15 months after implementation of the watermark reminder statement. Although this represents a decrease of 39.5%, the change was not statistically significant due in part to the small sample size (n=64).

Table 3.

Effect of Watermark Reminder Statement on the Incidence of Sig-related Quality Related Events in Notes by Prescriber (n=64)

	MEAN (STD DEV)	MEDIAN	MIN	MAX
Pre-test	4.1% (5.4)	1.9%	0.2%	25.9%
3 Months	2.9% (4.0)	1.1%	0%	15.1%
15 Months	2.5% (3.6)	1.1%	0%	15.1%

Discussion

In this analysis, we found that addition of an embedded watermark reminder statement in the Notes field of an EHR-based e-prescribing application resulted in a significant decrease in the incidence of Sig-related QREs in Notes three months after implementation. This effect was sustained after 15 months. The mean rate of Sig-related QREs in the Notes field of e-prescriptions issued by prescribers who had at least one such QRE at baseline also appeared to decrease but this effect did not achieve statistical significance due to the relatively small number of prescribers in the analysis (64).

In addition to reducing the incidence of Sig-related QREs in the Notes field, inclusion of the watermark reminder statement was associated with reduced use of the Notes field by prescribers’ altogether. This somewhat unexpected result suggests that while the watermark reminder statement was targeted to reduce one particular type of inappropriate Notes use, it may have served to encourage prescribers to consider whether use of this field was warranted at all. The potential unintended negative consequences of prescribers’ reduced use of the Notes field, if any, cannot be determined by this analysis but may be considered an appropriate objective of future analyses.

Despite better user training and selected use of targeted reminders like the one studied here, there may be a background incidence of inappropriate Notes usage that cannot be fully corrected. Most e-prescribing systems have implemented a Sig Builder Tool, typically a series of drop down menu options that is designed to generate structured patient directions. Prescribers often require auxiliary instructions to modify, clarify and highlight pertinent details on prescriptions and this may not be accommodated by the Sig Builder Tool [10, 14, 23]. While the option to add text information at the end of the structured Sig does exist in the e-prescribing application used in this study, the mechanism for doing so is not immediately evident and requires an additional mouse click to access. In contrast, the ease of accessibility to the Notes field on the same screen as the Sig Builder Tool thus could be a contributing factor.

Anecdotal reports suggest that the 140-character Sig field restriction as dictated by the current version of the NCPDP SCRIPT standard is simply insufficient for prescribers to accurately communicate complete Sig information on certain types of medications (e.g., tapered dosage schedules for oral glucocorticoids). This concern has now been addressed within the NCPDP SCRIPT standard and the length of the Sig field has been increased to 1000 characters in the in SCRIPT Standard Version 2012. While adequate end user training on the Sig builder functionality may help to mitigate the unneeded or inappropriate use of the Notes field, regular interface improvements targeted towards improving usability based on end user input and a continuous quality feedback loop to inform interface designers are critical as demonstrated in this study. [24] A small interface enhancement that was developed at a relatively minimum cost can have sustained meaningful impact on quality improvement.

Several limitations of this study must be noted. First, we analyzed the effect of a single, targeted watermark reminder statement on the incidence of inappropriate Sig-related Notes use. It is possible that the use of multiple reminders that targeted Sig-related Notes errors and/or other inappropriate uses of Notes would have produced different results. Indeed, it may be advantageous to rotate reminder messages in key error-sensitive fields to ensure that users do not become desensitized to seeing the same reminder in the field.[25] Second, the analysis was conducted using a single commercial EHR-based e-prescribing software application in a relatively limited prescriber population of one health system. These prescribers may not be representative of the population of prescribing clinicians. Finally, the classification of Notes comments as either appropriate or inappropriate is necessarily a somewhat subjective assessment. While the three quality specialists that performed the review are all experienced pharmacy technicians with hundreds of hours of experience applying Surescripts Quality Guidelines to e-prescribing messages, a certain amount of variance is to be expected given the complexities of the underlying data.

Conclusion

Insertion of a pre-formatted, watermark reminder statement embedded into the Notes field of an outpatient EHR-based e-prescribing application significantly reduced both the Sig-related Notes usage and the use of the Notes field altogether. This study suggests that targeted end-user reminders can improve the quality of e-prescribing messaging. Additional studies are needed to determine how best to implement this strategy to improve the quality of electronic prescribing while not creating undue burdens for users.