An e‐Delphi study to obtain expert consensus on the level of risk associated with preventable e‐prescribing events

Abstract

Aims

We aim to seek expert opinion and gain consensus on the risks associated with a range of prescribing scenarios, preventable using e‐prescribing systems, to inform the development of a simulation tool to evaluate the risk and safety of e‐prescribing systems (ePRaSE).

Methods

We conducted a two‐round e‐Delphi survey where expert participants were asked to score pre‐designed prescribing scenarios using a five‐point Likert scale to ascertain the likelihood of occurrence of the prescribing event, likelihood of occurrence of harm and the severity of the harm.

Results

Twenty‐four experts consented to participate with 15 pand 13 participants completing rounds 1 and 2, respectively. Experts agreed on the level of risk associated with 136 out of 178 clinical scenarios with 131 scenarios categorised as high or extreme risk.

Conclusion

We identified 131 extreme or high‐risk prescribing scenarios that may be prevented using e‐prescribing clinical decision support. The prescribing scenarios represent a variety of categories, with drug–disease contraindications being the most frequent, representing 37 (27%) scenarios, and antimicrobial agents being the most common drug class, representing 28 (21%) of the scenarios. Our e‐Delphi study has achieved expert consensus on the risk associated with a range of clinical scenarios with most of the scenarios categorised as extreme or high risk. These prescribing scenarios represent the breadth of preventable prescribing error categories involving both basic and advanced clinical decision support. We will use the findings of this study to inform the development of the e‐prescribing risk and safety evaluation tool.

What is already known about this subject

• Implementation of e‐prescribing (EP) reduces preventable adverse drug events; however, optimal system configuration is required to maximise benefits.

• In the United States, simulation tools have been used to evaluate the safety of EP systems

• High‐risk prescribing scenarios are used by healthcare professionals, to promote safer use of medicines

What this study adds

• This study has identified high‐risk prescribing scenarios amenable to clinical decision support and appropriate for use in the development of the e‐Prescribing Risk and Safety Evaluation tool (ePRaSE)

• This simulation tool will be rolled out, nationally, to all NHS organisations with implemented EP systems, to support optimisation.

1. INTRODUCTION

1.1. Background and significance

The World Health Organization's (WHO) third global patient safety challenge aims to reduce severe, medication‐related harm by 50% over 5 years, by strengthening prescribing systems at each stage of the medication process, including prescribing, ordering, dispensing, administering and monitoring. ¹ e‐Prescribing (EP) has been defined as “the utilisation of electronic systems to facilitate and enhance the communication of a prescription or medicine order, aiding the choice, administration and supply of a medicine through knowledge and decision support and providing a robust audit trail for the entire medicines use process”. ² The benefits of EP systems combined with clinical decision support (CDS) systems are well documented and include a reduction in medication errors and preventable adverse events. ³ , ⁴ , ⁵ Prescribing errors can continue to decrease over time as users adapt to the system and system configuration is optimised for local use. ⁶ Greater patient safety benefits are observed with experienced users rather than new implementers. ⁷ Slight et al. investigated the impact of system optimisation in a prospective observational study in a large UK hospital and reported how some types of errors reduced over time, with serial changes made to the system. ⁸ Clearly, optimal system configuration is required to maximise benefits.

Implementation of EP has also been shown to be associated with the introduction of new types of errors often referred to as the unintended consequences of EP. ⁹ A systematic review identified eight error categories associated with EP systems that related to both system design and user–system interactions. ¹⁰

Overuse of alerts or inappropriate alerting results in alert fatigue and a high incidence of alert overrides, which can adversely affect patient safety. ¹¹ Van de Sijs et al. reviewed the efficacy of alerts and found an override rate of between 49 and 96%, with low‐level alerts overridden more frequently than high‐level alerts. ¹² Slight et al. concluded that 5.5 million alerts are inappropriately overridden in the United States each year, which is associated with a related cost of between $871 million and $1.8 billion from preventable adverse drug events. ¹³

The UK government has launched several funding initiatives to drive digital advancement within National Health Service (NHS) organisations, including the implementation and optimisation of EP systems, which has resulted in increased uptake in recent years. ¹⁴ In 2013, Ahmed et al. reported that in response to a national survey of EP usage, more than two‐thirds of NHS hospitals had implemented at least one EP system, with many hospitals utilising multiple different EP systems; 60 different systems were operational across the respondent hospitals. ¹⁵ In September 2020, EP systems have been implemented in over 130 NHS trusts. ¹⁶

In the United States, simulation tools have been used to evaluate e‐prescribing systems; the Leapfrog computerised physician order entry (CPOE) evaluation tool is probably one of the best‐known ones and assesses hospital safety, quality and efficiency based on national performance measures. ¹⁷ The annual Leapfrog Hospital survey is voluntary and freely available to hospitals throughout the United States, with the results publicly reported. Participating hospitals are provided with information that benchmarks their progress towards improving patient care. The Leapfrog CPOE evaluation tool has been used to inform system configuration development, leading to improved system safety. ¹⁸ Variability in safety performance has been demonstrated between different hospital EP systems, independent of system vendor. ¹⁹ Uptake of the Leapfrog CPOE evaluation tool has increased 10‐fold over the last decade; however, longitudinal data has demonstrated only modest improvements in electronic health record (EHR) safety performance and the persistence of substantial safety risks. ²⁰

There is currently no standard method to evaluate the effectiveness of implemented EP systems in preventing medication errors or adverse drug events in the UK. Resources are available to support NHS hospitals in the implementation and optimisation of EP systems including “how to” guides and examples of good practice, but opportunities to test and receive feedback on system configuration is not currently available. The development of an e‐Prescribing Risk and Safety Evaluation tool (ePRaSE) has been commissioned by NHSX, which is a UK Government organisation responsible for the delivery and expansion of digital healthcare within NHS settings. Similar to the US Leapfrog tool, ¹⁹ ePRaSE is a web‐based tool designed to evaluate the capabilities available in electronic prescribing systems that are currently being used in UK hospitals. The assessment methodology offers a one‐time, cross‐sectional look at whether decision support provides advice to a prescriber. The ePRaSE tool uses test patients (i.e., fictitious patients) and test orders, which represent actual medication orders and include high‐risk prescribing scenarios that not only have potential to cause patient harm, but also could be prevented using EP systems. The ePRaSE tool is designed to give individual hospitals detailed and specific feedback on their safety performance.

1.2. Objectives

Our aim was to establish expert consensus on the high‐risk prescribing scenarios which are appropriate for use in the development of a UK simulation tool (the e‐Prescribing Risk and Safety Evaluation tool (ePRaSE)) that will be rolled out to all NHS organisations with implemented EP systems in England.

2. METHOD

The e‐Delphi technique is a structured process for assembling knowledge from a group of experts to achieve consensus on a specific theme. Where there is a lack of empirical evidence, the Delphi technique provides an opportunity to gather opinion from experts who may be located in geographically different regions and settings. ²¹ The e‐Delphi technique has been commonly adopted in healthcare research, including similar clinical informatics research. ²¹ , ²² Sweidan et al. utilised a modified Delphi technique to establish consensus on the features of e‐prescribing systems that are expected to support the safety and quality of prescribing practices and use of medicines in general practice. ²²

2.1. Development of the clinical scenarios

We conducted a literature search to identify clinical scenarios that relate to medication errors amenable to clinical decision support, which occurred with reasonable frequency within UK adult and paediatric inpatient populations.

Twenty‐two published papers were identified; many of the scenarios were primary care focused or contained incidents of inappropriate prescribing and potential prescribing omissions largely based on STOP/START ²³ and Beers criteria. ²⁴ Thomas et al. identified 80 high‐risk prescribing errors agreed by experts to result in possible patient harm. ²⁵ A related study by Fox et al. identified 41 prescribing indicators relevant to the paediatric inpatient setting. ²⁶ Both studies were conducted in the UK and included prescribing scenarios amenable to clinical decision support, which were highly relevant to this study. Other sources were utilised to identify relevant clinical scenarios including National Reporting and Learning System (NRLS) reports (n = 7), National Patient Safety Agency (NPSA) alerts (n = 6) and Medicines and Healthcare products Regulatory Agency (MHRA) guidance (n = 3).

After removal of duplicates, we extracted a final list of 170 clinical scenarios, which were presented to the Expert Panel as a statement. See Box 1 for further details. The statement described a potential prescribing error with an explanation of the significance of the error in brackets.

BOX 1: Example clinical indicators as presented in the e‐Delphi survey.

• Scenario 1.

  Low molecular weight omitted to be prescribed for prophylaxis when indicated (risk of venous thromboembolism)

• Scenario 2.

  Atazanavir prescribed concomitantly with proton pump inhibitor (risk of atazanavir treatment failure)

2.2. The e‐Delphi process

We set out to pilot our e‐Delphi survey first with a small group (n = 7), before asking our Expert Panel to complete a two‐round e‐Delphi survey, and have summarised the process in the accompanying flow chart (see Appendix).

2.2.1. Participants

Forty‐five experts were identified and invited to participate in the e‐Delphi study based on their expertise in medication safety and clinical informatics. The experts were known to the ePRaSE project board and included a range of UK healthcare professionals such as doctors (n = 10), nurses (n = 3), pharmacists (n = 31) and pharmacy technicians (n = 1), employed in a variety of health care settings and across a breadth of UK geographical locations.

2.2.2. e‐Delphi pilot

Prior to the launch of the e‐Delphi, a pilot was conducted to explore the suitability and usability of the survey. We invited seven senior clinical pharmacists, representing a range of clinical specialities, to participate in the pilot, sending them a link to the online survey. These pharmacists were not invited to be on the Expert Panel. All seven participants completed the survey and attended a group meeting at which they provided useful feedback on the suitability and readability of the clinical scenarios. Overall, the pilot e‐Delphi participants reported the scenarios were appropriate and clearly presented. Changes to the wording of two of the scenarios were made to improve clarity. Pilot participants raised concerns about the potential for external factors to influence the level of harm, such as dose adjustments recommended in renal impairment as these may be dependent on the presentation of the disease (whether acute or chronic) and the age of the patient. These concerns were fed back to ePRaSE board who acknowledged this. The e‐Delphi was presented as one continuous document and the pilot participants also reported how they were unable to save answers until the end, which was inconvenient. The survey was amended and scenarios separated by error category, thus allowing answers to be saved after each section. Three additional scenarios were suggested by the pilot participants for inclusion, resulting in a total of 173 scenarios in the final survey. The participants also recommended the addition of a “don't know” option for each risk score, as they felt that this would allow participants to provide answers related only to their expertise.

Defining the ePRaSE risk score

Initially, participants were asked to assign two risk scores to each prescribing scenario based upon the National Reporting and Learning System (NRLS) risk matrix, which utilises a numerical rating scale to relate the likelihood of an event (1–5) and the level of harm 1 (insignificant) – 5 (catastrophic). ²⁶ Pilot e‐Delphi participants reported a number of challenges with this, as the likelihood of the event and consequent harm occurring was subject to interpretation. For example, taking the clinical scenarios 1 (Box 1), participants reported that the prescribing event occurs frequently and the patient harm occurs infrequently, but the associated harm could be major. For scenario 2 (Box 1), the prescribing event would occur infrequently and patient harm is extremely likely, but the associated harm could also be major. Both of these clinical scenarios are significant for different reasons. Consequently, participants were asked to use a revised scoring system, adapted from the NRLS, but with three dimensions as described in Table 1.

TABLE 1.

ePRaSE risk scoring system adapted from NRLS risk score matrix

Likelihood of occurrence of the prescribing event	Level of harm associated with the prescribing event	Likelihood of harm occurrence
5. Very likely to occur on many occasions (e.g., at least once per month). 4. Likely to occur but not every day (e.g., quarterly) 3. May occur occasionally (e.g., at least annually) 2. Unlikely to occur, but possible (e.g., once every 5 years) 1. Very unlikely to occur (once in a decade/not at all)	5. Catastrophic – Incident causing death 4. Major – Incident that contributed to, but not the direct cause of death 3. Moderate–semi‐permanent harm taking 1 month to 1 year to resolve or requires a hospital stay 2. Minor – Short term harm, less than 1 month or requiring additional monitoring 1. Insignificant – Near miss or no harm to the patient.	5. Very likely to occur on many occasions (e.g., at least once per month) 4. Likely to occur but not every day (e.g., quarterly) 3. May occur occasionally (e.g., at least annually) 2. Unlikely to occur but possible (e.g., once every 5 years) 1. Very unlikely to occur (once in a decade/not at all)

Risk scores:

Potential risk = likelihood of occurrence of the prescription event (column 1) × level of harm associated with the prescribing event (column 2).

Actual risk = level of harm associated with the prescribing event (column 2) × likelihood of harm occurrence (column 3).

The ePRaSE risk scores were then categorised as low (1–3), moderate (4–6), high (8–12), and extreme (15–25) in line with the NRLS risk score. ²⁷

2.3. Round 1 (exploratory)

The e‐Delphi survey was presented to the Expert Panel via an online survey platform on 1 October 2018; background information was provided followed by a consent statement, which required completion prior to accessing the e‐Delphi survey. Participants were asked to rate the risk associated with each scenario, utilising the revised risk score, and comment on the suitability and wording of the scenarios. They could also suggest additional scenarios for inclusion. Participants were kindly requested to complete the survey within 2 weeks. On completion, the potential risk and actual risk score were calculated for each participant response and median scores were calculated for each scenario. The percentage of participant consensus with the median risk score was calculated.

2.4. Round 2

The second round utilised a modified survey, excluding both (a) the scenarios where consensus had been achieved, defined as ≥70% participant consensus ²⁸ for actual or potential harm risk score, and (b) scenarios with no consensus, which we defined as less than 50% consensus with the median score on both risk scores. We included new scenarios suggested by the expert participants. Participants were provided with their own individual risk scores from the first round and the median scores for each scenario. This provided the participants with an opportunity to modify their responses, in light of the judgements made by the rest of the Expert Panel, or to retain their original risk scores if deemed appropriate.

3. RESULTS

Of the 45 experts who were invited to participate, 24 consented to participate. Fifteen participants completed Round 1 of the e‐Delphi and 13 participants completed Round 2. The professional groups of the experts who completed each round are outlined in Table 2. Participants' additional roles included e‐prescribing/informatics leads and both national and regional roles in medication safety.

TABLE 2.

Professional groups of expert participants

Profession	Round 1 (n = 15)	Round 2 (n = 13)
Doctor	2	1
Nurse (informatics)	1	1
Pharmacist	11	10
Pharmacy technician (informatics)	1	1

3.1. Round 1

All 15 experts agreed on the category of risk for 51 of the 173 scenarios presented. Thirty‐seven scenarios achieved low levels of consensus. Of the 11 scenarios that participants suggested should be added, five were included in Round 2. Of the remaining six scenarios suggested, four duplicated some of the themes already represented and two represented error categories that were already well represented. In total, 90 scenarios were taken forward to Round 2.

3.2. Round 2

Thirteen participants completed the survey and consensus was achieved for a further 85 out of 90 scenarios; it was therefore not considered necessary to proceed to Round 3. Consensus was not achieved for both actual and potential harm for five scenarios which included a range of scenario categories; drug–drug interactions (n = 1), drug–dose (n = 1), drug–laboratory test (n = 1) and drug–brand (n = 2).

In total, expert consensus was reached on the risk category of 136 out of 178 scenarios across both e‐Delphi rounds. Of these, four scenarios were classed as extreme risk with a median potential or actual risk score between 15 and 25, 127 scenarios were classed as high risk with a median risk score of between 8 and 12, and five scenarios were classed as low or moderate risk with a median risk score of <8. An example of scenarios classified as extreme risk, high risk and low or moderate risk are provided in Table 3.

TABLE 3.

Example scenarios with risk category and consensus scores

Scenario category	Scenario description	Median risk score	Risk category	Percentage consensus with median
Drug–allergy	Any medication prescribed for a patient with a documented allergy to the medication (risk of severe adverse drug reaction)	16.0	Extreme	90
Drug–drug interaction	Potassium‐sparing diuretic (excluding aldosterone antagonists) prescribed concomitantly with angiotensin‐converting enzyme inhibitor or angiotensin II receptor antagonist (increased risk of severe hyperkalaemia)	12.0	High	100
Drug–age contraindication	Tetracycline prescribed to a child under 12 years (may result in deposition in growing bone and teeth causing staining/dental hypoplasia)	6.0	Low/moderate	88

There were very few scenarios classified as extreme risk (n = 4), which included drug‐allergy scenarios and scenarios relating to safe prescribing of anticoagulants. The most common categories represented include drug–disease scenarios (n = 37), drug–drug interactions (n = 25), and drug–laboratory tests (n = 24). The categories of the extreme and high‐risk scenarios are described in Table 4.

TABLE 4.

Prescribing scenario categories for extreme or high‐risk scenario

Prescribing scenario category	Number of extreme or high risk scenarios	Drug class	Number of extreme or high risk scenarios
Drug–allergy	2	Analgesics (including opioids)	12
Drug–age	11	Anticoagulants	11
Drug–brand	3	Antimicrobial	28
Drug–lab	24	Cardiovascular system	17
Drug–drug interactions	27	Central nervous system	13
Drug–disease	37
Drug–dose	18
Drug–omissions	4

4. DISCUSSION

This e‐Delphi study established expert consensus on the level of risk associated with 136 out of a total of 178 clinical scenarios, with the majority of scenarios categorised as high‐risk. We plan to use these scenarios in the development of the ePRaSE tool, as they cover a wide range of clinical events and are amenable to clinical decision support. This includes both basic clinical decision support, such as drug–drug interactions (DDIs), drug–allergy identification, therapeutic duplication and basic dosage recommendations as well as advanced clinical decision support such as renal and age‐related dosing advice; drug–disease contraindications and advice relating to corollary tests. ²⁹ The most common categories represented in the e‐Delphi survey included drug–disease contraindications, drug–drug interactions and scenarios involving drug–laboratory test interventions, which represent the categories of clinical decision support with the greatest volume of evidence related to change in prescriber behaviour and improved patient safety. ³⁰ , ³¹

The drug–drug interactions identified as high‐risk in this study show some similarities and some differences with previous published high priority drug–drug interactions. ³² Phansalkar et al. gained consensus regarding high priority drug–drug interactions in a US setting, therefore differences in prescribing rates may, in part, explain the different priorities obtained by experts in the UK. Some of our drug–drug categories reflected themes identified in a systematic review and meta‐analysis of drug–drug interactions associated with hospital admission/visits, such as interactions resulting in increased risk of bleeding complications and interactions associated with prolongation of the QT interval, but a similarity in the specific drug–drug interactions was not observed. ³³ This lack of consistency reflects the subjective nature of drug–drug interaction categorisation.

There are challenges associated with the prevention of high‐risk prescribing errors involving drug–disease contraindications and drug–laboratory tests utilising advanced clinical decision support. Interaction with other components of the health record is required, in particular the EHR and laboratory information management system (LIMS), to access the clinical data required. Many different EP systems are employed in NHS hospitals, with some hospitals utilising several systems concurrently. ¹⁵ Some hospitals employ integrated EP systems, which form a component of the EHR, whereas others employ stand‐alone EP systems used in isolation or combined with other stand‐alone packages. Consequently, these different information technology and software applications need to be able to communicate, exchange and use data accurately and effectively (system interoperability). There are substantial benefits associated with improving access to complete and accurate patient records and enhancing communication between healthcare professionals; however, barriers to system interoperability persist due to the complexity of the healthcare domain, system incompatibilities and resistance to change. ³⁴ In addition, the information regarding drug–disease contraindications listed in the electronic Medicines Compendium Summaries of Product Characteristics is vast, and so prioritisation is required to avoid overalerting and associated alert fatigue. ³⁵ , ³⁶

A systematic review of the incidence, causes and consequences of preventable adverse drug reactions occurring in inpatients reported cardiovascular drugs, analgesics, anticoagulants, opioids and antibiotics/anti‐infective agents to be the drug classes most frequently associated with preventable adverse events. ³⁷ Our study included medications from all of these classes, with scenarios involving antimicrobial therapies most commonly represented. This is also consistent with other studies that identify prescribing scenarios amenable to clinical decision support. ²⁵ Many of the scenarios involve high‐risk medicines, which, by definition, are more likely to cause significant patient harm. ³⁸ A recent systematic review concluded that clinical decision support can improve the safe use of high‐risk medicines like anticoagulants with both improved adherence to guidelines, and increased therapeutic drug monitoring reported. ³⁹ In addition, EP with clinical decision support is perceived to be a key enabler of the implementation of antimicrobial stewardship initiatives to reduce global antimicrobial resistance. ⁴⁰

4.1. Limitations

Our study has a few limitations. Firstly, the number of expert participants who completed both rounds of the e‐Delphi was lower than anticipated. Although the number of participants recruited to a Delphi study in the literature can be anywhere between 10 and 50, a response rate of ≥70% should be maintained. ⁴¹ Fifteen participants completed Round 1 and 13 completed Round 2, which equates to a response rate of 60% and 52% respectively. Our low overall response rate may have been associated with the number of scenarios we included, which represented a considerable time commitment for participants. This hypothesis is supported by research that suggests Delphi studies with higher numbers of items are associated with significantly lower response rates. ⁴² Although the response rate was lower than expected, retention of participants between Round 1 and Round 2 was 87%, which improves the reliability of the findings. Secondly, 42 of the scenarios presented to the expert participants did not achieve a consensus. This included scenarios that were classified as high risk by some participants. These scenarios were not incorporated into the ePRaSE tool, however, may represent significant prescribing safety concerns. Thirdly, the experts invited to participate in the e‐Delphi included a range of healthcare professionals from across the UK; however, pharmacists were over‐represented in the final sample with pharmacists representing 59% and 69% of participants in Rounds 1 and 2, respectively. Further research involving a diverse range of prescribers could be recommended to strengthen the findings and gain consensus on the risk represented by the remaining 42 scenarios. Fourthly, in Round 2, participants were provided with their individual risk scores from Round 1 along with median scores for each prescribing scenario. This could potentially bias their responses in Round 2. Fifthly, we developed a risk score based on the likelihood of the prescribing event, level of harm and likelihood of harm. Variability in interpretation is likely, based upon participants' personal characteristics and experiences. Finally, the study was carried out in the UK to inform the development of an e‐prescribing risk and safety evaluation tool to be used within UK hospitals, and so the findings may not be generalisable beyond the UK.

5. CONCLUSION

The e‐Delphi technique has been used to reach consensus on a set of high‐risk prescribing scenarios to be used to inform the development of a simulation tool to evaluate the safety of e‐prescribing systems in the UK. The scenarios represent prescribing events frequently associated with patient harm, including high‐alert medication and antimicrobial therapies, and address key concepts in e‐prescribing system optimisation such as improving patient specificity and promoting system interoperability. As well as providing individual feedback to NHS hospitals, national data will identify good performance to promote shared learning. Regular review of this list will be required to ensure continuing clinical relevance and to identify new emerging prescribing risks.

COMPETING INTERESTS

The authors report there are no conflicts of interest to declare.

CONTRIBUTORS

N.W. and A.S. conceived the presented idea. J.H. and S.K. developed the e‐Delphi and analysed the results. J.H., S.P.S. and A.H. wrote the manuscript. All authors discussed the results and approved the final manuscript.

APPENDIX A.

The Delphi Process

Heed J, Klein S, Slee A, Watson N, Husband A, Slight SP. An e‐Delphi study to obtain expert consensus on the level of risk associated with preventable e‐prescribing events. Br J Clin Pharmacol. 2022;88(7):3351-3359. doi: 10.1111/bcp.15284

DATA AVAILABILITY STATEMENT

The datasets generated during the current study are not publicly available, to maintain the integrity of the e‐prescribing risk and safety evaluation (ePRaSE) assessment, which the data has been used to develop.