Skip to main content
PMC home page
British Journal of Clinical Pharmacology logoLink to British Journal of Clinical Pharmacology
. 2013 Sep 6;76(Suppl 1):78–90. doi: 10.1111/bcp.12185

Clinical evaluation of the ADE scorecards as a decision support tool for adverse drug event analysis and medication safety management

Werner O Hackl 1, Elske Ammenwerth 1, Romaric Marcilly 2, Emmanuel Chazard 3, Michel Luyckx 4,5, Pascale Leurs 4, Regis Beuscart 3
PMCID: PMC3781682  PMID: 24007454

Abstract

Aims

The prevention of adverse drug events (ADEs) demands co-ordination of different health care professionals. ADE scorecards are a novel approach to raise the team awareness regarding ADE risks and causes. It makes information on numbers and on possible causes of possible ADE cases available to the clinical team. The aim of the study was to investigate the usage and acceptance of ADE scorecards by healthcare professionals and their impact on rates of possible ADEs.

Methods

ADE scorecards were introduced in three departments of a French hospital. A controlled time series analysis of ADE data was conducted to assess the impact of the ADE scorecards. In addition, qualitative interviews and a standardized survey with all participating staff members were performed.

Results

Physicians, nurses and pharmacists found ADE scorecards effective to increase medication safety and recommended future usage. The time-series analysis did not show changes in rates of possible ADEs.

Conclusion

ADE scorecards appear to be useful to raise awareness of ADE-related issues among professionals. Although the evaluation did not show significant reductions of ADE rates, the participating physicians, nurses and pharmacists believed that the ADE scorecards could contribute to increased patient safety and to a reduction in ADE rates. Strategies need to be designed to integrate ADE scorecards better into the clinical routine and to increase the precision of ADE detection.

Keywords: evaluation studies, medical order entry systems, medication therapy management, patient safety

WHAT IS ALREADY KNOWN ABOUT THIS SUBJECT

Adverse drug events (ADEs) occur frequently in all types of hospitals. Their prevention demands careful co-ordination of different professionals. Decision support tools integrated into the medication management cycle can improve medication safety, but mainly target individual professionals. Integrated team-oriented tools to increase ADE awareness and to increase medication safety are still rare.

WHAT THIS STUDY ADDS

ADE scorecards automatically identify and present cases of possible ADEs in a given department. A clinical study has shown that ADE scorecards are useful to raise awareness of ADE-related issues among professionals and that professionals appreciate their potential to contribute to medication safety. However, an impact of ADE scorecards on rates of possible ADEs has not yet been shown.

Introduction

Adverse drug events (ADEs) are an important challenge for health care worldwide [1]. An ADE is defined as ‘any injury occurring during the patient's drug therapy and resulting either from appropriate care, or from unsuitable or suboptimal care’ [2]. At least 25% of ADEs are related to medication errors and are thus preventable [3].

The medication cycle in hospitals covers several inter-related steps including prescribing, communicating, dispensing, administering and monitoring [4]. It is a highly distributed process in which information has to be passed along to different professionals such as physicians, pharmacists and nurses [5, 6] as well as to the patient [7], with all steps requiring careful co-ordination [8]. Medication safety, defined as ‘freedom from accidental injury during the course of medication use’ [2], can thus be seen as a team task, as all professionals need a ‘common ground’ of knowledge and representation [9], or ‘team situation awareness’ [10]. Several approaches to prevent ADEs have been described, including organizational [11] and technical interventions such as computerized physician order entry (CPOE) systems with decision support [12]. CPOE systems can issue ADE-related alerts through active monitoring procedures [13, 14]. However, the impact of CPOE systems on ADE rates has been found to be limited [15–17].

A novel approach to prevent ADEs is to make the team aware of ADE risks and their underlying causes, thus raising their ‘team ADE awareness’ [18]. Other approaches, such as CPOE systems, do not support this very well, as their alerts normally target only one professional. We attempted to increase ‘team ADE awareness’ by making automatically derived information on the number and on the possible causes of recent possible ADE cases available to the entire team [19] using a tool called ADE scorecards [20].

Scorecards have long been used in the economic sciences and in various areas of industry to support strategic management decisions. Key data from domains of interest are collected, aggregated and presented to the decision maker to allow a comprehensive view on the subject of interest [21].

The ADE scorecards were developed within the European Union project ‘Patient Safety through Intelligent Procedures in Medication’ (PSIP) [22, 23]. The project aimed, among other things, at automatically identifying and reducing preventable ADEs, characterized according to the National Council for Medication Error Reporting and Prevention (NCCMERP) [24] as ‘monitoring errors’, mostly ‘inadequate monitoring of clinical/lab parameters’. As part of a first clinical study, the ADE scorecards were introduced into a French hospital. A public version of the tool is available on the web for demonstration [25].

The aims of this study were 1) to investigate the usage and acceptance of ADE scorecards by the involved health care professionals and 2) to investigate the impact of ADE scorecards on rates of possible ADEs.

Methods

Clinical setting

The study took place in Denain General Hospital, a 416-bed hospital situated in northern France. This hospital is equipped with a comprehensive clinical information system (DxCare®), which includes an electronic medical record (EMR) and a CPOE to facilitate the computerized ordering of drugs, imaging and laboratory tests.

ADE scorecards

Design and objective

The ADE scorecards tool was developed following a user-centred design strategy [26]. Representatives of end-users were involved in the design process from the beginning in an iterative design evaluation process [27, 28].

The ADE scorecards aimed to provide health care professionals (e.g. physicians, nurses, pharmacists, quality managers) with detailed information about ADE cases suspected of having previously occurred in their department. The motivation for using ADE scorecards was to make the team aware of possible ADE cases and to learn how to avoid such ADEs in the future.

Generation of ADE data for ADE scorecards

The ADE data presented in the ADE scorecards were generated by applying pre-defined rules to patient data. These rules were developed in the PISP project using the following approach:

First, a common data model containing 92 data fields describing supposed ADE-related information items (e.g. demographic data, admission and discharge data, diagnoses, drug administrations, medical procedures, laboratory results, etc.) was developed [29]. All data were standardized using ICD-10 30 for diagnoses, ATC 31 for drugs and C-NPU classification (IUPAC, 32) for laboratory results. This data model represents a ‘minimal ADE detection dataset’.

In a second step, clinical data from 130 700 inpatient stays from the hospital in Denain and other PSIP partner hospitals were imported into the data model. The following data mining steps were then conducted [19]:

  1. Aggregation and transformation of the data into time-related events with start and stop dates (e.g. the 18 000 ICD 10 codes were aggregated to 48 categories of chronic diseases, the 5400 ATC codes were aggregated into 250 drug categories).

  2. Classification of the events as either ‘potential cause of ADE’ (e.g. administration of a drug) or ‘potential outcome of ADE’ (e.g. abnormal laboratory value or antidote administration).

  3. Automatic calculation of statistical associations between causes and outcomes (e.g. ‘age >70 years and potassium sparing diuretic ⇒ hyperkalaemia’) using decision trees and association rules [19, 33].

  4. Filtering of associations: only associations containing drugs in their list of potential causes were kept and validated against scientific knowledge by a group of experts (pharmacologists, pharmacists and physicians).

As a result, data mining resulted in 236 validated rules to detect automatically 27 classes of possible ADEs and their causes. While the approach was able to detect various kinds of possible ADEs, it was particularly optimized to detect hyperkalaemia, vitamin K antagonist (VKA) overdose, renal failure and anaemia. These ADE classes represented frequently occurring ADEs in the involved departments and were selected based on the preferences of the participating departments (Table 1).

Table 1.

Departments and practitioners participating in the ADE scorecard study (numbers from July 2010)

Department Involved health care professionals Number of beds (number of patients/year) Outcomes of special interest (defined by department head physician)
Study Department A (Cardiology and Gastroenterology) 2 physicians, 1 head nurse, 3 nurses 25 (1,340) Hyperkalaemia Renal failure Vitamin K antagonist overdose
Study Department B (Internal Medicine and Infectious Diseases) 1 physician, 1 head nurse, 2 nurses 25 (800) All
Study Department C (Acute Geriatric Care ) 1 physician, 1 head nurse, 1 nurse 10 (390) All (especially interested in renal failure)
Control Department D (Surgery) No use of ADE scorecards 56 (1,500) –
Control Department E (Pulmonology) No use of ADE scorecards 30 (880) –
Pharmacy 2 pharmacists n.a. (5,000) All
Open in a new tab

The rules used in the ADE scorecards consist of a set of conditions that lead to an effect. The conditions can be related to demographic characteristics (such as age and gender), drug administrations and drug discontinuations, laboratory results or diagnoses. The rules do not take into account drug dosing and route of administration. Details of the rules can be found in [19] and [20].

ADE scorecards were generated based on these 236 ADE detection rules. Patient data were imported into the data model and the rules were applied to these data to detect possible ADEs. These possible ADEs are not necessarily actual ADEs and need to be confirmed by an expert review.

To assess the positive predictive value of the ADE detection rules, a sample of 24 753 patient records was imported into the data model and 997 possible ADEs (= 4% of all hospitalizations) were detected by the ADE scorecards, including 507 cases of possible drug-associated hyperkalaemia. A manual expert review confirmed that 271 of these hyperkalaemia cases were in fact actual ADEs. The positive predictive value of the ADE scorecards in this sub-study was found to be 53.5%, the sensitivity was 95.1% and the specificity 52% [19]. This was found sufficient for further clinical evaluation.

User interface of the ADE scorecards

The ADE scorecards provide users with web-based, password-restricted access to the following ADE data concerning their own department:

  1. ‘Synthesis’ (Figure 1): First page that presents classes of possible ADEs that occurred in a department and number of related cases per month. From this page, users can select an ADE class of interest (part 3) for which ‘detailed statistics’ will be generated.

  2. ‘Detailed statistics’ (Figure 2) displays:

    1. characteristics of the patients for whom the given ADE class occurred

    2. conditions (causes) that lead to the given ADE class and information on available evidence (e.g. literature)

    3. different statistical measures (confidence, which is the probability of the outcome once the causes are met, and median delay of the outcome after occurrence of the causes)

Figure 1.

Figure 1

Open in a new tab

ADE scorecards ‘Synthesis’ page. It provides an overview of detected possible ADEs in a department. Part 1 displays the number of detected ADEs per month. Part 2 is the graphical presentation of this information. Part 3 allows the generation of detailed statistics for selected ADE classes

Figure 2.

Figure 2

Open in a new tab

ADE scorecards ‘Detailed statistics’ page for hyperkalaemia. It provides details for an ADE class. Part 1 displays characteristics of the patients presenting with this ADE. Part 2 presents graphically the number of the selected ADEs that appeared per month (left) and their delay of appearance (right). Part 3 contains the conditions (patients' conditions, administered drugs) potentially leading to the selected possible ADE, with the number of identified cases, the statistical confidence of the rule and the median delay of appearance of the ADE. Supplementary information (description of the underlying rules, including scientific explanations, references and advice) are provided in part 4

It is possible to select the patients for whom a possible ADE has been detected:

  • 3 ‘Case review facility’ (Figure 3) contains a patient record synthesis, presenting details of the inpatient stays that were affected by a possible ADE (such as demographic information, diagnoses, procedures, medications, laboratory results and important free-text documents).

Figure 3.

Figure 3

Open in a new tab

ADE scorecards ‘Case review facility’ page. It displays detailed information for a patient where a possible ADE was found. In this case, the patient had renal failure and received high molecular weight heparin, which caused hyperkalaemia. Part 1 represents the medication time line of the patient, part 2 the development of laboratory parameters of interest and part 3 diagnoses and other characteristics of interest

Study design

The evaluation questions were:

  1. Are the ADE scorecards used and accepted by the different user groups?

  2. Do the ADE scorecards have an effect on rates of possible ADEs in a given department?

The study was designed as a quasi-experimental field study using an interrupted time series design with control group to assess the impact of the ADE scorecards on ADE rates. The targeted outcome measure was monthly rates of possible ADEs. In addition, log files, qualitative interviews and a standardized survey were conducted.

Participants and study flow

Five medical units of the Denain Hospital where sufficient structured clinical EHR data to match the PSIP data model were available were eligible for the study. Those already involved in the PSIP project (cardiology/gastroenterology, internal medicine/infectious diseases and the acute geriatric care departments) were chosen as study wards and their head physicians were asked for consent. In these study wards, ADE scorecards were implemented and physicians and (head) nurses were invited to participate. The two remaining wards (surgery and the respiratory departments) were chosen as control wards without implementation of ADE scorecards.

In the study wards, ADE scorecards were made available via Intranet to physicians, (head) nurses and hospital pharmacists (Table 1). While physicians and nurses had access only to their own unit, pharmacists had access to ADE scorecards for all study wards.

The users were encouraged to consult at least those ADEs defined by the head physician (c.f. Table 1). At the beginning of the study in July 2010, scorecards contained ADE information for the first 4 months of 2010 and for all of 2009, 2008 and 2007. The ADE information was then updated about every 2 months.

To ensure that users would regularly look at the ADE scorecards, meetings were organized at regular intervals in which the ADE scorecards were presented and discussed. The evaluation team, composed of ergonomists and physicians, animated these meetings. In total, there were 21 meetings: three rounds of meetings for physicians in each study ward, three for pharmacists and three for nurses in each of the study departments.

Methods for data acquisition and data analysis

To assess usage of ADE scorecards, log files recording all activities on the ADE scorecards were analyzed regarding time, user, department and accessed ADE scorecard. Due to technical changes, the recording of the log files ended in March 2011.

To assess user acceptance, interviews and a survey were conducted. The semi-structured interviews with all user groups were conducted as part of the presentation meetings described above to discuss experienced benefit and future intention to use. These interviews were audio-recorded and analyzed using qualitative and quantitative content analysis [34]. This analysis developed categories for the topics addressed in the interviews: experienced benefit and future intention to use. In addition, at the end of June 2011, after nearly 1 year of use, a standardized short survey comprising 15 items was distributed to all users. This survey was pre-tested but not formally validated. Users were asked to give their summarizing opinion on the ADE scorecards after they had used them for 1 year. Descriptive statistics were used to evaluate this survey.

To assess changes in ADE rates, the available ADE numbers from January 2007 to March 2012 were exported from the ADE scorecards using an open source business intelligence and reporting platform (BIRT) [35]. First, a simple comparison of ADE numbers and ratios in equal intervals of 15 months pre- (April 2009–June 2010) and post-intervention (July 2010–September 2012) was performed.

To assess the longitudinal impact of the intervention ‘ADE scorecards introduction’ on ADE rates, segmented regression analysis according to [36] was performed using SPSS® version 17. The analyses took into account the baseline levels of the ADE rates at the start of the observation and tested for changes in the level of ADE rates directly after the introduction of ADE scorecards as well as for changes in slope of the ADE rates from pre- to post-introduction of the ADE scorecards. The analyses were performed for each single department and for a comparison of study and control wards. To test the time series for serial autocorrelation, the Durbin–Watson statistic was used. Statistical significance was defined as P < 0.05 for all tests.

Results

ADE scorecards usage

Between July 2010 and the end of March 2011, 441 connections to the ADE scorecards were recorded and 405 inpatient stays were accessed via the ‘Expert Explorer’ (cf. Table 2).

Table 2.

Number of times the ADE scorecards were accessed from the start of the intervention in July 2010 to 31 March 2011. The ADE classes marked in red are those considered ‘of special interest’ (c.f. Table 1)

Inline graphic
Open in a new tab

Overall, users in the different units looked at the same kinds of ADEs. Among the six top assessed ADE scorecards, the three which were considered to be ‘of special interest’ by the Department A chief physician were most frequently accessed (c.f. Table 2).

Every professional group used the ADE scorecards, but pharmacists looked at the ADE scorecards more than other professionals. While the physicians in Department C and in Department A accessed the ADE scorecards numerous times, the involved Department B physician accessed these only once.

User acceptance, experienced benefit and future intention to use

Among the 15 healthcare professionals involved, 13 were interviewed during the first two presentation meetings and 11 answered every topic tackled.

Experienced benefit

Three major categories of experienced benefits were present in the interviews: 1. ADE scorecards as learning and supporting tool; 2. usefulness of information presented by ADE scorecards; 3. ADE scorecards as an inter-professional dialogue-supporting tool.

First category: All 11 participants found the information presented by the ADE scorecards useful to learn and to improve work: ‘to have feedback on the practice’ and ‘to avoid the recurrence of an ADE’ (pharmacist), ‘to get information’ (Department B nurse), ‘to change bad habits’ (Department A physician), like a ‘refresher course’ (Department A nurse). For example, before the ADE scorecards, some nurses seemed not to be aware that antibiotics increased the effect of VKAs on the International Normalized Ratio (INR). Some participants mentioned that such a tool could also train students: ‘very interesting for medical students’ (Department A physician).

Second category: Almost all physicians (except one) and all pharmacists stated that ADE scorecards presented new and valuable information: ‘There are things [in the ADE scorecards] we see every day, [but that] we don't even notice’ (Department A physician); ‘There are very interesting things: some medications are given easily and are not always monitored’ (Department C nurse).

Third category: ADE scorecards were also seen as an inter-professional dialogue-supporting tool. On several occasions, pharmacists and physicians discussed the opportunity of using alternative medications. For instance, a pharmacist and physicians (Department A) discussed the justification of medications detected as causing hyperkalaemia. Also, discussions between nurses and physicians were reported that were triggered by information from the ADE scorecards. ADE scorecards were perceived as supporting decision-making: ‘to change practices’ by ‘support[ing] the physicians' decisions’ (Department B nurse), ‘[to] support the therapeutic decision’ (Department A physician).

The following two clinical situations show how ADE scorecards helped to teach and to optimize drug management, contributing to prevent iatrogenic risks that may lead to extended hospitalizations:

  1. One pharmacist became more aware of the risk of INR increase when giving paracetamol (acetaminophen) to patients already being treated with VKAs. In France, acetylsalicylic acid is contraindicated for patients receiving VKAs and paracetamol is often chosen as an alternative. Paracetamol, however, may also increase the activity of VKAs due to a specific hepatic enzyme inhibition. Using information from the ADE scorecards, the pharmacist became more vigilant and now recommends, after a delay depending on the type of VKA used, checking INR values in cases where physicians prescribe paracetamol in high dosages to patients being treated with VKAs. The pharmacist felt that this reduced the risk of haemorrhage for these patients.

  2. A physician from Geriatrics reported on situations where an antibacterial therapy is associated with proton pump inhibitors (PPI) for patients with renal insufficiency. In these cases, ADE scorecard information made the physicians aware that the association of antibacterial therapy with a PPI may have an impact on renal function. Therefore, renal function is now more closely monitored.

Future intention to use

All 11 interviewees expressed their intention to use the ADE scorecards if they were integrated into an ADE prevention approach. Ten out of 11 participants were convinced that ADE scorecards could help them to prevent the appearance of ADEs. One participant (Department B physician) thought that ‘retrospective data are not useful’ to prevent ADEs. This physician would prefer getting information through a decision support system integrated into the CPOE.

Users' opinions on ADE scorecards after 1 year of use

Twelve respondents (four physicians, two pharmacists, three nurses and three head nurses) answered the standardized survey (Figure 4). All respondents believed that ADE scorecards were a useful tool to learn about ADEs that occurred in their department and that ADE scorecards had the capacity to improve patient safety. All responding users would recommend using the ADE scorecards to their colleagues and all physicians believed that the use of the ADE scorecards influenced their prescriptions.

Figure 4.

Figure 4

Open in a new tab

Results of user survey after 1 year of usage (n = 12). Inline graphic, agree; Inline graphic, partly agree; Inline graphic, partly disagree; Inline graphic, disagree; Inline graphic, no statement

Impact on rates of possible ADEs

ADE rates in all departments

Overall, ADE scorecards detected 3586 ADE cases in 20 983 patient stays in all observed departments (study and control) between January 2007 and the end of March 2012. From the 27 classes of ADEs detectable by PSIP [37], 24 were detected in the involved departments.

The simple comparison of ADE numbers and rates 15 months before and 15 months after implementation of the ADE scorecards showed that the ratio of possible ADEs detected by the ADE scorecards ranged from 78 to 305 per 1000 inpatient stays (Table 3).

Table 3.

Comparison of possible ADEs 15 months pre-intervention (April 2009–June 2010) and 15 months post-intervention (July 2010–September 2011) in each department

Number of inpatient stays (n) Detected cases of possible ADE (n) Detected ADE cases per 1000 inpatient stays
Pre/Post Pre/Post Pre/Post
Study Department A (Cardiology and Gastroenterology) Pre 1675 Pre 365 Pre 218
Post 1707 Post 294 Post 172
Study Department B (Internal Medicine and Infectious Diseases) Pre 1282 Pre 370 Pre 289
Post 1277 Post 366 Post 287
Study Department C (Acute Geriatric Care) Pre 452 Pre 138 Pre:305
Post 490 Post 121 Post:247
Control Department D (Surgery) Pre 1478 Pre 115 Pre:78
Post 2258 Post 192 Post:85
Control Department E (Pulmonology) Pre 1122 Pre 231 Pre: 21
Post 1052 Post 249 Post 24
Open in a new tab

ADE rates of selected ADE classes

Results of segmented regression analysis comparing the pre- and post-period in each department and comparing study departments vs. control departments up to September 2011 for the top four accessed ADE classes and for all ADE classes together, taking into account baseline ADE trends in all departments, showed no significant changes in ADE rates after the introduction of the ADE scorecards. No serial autocorrelation was found (Durbin–Watson test). As examples, the developments in possible ADE rates are presented for hyperkalaemia (Figure 5) and for renal failure (Figure 6).

Figure 5.

Figure 5

Open in a new tab

Percentage of patients for whom ADE scorecards detected hyperkalaemia. Time series January 2007–March 2012: control vs. study departments. Start and end of PSIP project are also indicated. Inline graphic, control wards; Inline graphic, study wards

Figure 6.

Figure 6

Open in a new tab

Percentage of patients for whom ADE scorecards detected renal failure. Time series January /2007–March 2012: control vs. study departments. Start and end of PSIP project are also indicated. Inline graphic, control wards; Inline graphic, study wards

Discussion

Answers to study questions

The participating physicians, nurses and pharmacists used the ADE scorecards repeatedly. Whereas the pharmacists consulted the scorecards for almost all ADE classes, the physicians and the nurses focused on certain ADE classes. In study department B, the participating physician used the scorecards only once. He expressed that he would prefer active clinical decision support functionality during the prescription.

All interviewed participants considered the ADE scorecards to be useful to support decision-making and they expressed their intention to use the ADE scorecards as part of an ADE prevention approach. In the survey conducted after 1 year of use, all respondents stated that they would recommend using the ADE scorecards to their colleagues. With the exception of one physician, the involved health care professionals were convinced that ADE scorecards could contribute to increased medication safety.

The controlled time series analysis of the top four accessed ADE classes from January 2009 to September 2011 did not show significant changes in rates of possible ADEs. Thus, neither a reductive nor a magnifying effect of the ADE scorecard implementation could be found.

Strengths and weaknesses

To our knowledge, this is the first time in the literature that the impact of the presentation of department-specific ADE statistics, including access to related clinical data of affected patients, has been evaluated and reported.

This study was designed as a field study and involved different professional groups. All participants volunteered to take part. This possible selection bias may have led to a more positive attitude in the interviews and surveys. Not all physicians and nurses from the study wards participated, which may have led to an underestimation of the possible effects of the ADE scorecards.

Another source of bias was the role of the pharmacists. As the pharmacists were responsible for checking the prescriptions and for giving feedback to prescribers in the entire hospital (including the control wards), it seems possible that a ‘wash-over effect’ diluted the potential effect of the ADE scorecards on preventing ADEs.

Rates of possible ADEs were analyzed over a period of 4 years, which is quite a long period of time. The ADEs were detected using validated rules derived from data mining of patient chart information. These rules were validated beforehand and showed a positive predictive value of around 50% in a sub-study. This means that around half of the cases presented to the users may not be real ADE cases (for example, anaemia caused by the underlying disease and not by drug management). This may explain the relatively low uptake of the ADE scorecards. For future usage of ADE scorecards, the positive predictive value of the rules needs to be further increased.

As a randomized controlled trial was not possible due to organizational reasons, we chose the strongest quasi-experimental approach for evaluating longitudinal effects of interventions, the interrupted time series design [36]. We further strengthened this by involving a control group. We took into account the heterogeneity of the involved departments by first comparing pre- and post-intervention periods for each department and then by comparing study departments with control departments. For these comparisons, the baseline levels of ADE rates in the different departments were also included as parameters in the regression models. Moreover, we conducted interviews and surveys to gather additional information about the effect of the ADE scorecards.

Due to the nature of the chosen study design and the limited number of available wards, study wards and control wards were not fully comparable. The rates of detected possible ADEs in the study departments were significantly higher than in the control departments (Figures 5 and 6). In addition, the population of treated patients in the involved wards, especially in surgery, differs. However, alternatives for selecting wards that were more comparable were not available. The time series analysis showed that, despite these heterogeneities, the general trends in possible ADE rates were comparable in all departments.

The data show that the possible ADE rates for some ADE classes declined in both study and control wards long before the introduction of the ADE scorecards (c.f. Figures 5 and 6). Several explanations are possible here, such as changes in clinical work flow. Another explanation is the PSIP project that started in November 2008. Several clinicians and pharmacists in the hospital of Denain participated in developing the ADE scorecards in 2009 and 2010, and meetings and surveys with clinicians and managers were conducted on ADE-related issues. Even if they did not see information on ADE rates and causes in their own medical units during this time, their awareness of the ADE issue may have been increased before the actual beginning of the study. This could also have contributed to the continuous reduction of the ADE rate. This shows that ADE reduction is a complex process, where many factors can contribute. ADE scorecards need to be integrated in such an overall ADE reduction initiative.

Results in relation to other studies

Different approaches to detect ADEs and adverse drug reactions in hospitals exist, including spontaneous reporting of ADE cases by clinicians, detection of ADEs by trained observers (usually based on chart review), or automatic detection of ADEs by computer-assisted analysis of clinical data [38]. ADE scorecards are an example for the computerized detection of ADEs. Comparable with other approaches such as automatic laboratory signalling [39, 40], ADE scorecards use structured clinical information and pre-defined rules to detect possible cases of ADEs.

The uniqueness of the ADE scorecard lies in several aspects: Firstly, the rules used are derived from a data mining process using retrospective clinical data of the participating department. They are thus specifically tailored to each department. In addition, this statistical contextualization enables automated alert filtering based on statistics, which in turn enables decreased overalerting. Secondly, the rules consist of a set of conditions that may lead to an ADE, with those conditions not only including drug administrations, but also demographic characteristics, laboratory values, specific procedures considered equivalent to drug administrations and diagnoses. In addition, the rules take into account the effects of drug discontinuation. They do not, however, take into account drug dose or route. Thirdly, and maybe most importantly, ADE scorecards support a team awareness approach to drug safety. They do not address the individual physician at the time of prescription, but instead address the entire team (physicians, nurses, pharmacists) by making available detected possible ADE cases, possible causes and characteristics of affected patients. ADE scorecards are intended to be used for a retrospective analysis of recent ADE cases. For a more detailed discussion on the difference between the chosen approach and other ADE detection approaches, see [19].

The potential positive effects of ADE scorecards that were stated by the participants has also been supported in an international Delphi study with 69 CPOE experts [41]. In a survey, more than 70% of these experts stated that the concept of ADE scorecards has the potential to prevent ADEs. The experts also estimated that ADE scorecards could prevent around 10% of all ADE cases in a hospital where electronic prescription is in use. Our study was not yet able to show a reduction of ADE rates.

In the meantime, the ADE scorecards have also been successfully introduced in a specialized endocrinology hospital in Sofia, Bulgaria and a comparable survey was conducted. The 33 Bulgarian physicians responded very positively, all of them stating that they would recommend using the ADE scorecards to their colleagues (94.7% agreed, 5.3% partly agreed) and all of them believing that ADE scorecards may reduce ADEs (57.9% agreed, 42.1% partly agreed) [42]. The initial ADE scorecard implementation effort was around 8 person-months. Later regular updates of the ADE data took around 2 person-months for each update of the ADE scorecard data.

To our knowledge, this study is the first to evaluate the impact of the presentation of automatically-generated department-specific ADE statistics, and our experiences may be worthwhile for comparable projects. We assume that ADE scorecards will be used more often when their positive predictive value is improved, when usability issues are addressed, and when the ADE scorecards are better integrated into ongoing quality initiatives to improve medication safety in a hospital.

Significance and generalizability of the study

In general, while computer-based tools are considered useful for patient safety, controlled studies were often not able to show a significant reduction of ADEs [16, 17, 43, 44]. This may be due to limited study power, but also to the complexity of the medication process. In our opinion, ADE scorecards are a promising tool to raise ‘team ADE awareness’, although not in isolation, but as part of a ‘multifaceted approach’ [11], including other team interventions as well as technical interventions. ADE scorecards could also, for example, support the introduction of a clinical decision support tool by familiarizing professionals with a set of rules used.

In general, the ADE scorecards appear to be transferable to other hospitals when the same set of rules to detect ADEs is used. Types, numbers and causes of ADEs could then be compared between different hospitals, establishing a benchmark for ADE rates.

In conclusion, the clinical study indicated that ADE scorecards may be useful to raise awareness of ADE-related issues among professionals in their own department. Although the evaluation did not show significant reductions for the top four accessed ADE classes, the clinical users in the study departments and in the hospital pharmacy believed that the ADE scorecards might contribute to increased patient safety and to a reduction of ADEs. Strategies need to be designed to integrate ADE scorecards more successfully into the clinical routine and to increase precision of ADE detection. The hospital in Denain is preparing to re-launch an updated version of the ADE scorecards in additional departments in 2013.

Competing Interests

All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors had support from the European Community (Seventh Framework Programme (FP7/2007–2013), grant agreement no. 216130) for the submitted work, no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.

The research leading to these results received funding from the European Community's Seventh Framework Programme (FP7/2007–2013) under grant agreement no. 216130 – the PSIP project. The authors would like to acknowledge all healthcare professionals and technicians whose involvement made completion of this study possible.

References

  • 1.Bates D, Spell N, Cullen D, Burdick E, Laird N, Petersen L, Small S, Sweitzer B, Leape L. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. J Am Med Assoc. 1997;277:307–311. [PubMed] [Google Scholar]
  • 2.Council of Europe. Committee of Experts on Management of Safety and Quality in Health Care (SP-SQS) – Expert Group on safe medication practices: glossary of terms related to patient and medication safety. 2005.
  • 3.Institute of Medicine. Preventing Medication Errors: Quality Chasm Series. Washington, DC: National Academy Press; 2006. [Google Scholar]
  • 4.McKibbon KA, Lokker C, Handler SM, Dolovich LR, Holbrook AM, O'Reilly D, Tamblyn R, Hemens BJ, Basu R, Troyan S, Roshanov PS. The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials. J Am Med Inform Assoc. 2012;19:22–30. doi: 10.1136/amiajnl-2011-000304. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Beuscart-Zephir M, Pelayo S, Anceaux F, Meaux J, Degroisse M, Degoulet P. Impact of CPOE on doctor-nurse cooperation for the medication ordering and administration process. Int J Med Inform. 2005;74:629–641. doi: 10.1016/j.ijmedinf.2005.01.004. [DOI] [PubMed] [Google Scholar]
  • 6.Gillespie U, Morlin C, Hammarlund-Udenaes M, Hedstrom M. Perceived value of ward-based pharmacists from the perspective of physicians and nurses. Int J Clin Pharmacol. 2012;34:127–135. doi: 10.1007/s11096-011-9603-1. [DOI] [PubMed] [Google Scholar]
  • 7.Schwappach D, Frank O, Davis R. A vignette study to examine health care professionals' attitudes towards patient involvement in error prevention. J Eval Clin Pract. 2012 doi: 10.1111/j.1365-2753.2012.01861.x. doi: 10.1111/j.1365-2753.2012.01861.x. [DOI] [PubMed] [Google Scholar]
  • 8.Pirnejad H, Niazkhani Z, Berg M, Bal R. Intra-organizational communication in healthcare – considerations for standardization and ICT application. Methods Inf Med. 2008;47:336–345. [PubMed] [Google Scholar]
  • 9.Clark H. Using Language. Cambridge: Cambridge University Press; 1996. [Google Scholar]
  • 10.Salas E, Prince C, Baker D, Shrestha L. Situation awareness in team performance: implications for measurement and training. Hum Factors. 1995;37:123–136. [Google Scholar]
  • 11.Rommers MK, Teepe-Twiss IM, Guchelaar HJ. Preventing adverse drug events in hospital practice: an overview. Pharmacoepidemiol Drug Saf. 2007;16:1129–1135. doi: 10.1002/pds.1440. [DOI] [PubMed] [Google Scholar]
  • 12.Bates DW. Using information technology to reduce rates of medication errors in hospitals. BMJ. 2000;320:788–791. doi: 10.1136/bmj.320.7237.788. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Institute of Medicine. Preventing Medication Errors: Quality Chasm Series. Washington, DC: The National Academic Press; 2007. [Google Scholar]
  • 14.Kane-Gill SL, Visweswaran S, Saul MI, Wong AK, Penrod LE, Handler SM. Computerized detection of adverse drug reactions in the medical intensive care unit. Int J Med Inform. 2011;80:570–578. doi: 10.1016/j.ijmedinf.2011.04.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, Vander Vliet M, Seger DL. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:1311–1316. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]
  • 16.Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc. 2008;15:585–600. doi: 10.1197/jamia.M2667. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Eslami S, Abu-Hanna A, de Keizer NF. Evaluation of outpatient computerized physician medication order entry systems: a systematic review. J Am Med Inform Assoc. 2007;14:400–406. doi: 10.1197/jamia.M2238. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Marcilly R, Pelayo S, Hackl W, Beuscar-Zephir M, Ammenwerth E. Scorecards as ‘team ADE awareness’ support. 2011. In: Proceedings of the 5th International Symposium on Human Factors Engineering in Health Informatics, Trondheim, Norway, eds Svanaes, Faxvaag: Tapir Acdemic Press.
  • 19.Chazard E, Ficheur G, Bernonville S, Luyckx M, Beuscart R. Data mining to generate adverse drug events detection rules. IEEE Trans Inf Technol Biomed. 2011;15:823–830. doi: 10.1109/TITB.2011.2165727. [DOI] [PubMed] [Google Scholar]
  • 20.Chazard E, Baceanu A, Ferret L, Ficheur G. The ADE scorecards: a tool for adverse drug event detection in electronic health records. Stud Health Technol Inform. 2011;166:169–179. [PubMed] [Google Scholar]
  • 21.Kaplan RS, Norton DP. The balanced scorecard – measures that drive performance. Harv Bus Rev. 1992;70:71–79. [PubMed] [Google Scholar]
  • 22.Beuscart R, Hackl W, Nohr C. Detection and Prevention of Adverse Drug Events – Information Technologies and Human Factors. Amsterdam: IOS Press; 2009. [PubMed] [Google Scholar]
  • 23.The PSIP Consortium. Patient safety through intelligent procedure in medicine (PSIP). Lille, 2012. Available at http://www.psip-project.eu/ (last accessed 1 July 2013)
  • 24.NCC MERP – National Coordinating Council for Medication Error Reporting and Prevention. Taxonomy of medication errors. 2001. Available at http://www.nccmerp.org/medErrorTaxonomy.html (last accessed 1 July 2013) [DOI] [PubMed]
  • 25.PSIP Consortium. Online demonstration of the ADE scorecards. 2012. Available at http://psip.univ-lille2.fr/prototypes/public (last accessed 1 July 2013)
  • 26.ISO. International Standardization Organization. Ergonomics of Human System Interaction – Part 210: Human Centred Design for Interactive Systems (Rep N°9241-210) Genf: ISO; 2010. [Google Scholar]
  • 27.Marcilly R, Chazard E, Beuscart-Zéphir M, Hackl W, Băceanu A, Kushniruk A, Borycki E. Design of adverse drug events-scorecards. Stud Health Technol Inform. 2011;164:377–381. [PubMed] [Google Scholar]
  • 28.Marcilly R, Hackl WO, Pelayo S, Hassler S, Riccioli C, Beuscart-Zéphir M. Human-centered design of a scorecard tool for adverse drug events. Poster Presentation at Medical Informatics Europe (MIE 2011), Oslo, August 2011.
  • 29.Chazard E, Merlin B, Ficheur G, Sarfati JC, Beuscart R. Detection of adverse drug events: proposal of a data model. Stud Health Technol Inform. 2009;148:63–74. [PubMed] [Google Scholar]
  • 30.WHO. International classification of diseases: ICD-10. Available at http://www.who.int/classifications/icd/en (last accessed: 1 July 2013)
  • 31.WHO. Anatomical therpeutic chemical (ATC) classification system. Available at http://www.whocc.no (last accessed 1 July 2013)
  • 32.IUPAC. Nomenclature, properties, and units in laboratory medicine (C-NPU). Available at http://www.iupac.org (last accessed 1 July 2013)
  • 33.Chazard E, Bernonville S, Ficheur G, Beucart R. A statistics-based approach of contextualization for Adverse Drug Events detection and prevention. Stud Health Technol Inform. 2012;180:766–770. [PubMed] [Google Scholar]
  • 34.Mayring M. Qualitative Inhaltsanalyse: grundlagen und techniken: Utb, 2007.
  • 35.The Eclipse Foundation. BIRT: business intelligence and reporting tools. 2011. Available at http://www.eclipse.org/birt (last accessed 1 July 2013)
  • 36.Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27:299–309. doi: 10.1046/j.1365-2710.2002.00430.x. [DOI] [PubMed] [Google Scholar]
  • 37.Chazard E, Preda C, Bernonville C, Baceanu A, Ficheur G, Genty M, Darmoni S, Sakji S, Pereira S, Tessier S, Saur F, Serrot E, Kergourlay I, Beuscart R, Cacciabue C. D2.3 results of data and semantic mining. 2010. PSIP report.
  • 38.Thurmann PA. Methods and systems to detect adverse drug reactions in hospitals. Drug Saf. 2001;24:961–968. doi: 10.2165/00002018-200124130-00003. [DOI] [PubMed] [Google Scholar]
  • 39.Azaz-Livshits T, Levy M, Sadan B, Shalit M, Geisslinger G, Brune K. Computerized survelliance of adverse drug reactions in hospital: pilot study. Br J Clin Pharmacol. 1998;45:309–314. doi: 10.1046/j.1365-2125.1998.00685.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Tegeder I, Levy M, Muth-Selbach U, Oelkers R, Neumann F, Dormann H, Azaz-Livshits T, Criegee-Rieck M, Schneider HT, Hahn E, Brune K, Geisslinger G. Retrospective analysis of the frequency and recognition of adverse drug reactions by means of automatically recorded laboratory signals. Br J Clin Pharmacol. 1999;47:557–564. doi: 10.1046/j.1365-2125.1999.00926.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Riedmann D, Jung M, Hackl W, Ammenwerth E. How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study. J Am Med Inform Assoc. 2011;18(6):760–766. doi: 10.1136/amiajnl-2010-000006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Dimitrov H. Testbed for integration of CDSS modules and PSIP validation in University Specialized Hospital for Active Treatment of Endocrinology, Medical University, Sofia. In: Proceeding of the PSIP International Workshop, Sofia, 23 June 2011, eds Beuscart R, Tcharaktchiev D, Angelova G, Sofia: Incoma Ltd, 2011: 39–48.
  • 43.Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163:1409–1416. doi: 10.1001/archinte.163.12.1409. [DOI] [PubMed] [Google Scholar]
  • 44.Wolfstadt JI, Gurwitz JH, Field TS, Lee M, Kalkar S, Wu W, Rochon PA. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23:451–458. doi: 10.1007/s11606-008-0504-5. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from British Journal of Clinical Pharmacology are provided here courtesy of British Pharmacological Society

RESOURCES