Comparison of a basic and an advanced pharmacotherapy-related clinical decision support system in a hospital care setting in the Netherlands

Abstract

Objective

To compare the clinical relevance of medication alerts in a basic and in an advanced clinical decision support system (CDSS).

Design

A prospective observational study.

Materials and methods

We collected 4023 medication orders in a hospital for independent evaluation in two pharmacotherapy-related decision support systems. Only the more advanced system considered patient characteristics and laboratory test results in its algorithms. Two pharmacists assessed the clinical relevance of the medication alerts produced. The alert was considered relevant if the pharmacist would undertake action (eg, contact the physician or the nurse). The primary analysis concerned the positive predictive value (PPV) for clinically relevant medication alerts in both systems.

Results

The PPV was significantly higher in the advanced system (5.8% vs 17.0%; p<0.05). Significant differences were found in the alert categories: drug–(drug) interaction (9.9% vs 14.8%; p<0.05), drug–age interaction (2.9% vs 73.3%; p<0.05), and dosing guidance (5.6% vs 16.9%; p<0.05). Including laboratory values and other patient characteristics resulted in a significantly higher PPV for the advanced CDSS compared to the basic medication alerts (12.2% vs 23.3%; p<0.05).

Conclusion

The advanced CDSS produced a higher proportion of clinically relevant medication alerts, but the number of irrelevant alerts remained high. To improve the PPV of the advanced CDSS, the algorithms should be optimized by identifying additional risk modifiers and more data should be made electronically available to improve the performance of the algorithms. Our study illustrates and corroborates the need for cyclic testing of technical improvements in information technology in circumstances representative of daily clinical practice.

Introduction

The Hospital Admissions Related to Medication (HARM) study showed that 16 000 HARM are potentially avoidable each year in the Netherlands.¹ This finding prompted the HARM-wrestling report, which proposed about 40 practical recommendations to reduce the most frequently occurring and potentially avoidable HARM.² About half of these recommendations concerned appropriate prescribing (eg, adding a protective drug), a quarter concerned follow-up procedures (eg, laboratory monitoring), and another quarter concerned communication (with the patient and other healthcare providers). Many of the recommended actions cannot be postponed until the next medication review but should be carried out as soon as a treatment is started or changed.³ An important general recommendation was to improve, innovate, and implement information and communication technology in the pharmacotherapy process.

The currently used clinical decision support system (CDSS) in our daily hospital practice (CDSS 1, a so-called basic CDSS) has several limitations. First, it does not include specific individual patient data, such as laboratory test results, in its algorithms. Second, it cannot deal with different problems simultaneously: it assesses the clinical risk of a drug–drug interaction and that of renal insufficiency separately from each other. Another limitation concerns the complexity of clinical rules. CDSS 1 only screens drug therapies when a medication order (start, repeat, change, or stop) is entered into the system and not when other individual patient data (such as a new laboratory test result) become available. CDSS 1 generates many medication alerts which are clinically irrelevant, thereby reducing actual benefit in daily practice through alert fatigue.^4–6 All in all there is an urgent need for a more advanced pharmacotherapy-related CDSS (CDSS 2) which combines medication orders with laboratory test results and other patient characteristics and which also responds as soon as these types of data become available. Such a system should be more effective (generate more relevant alerts) and more efficient (generate fewer irrelevant alerts) compared to the current CDSS 1.

Geerts et al have shown that in 36.7% of patients with a potential drug–drug interaction in the community pharmacy, a laboratory test is required for the assessment of the clinical relevance of the potential drug–drug interaction.⁷ Other studies have shown that combining medication information with laboratory values in a CDSS results in better monitoring of adverse drug events (ADEs) in patients.^8–12

We developed CDSS 2 based on data from different clinical information systems. In this study we had access to inpatient medication and laboratory data. As in other CDSS, our system generates medication alerts invoked by events (‘triggers’), based on available data (‘input data’), leading to a possible action (‘intervention’). This application of simple or complex ‘if–then’ rules is comparable to the mode of operation of systems used in US hospitals.¹³

We considered it important to test CDSS 2 in direct comparison with CDSS 1 before it would be implemented, since technological improvements do not necessarily translate into improvements in clinical practice. The aim of this study was to compare the positive predictive value (PPV) for clinically relevant medication alerts in CDSS 2 with that in CDSS 1. In addition, differences between specific categories of medication alerts were explored with respect to their clinical relevance.

Background

In the Netherlands, the Royal Dutch Association for the Advancement of Pharmacy publishes a national drug database, the so-called G-standard, which is used by general practitioners, community pharmacies, and hospitals and provides digitalized safety information on all drug products registered in the Netherlands (eg, concerning dosing, drug–drug interactions, drug duplication, drug–disease interaction, and pharmacogenetic factors).^{14 15} It also presents standardized alert texts which contain information about potential adverse drug reactions and recommendations on how to respond to the alert, as well as details about clinical consequences, the underlying mechanism, and consulted references.¹⁶ Both the basic and advanced pharmacotherapy-related CDSS investigated in this study were based on the G-standard.

The generation of specific alerts varies in different hospitals. In the Jeroen Bosch hospital (JBZ), where this study was conducted, the physician initiates medication orders by means of a computerized physician order entry system (CPOE). If necessary, the CPOE will generate an alert for the physician, who can either take action or not. Several times each day, a pharmacist evaluates all generated medication alerts even though the physician may have already seen the alert. The pharmacist decides if further action is needed. In JBZ, the pharmacists are not allowed to cancel or change a medication order or to request for a laboratory value, but they contact the physician and advise an appropriate action.

Method

Setting

A prospective observational study was performed in the hospital pharmacy (ZANOB) of JBZ, which is a teaching top clinical hospital in the Netherlands with 800 beds.

Basic pharmacotherapy-related clinical decision support system (CDSS 1)

In 1981, Centrasys (iSOFT) was implemented in the hospital pharmacy (hereafter called system 1). The CPOE part of this system was introduced in the hospital in 2007. Up to 2011 the hospital pharmacy has used system 1 in daily practice.

System 1 is based on the G-standard and cannot cope with data from other databases, such as laboratory values. To account for the risk that reduced renal function in elderly patients may remain unnoticed, system 1 assigns the attribute ‘renal impairment’ to every patient above 70 years of age, regardless of their actual renal function. It is also possible to provide individual drugs in system 1 with a maximum dose of 0 mg per day. This guarantees that these drugs always generate a dose alert when prescribed, and allows the pharmacist to manually check the dose. An example is the drug methotrexate because medication errors leading to overdosing can easily occur (eg, a high oncolytic dose given instead of a low antirheumatic one).¹⁷

The following clinical decision support (CDS) categories, as stated by Kuperman et al,¹⁸ were operational in system 1: drug–allergy checking, basic dosing guidance, duplicate therapy checking, drug–drug interaction checking, drug–disease interaction checking, and drug–pregnancy checking. It should be noted that the performance of these categories depends on the availability of the input data. Medication control takes place in real-time.

The version of Centrasys used in this study was 4.31, service pack 6 with the G-standard update of July 2010.

Advanced pharmacotherapy-related clinical decision support system (CDSS 2)

Since June 2008, the hospital pharmacy has been cooperating with the software company Pharmaps to develop an advanced CDSS, known as Pharmaps Medicatiebewaking PLUS (hereafter called system 2).

System 2 can handle data from different databases. Data from the clinical chemistry department and the pharmacy were available during the study period. System 2 covers all the CDS categories in system 1 except for the basic dosing guidance. In addition, it also covers the categories advanced guidance on medication-associated laboratory testing and advanced dosing guidance in relation to renal function.¹⁸ System 2 performs all the medication surveillance covered by the G-standard (with the exception of basic dosage control) similarly to system 1, as well as additional medication surveillance (degree of renal impairment, hyperkalemia, lack of potassium, and creatinine levels and other specific recommendations of the HARM-wrestling report²). In addition, the system generates medication alerts concerning thyroid dysfunction for two drugs (amiodarone and lithium).¹⁹ Generation of the medication alerts took place once a day (at 03:00 h). System 2 can generate medication alerts in real-time, but for practical reasons we chose to generate medication alerts once a day during this test phase.

Some aspects of medication surveillance could only be partially realized because: (1) not all patient characteristics (such as diagnosis) were electronically available; and (2) the functionality of reasoning with time-frames was not yet available. For example, when a patient above 80 years of age uses a renin-angiotensin-aldosterone system (RAAS) inhibitor, renal function should be checked every 6 months.² Consequently, the CDSS should check whether renal function has been determined in the preceding 6 months and should generate an alert if this is not the case.

Pharmaps Medicatiebewaking PLUS v 3.1.2.8 was used in this study with the G-standard update of July 2010.

Data collection

During 5 randomly chosen consecutive days in July 2010, all prescribed drugs were evaluated in systems 1 and 2 on the basis of the same medication orders for all patients hospitalized in the JBZ. All medication alerts were assessed independently by two pharmacists. It was not possible to blind the pharmacists regarding the systems for practical reasons. A medication alert was considered ‘relevant’ when the evaluating pharmacist concluded that the physician or nurse should be contacted. An alert was considered ‘irrelevant’ when the evaluating pharmacist concluded that no specific action was necessary. If the two pharmacists did not agree on the same alert, a third pharmacist evaluated the medication alert for the final judgment. Our method of determining the relevance of medication alerts was similar to that of van Doormaal et al.²⁰

The 10 pharmacists involved in the study were well trained to handle basic and advanced medication alerts and consisted of one hospital pharmacist-toxicologist, two hospital pharmacists, two hospital pharmacists-clinical pharmacologists, two hospital pharmacist trainees, two pharmacists working in the hospital on a project basis and one community pharmacist. The third appraiser was a hospital pharmacist-toxicologist or hospital pharmacist trainee.

Medication alert categories

We defined categories of medication alerts in order to explore differences between systems 1 and 2 regarding the clinical relevance of the medication alerts. Classification of the alert categories was based on the content of the signals: drug–drug interaction, drug–age interaction, drug duplication, drug–disease interaction, dosing guidance, and missing laboratory value. We will give an example for each category.

The category drug–(drug) interaction includes medication alerts caused by the combination of two different drugs or the presence of one drug without another, for example, the absence of a laxative in opioid therapy. The category drug–age interaction includes medication alerts caused by drugs in combination with age. For example, when a non-steroidal anti-inflammatory drug (NSAID) is started and the patient is over 70 years of age a proton pump inhibitor is recommended.²¹ In system 1, age >70 years is a proxy for the contraindication renal impairment. Medication alerts in the category drug duplication are generated by the combination of two similar drugs. The category drug–disease interaction covers medication alerts advising against particular drugs in certain conditions. For example, thiazide diuretics are contraindicated in patients with a renal function <30 ml/min. The category dosing guidance includes all types of medication alerts with dosage advice. Medication alerts caused by a missing laboratory value belong to the category missing laboratory value.

The triggers which caused a medication alert in systems 1 and 2 were different. System 2 is capable of applying laboratory values, other patient characteristics, a combination of three or more drugs, and the absence of a drug or laboratory value in its algorithms. We checked whether each medication alert in system 2 was generated by one of the advanced properties. We defined such medication alerts as ‘advanced’. The other system 2 medication alerts we defined as ‘basic medication alerts’. These medication alerts were based only on medication data and the basic properties of system 2, using G-standard data.

Analysis

The proportion of all medication alerts produced which were considered clinically relevant was expressed as the PPV and was calculated separately for each system as follows:

$$\text{Positive predictive value=}\text{number of relevant medication alerts}/\left(\text{number of relevant medication alerts}+\text{number of irrelevant medication alerts}\right)$$

Total positive predictive values (PPVs) were stratified for the different alert categories in both systems. PPVs were also calculated for each individual medication alert.

Statistical tests were performed using SPSS v 16.0. The χ² test was used to calculate if the difference in PPV between both systems was statistically significant. A p value ≤0.05 was considered to be statistically significant.

Results

A total of 619 inpatients were included in the sample. Their mean age was 53.1 years and 45.7% were male. The total number of medication orders was 4023. The number of patients with medication orders which generated a medication alert was 438 for system 1 and 454 for system 2. The mean age of these patients was 67.2 and 67.0 years in systems 1 and 2, respectively (p=0.60), and 51.8% and 53.5% of these patients, respectively, were male (p=0.57).

The 4023 medication orders generated 2607 medication alerts in system 1 and 2256 in system 2. Table 1 shows the PPVs for all the medication alerts in both systems and for each category of medication alert.

Table 1.

Positive predictive values for the clinical relevance of medication alerts

	System 1			System 2
	Nrelevant†	Ntotal‡	PPV* (%)	Nrelevant†	Ntotal‡	PPV* (%)	p Value
Total number of medication alerts	150	2607	5.8	384	2256	17.0	<0.05
Alert category
Drug–(drug) interaction¶	82	828	9.9	172	1163	14.8	<0.05
Drug–age interaction**	23	784	2.9	44	60	73.3	<0.05
Drug duplication††	30	724	4.1	19	344	5.5	0.31
Drug–disease interaction‡‡	0	2	0	34	139	24.5	0.42
Dosing guidance§§	15	269	5.6	73	432	16.9	<0.05
Missing laboratory value¶¶	–	–	–	42	118	35.6	–§

^*PPV (positive predictive value)= $\frac{\text{number of relevant medication alerts}}{\left(\text{number of relevant medication alerts + number of irrelevant medication alerts}\right)}$.

^†N_relevant indicates the number of clinically relevant medication alerts.

^‡N_total indicates the total number of medication alerts.

^§The PPV could not be calculated because this category was not operational in system 1.

^¶Medication alerts caused by the combination of two different drugs or the presence of one drug without another.

^**Medication alerts caused by drugs in combination with age (in system 1, age is a proxy for the contraindication renal impairment).

^††Medication alerts caused by the combination of two similar drugs.

^‡‡Medication alerts with the advice not to give a certain drug in certain conditions.

^§§Medication alerts with dosage advice.

^¶¶Medication alerts caused by a missing laboratory value.

The number of relevant medication alerts increased from 150 in system 1 to 384 in system 2 for the same sample of medication orders. The difference between the PPVs of the medication alerts in system 1 and system 2 was statistically significant (5.8% vs 17.0%; p<0.05). Stratification into the medication alerts categories showed statistically significant differences for the following categories: drug–(drug) interaction (9.9% vs 14.8%; p<0.05), drug–age interaction (2.9% vs 73.3%; p<0.05), and dosing guidance (5.6% vs 16.9%; p<0.05). Remarkably, the two appraising pharmacists disagreed more often regarding the clinical relevance of medication alerts in system 2 than in system 1. The third pharmacist had to review 8.6% of the medication alerts in system 1 versus 32.9% in system 2. This difference was also statistically significant (p<0.05).

PPVs were also calculated for each type of medication alert. Table 2 shows the top five medication alerts (occurring at least 20 times) with the highest PPVs in system 2. The highest PPV of 30.8% in system 1 was found for the medication alert ‘clopidogrel+omeprazole/esomeprazole’. To improve system 2 further, it would be helpful to investigate and then adapt or eliminate frequently occurring medication alerts with the lowest PPV. Table 3 shows the top 10 medication alerts (occurring at least 20 times) with the lowest PPVs in system 2.

Table 2.

The top five medication alerts occurring at least 20 times that were generated by system 2 and had the highest positive predictive values

Alert text	Nrelevant†	Ntotal‡	PPV* (%)
Advice: give a proton pump inhibitor	44	60	73.3
Beware of induction or exacerbation of hyperkalemia	15	21	71.4
Clopidogrel+omeprazole/esomeprazole§	13	20	65.0
This drug requires dosage adjustment in renal function impairment. Determine renal function, since the renal function is unknown in this patient.	19	55	34.5
This drug requires attention when the serum potassium level is high (>5 mmol/l).	14	41	34.1

^*PPV (positive predictive value)= $\frac{\text{number of relevant medication alerts}}{\left(\text{number of relevant medication alerts + number of irrelevant medication alerts}\right)}$.

^†N_relevant indicates the number of clinically relevant medication alerts.

^‡N_total indicates the total number of medication alerts.

^§At the time this study was conducted, this interaction had just been noted as relevant by various (inter)national authorities. However, the effect of the combination of clopidogrel with omeprazole on cardiovascular events remains controversial.^{22 23}

Table 3.

The top ten medication alerts occurring at least 20 times that were generated by system 2 and had the lowest positive predictive values

Alert text	Nrelevant†	Ntotal‡	PPV* (%)
Drug duplication: different strength, same mechanism of action	0	74	0.0
Selective β-blockers+insulin	0	43	0.0
β-Blockers+oral hypoglycemic drugs	0	25	0.0
α-Blocking drugs (for benign prostate hyperplasia)+β-blockers/calcium channel blockers	0	23	0.0
Renal function: bumetanide	0	20	0.0
Renin-angiotensin-aldosterone system (RAAS) inhibitors+diuretics	2	120	1.7
Salicylates antithrombotic+NSAIDs (other than ibuprofen)	1	51	2.0
RAAS inhibitors+potassium(-saving diuretics)	1	41	2.4
NSAIDs+corticosteroids	2	61	3.3
Bisphosphonates+antacids/iron/calcium	2	44	4.5

^*PPV (positive predictive value)= $\frac{\text{number of relevant medication alerts}}{\left(\text{number of relevant medication alerts + number of irrelevant medication alerts}\right)}$.

^†N_relevant indicates the number of clinically relevant medication alerts.

^‡N_total indicates the total number of medication alerts.

NSAID, non-steroidal anti-inflammatory drug.

The impact of including laboratory values and other patient characteristics in the medication surveillance by system 2 is shown in table 4. The PPV was calculated for both the basic medication alerts and the advanced medication alerts in each medication alert category. The advanced medication alerts show a significantly higher PPV than the basic medication alerts (12.2% vs 23.3%; p<0.05). The highest PPV (73.3%) was seen in the category drug–age interaction (advanced) and the lowest PPV (5.5%) in the category drug duplication (basic).

Table 4.

Comparison of basic and advanced medication alerts in system 2

	Basic medication alerts			Advanced medication alerts			p Value
	Nrelevant†	Ntotal‡	PPV* (%)	Nrelevant†	Ntotal‡	PPV* (%)
Total number of medication alerts	155	1274	12.2	229	982	23.3	<0.05
Alert category
Drug–(drug) interaction	136	930	14.6	36	233	15.5	0.75
Drug–age interaction	–	–	–§	44	60	73.3	–
Drug duplication	19	344	5.5	–	–	–§	–
Drug–disease interaction	–	–	–§	34	139	24.5	–
Dosing guidance	–	–	–§	73	432	16.9	–
Missing laboratory value	–	–	–§	42	118	35.6	–

^*PPV (positive predictive value)= $\frac{\text{number of relevant medication alerts}}{\left(\text{number of relevant medication alerts + number of irrelevant medication alerts}\right)}$.

^†N_relevant indicates the number of clinically relevant medication alerts.

^‡N_total indicates the total number of medication alerts.

^§There were no medication alerts in this category.

Discussion

The difference between the PPVs of systems 1 and 2 was substantial (5.8% vs 17.0%; p<0.05). This shows that the clinical relevance of medication alerts was improved by including more data on patient characteristics (especially individual laboratory values) in the medication surveillance. Differences between systems 1 and 2 were mainly found in the categories drug–(drug) interaction, drug–age interaction, and dosing guidance. Regarding system 2, the advanced medication alerts showed a significantly better PPV than the basic medication alerts (12.2% vs 23.3%; p<0.05). Thus, this study confirms the added value of the advanced pharmacotherapy-related CDSS for medication surveillance in a realistic sample of hospitalized patients from one hospital.

The PPVs for drug–(drug) interaction and dosage guidance in system 1 (9.9% and 5.6%, respectively) corresponded well with the results of a previous Dutch study (12% and 6%, respectively) which evaluated a system based on the G-standard.²⁰ In a study by Murphy et al, conducted in the US with online prospective drug-use review systems, 73.8% of the drug–drug interactions were overridden by pharmacists in a community pharmacy, giving a PPV of 26.2%.²⁴

The PPVs found for individual medication alerts in systems 1 and 2 varied from 0% to 73.3%. This wide variation has also been found in some previous studies which investigated one or more specific advanced medication alerts.^25–27 Handler et al conducted a systematic review of medication alerts based on pharmacy and laboratory data to detect ADEs such as elevated serum creatinine and hyperkalemia. They found PPVs ranging from 3% for hypokalemia to 50% for supratherapeutic quinidine levels.²⁶ Raschke et al also found a wide variation in the PPVs for detecting ADEs (24%–97%).²⁷ One explanation for the broad range of PPVs is that laboratory values are often abnormal because of the onset or worsening of clinical conditions unrelated to the use of medication.²⁶ Another explanation is that some algorithms detect rare but immediately life-threatening ADEs, while others detect common situations with a lower potential to result in injury.²⁷

This study has several limitations. First, the local user settings in both systems were different. The G-standard in system 1 had already been modified following years of experience by the hospital pharmacists. This fine-tuning had not yet taken place in system 2. Adjustment of the G-standard generally leads to a higher PPV, which implies that the contrast in the PPV between systems 1 and 2 probably would have been greater if the G-standard in system 2 had been amended. In addition, not all features of the G-standard were fully used in both systems.

Second, the PPV was chosen as the primary outcome measure for comparing systems 1 and 2. The PPV is a useful measure for the correctness and clinical relevance of the generated medication alerts but does not indicate how often relevant alerts were missing, which requires assessment of the sensitivity of the CDSS. However, this parameter requires that all truly positive alerts are recognized as such (the so-called ‘gold standard’), which was not a goal of our study. One way to establish the sensitivity of a system is through the use of a set of test patients. Van der Sijs et al and Saverno et al calculated the sensitivity and specificity for a variety of medication alerts, including drug–drug interactions in several different CPOE and CDSS. Both groups found a wide variation in sensitivity (38%–79% and 23%–100%, respectively) and specificity (11%–84% and 83%–100%, respectively).^{28 29}

The capabilities of CDSS differ between providers,^{13 30} which may have influenced the generalizability of our results. However, CDSS algorithms generally rely on literature-based evidence and practice-based experience, which is accepted worldwide as the best foundation for improving clinical outcomes.³¹ Therefore, the algorithms leading to medication alerts are likely very similar across countries. Consequently, we believe that the results of our study are also applicable to other CDSS.

We have identified several reasons why the PPV of system 2 may have been relatively low. First, not all information needed, such as the patient problem list (diagnosis), was electronically available. For example, the severity of the drug–drug interaction NSAIDs and RAAS inhibitors is more important in patients with heart failure than in those with uncomplicated hypertension. A second limitation of system 2 was its inability to combine different algorithms. For example, system 2 recognized the combination of a RAAS inhibitor and potassium(-saving diuretics) and generated a medication alert with the advice to monitor the serum potassium level. Another algorithm checked if the serum potassium level was available and within the range of 3.5–5.0 mmol/l. The first algorithm should be suppressed if the second algorithm does not generate a medication alert, because the potassium level has already monitored. Third, some of the algorithms in both systems had been incorporated to exclude any risk. For example, the alert ‘drug duplication’ is generated when the physician prescribes two medication orders for the same drug. Such a combination may have been ordered by accident, but may also have been intentional (eg, two drug products to provide an unavailable dose strength). As the pharmacists in our study only contacted the physician in one out of 20 medication alerts, intentional combination appears to have occurred more often than accidental duplication.

Agreement between the two appraising pharmacists occurred less often for system 2. The third pharmacist, who passed the final judgment, was involved significantly more often in system 2 (8.6% vs 32.9%; p<0.05), probably because of the complexity of the advanced medication alerts. Laboratory values were manually searched with system 1 and so the pharmacists were familiar with advanced medication alerts, but in system 2 more and other triggers were involved in generating a medication alert. Judging a medication alert based on two drugs is easier to interpret than medication alerts based on a drug and a variable laboratory value or the addition of another drug, for example, the medication alert that a laxative should be added to an opiate. In case of piritramide 15 mg subcutaneous 4 times a day after surgery, some pharmacists will advise a laxative from day 1, whereas other pharmacists will advise a laxative only after 3 days. Therefore, interpretation of a medication alert requires particular attention when a new type of medication alert is implemented in clinical practice.

The more information is incorporated into algorithms, the more precise the generated medication alerts should be. Therefore, to better develop and optimize the algorithms, risk modifiers should be identified from evidence-based and clinical practice-based medicine. For instance, multivitamin supplements providing 25 μg of vitamin K1 have long been considered harmless for patients on warfarin. However, this view was seriously challenged by three cases of stabilized warfarin users in whom anticoagulant treatment was compromised by the initiation or cessation of a low-dosed multivitamin supplement.³² Subsequent research showed that 25 μg of vitamin K1 daily produces subtherapeutic INRs in users with a low vitamin K1 level.³³ In other words, vitamin K1 status is an important modifier of the risk that stabilized warfarin users are affected by dietary supplements providing a small dose of vitamin K1. Therefore, identified potentially interesting risk modifiers should be made electronically available in the database and also built into the algorithms.

The knowledge database of a CDSS should be continuously maintained by evaluating what effects medication alerts have in daily practice. The Plan-Do-Check-Act (PDCA) cycle, also known as the Deming cycle, might be helpful.^{25 34} Wessels-Basten et al improved the PPV of the lithium logarithm from 63% to 83% by using the PDCA cycle.²⁵ The next step to improve the PPV of system 2 is to determine why the pharmacists respond to some alerts and not to others. This could be followed by two strategies. First, if the pharmacists do not respond correctly to medication alerts, education is needed. Second, the algorithm should be fine-tuned until the PPV has improved. It is expected that the PPV of system 2 will increase when the PDCA cycle is completed.

Conclusion

The main conclusion of this study is that system 2 had a significantly higher PPV than system 1 (5.8% vs 17.0%; p<0.05). The study shows that system 2 was more effective and more efficient than system 1 in carrying out medication surveillance. However, the number of irrelevant medication alerts remained relatively high.

To improve the PPV of system 2, the algorithms should be further optimized as follows: (1) by identifying risk modifiers from the existing scientific literature; (2) by making these additional risk modifiers electronically available; and (3) by cyclic testing of the effects of medication alerts in daily clinical practice (PDCA cycle).

Our study illustrates and corroborates the need to test technical improvements in information technology in circumstances representative of daily clinical practice. This type of research will contribute to further optimization of CDSS. It should also be kept in mind that maintenance of a knowledge database is a continuous process.