TABLE 2.
Validation methods and outcomes of the included studies
| Authors | Group A/B/C and description methods | Outcomes, relevance (R), applicability (A) | Compliance framework 14 step 2, 3, 4 and outcomes | ||||||
|---|---|---|---|---|---|---|---|---|---|
| R | A | 2 | 3 | 4 | Outcomes | ||||
| Arvisais et al. (2015) 26 | A | Two junior pharmacists (independently) analysed alerts for clinical relevance |
Clinical relevance = 75% (149 with ≥1 clinically relevant alert/200 patient days) |
+ | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Azaz‐Livshits et al. (1998) 46 | B | Signals evaluated by expert team |
37% (78 signals related to ADR/212 signals) 29% (25 admissions ADR & signal/86 admissions with alert) |
++ | ++ | X | _ | _ |
PPV (clinical relevance) = no NPV = no Extra = yes |
| C | Clinical pharmacologist reviewed charts for ADRs, evaluated by expert team |
Sn = 66% (25 CDSS/38 admissions by expert team) Sp = 49% (56/115 admissions) |
+++ | + | |||||
| Buckley et al. (2018) 47 | B | Pharmacist reviewed alerts and patient charts to determine causality for DRHCs/ADEs using validated tools |
PPV (DRHCs) = 29% (249/870 alerts) PPV (ADEs) = 5% (47/870) |
++ | ++ | _ | _ | X |
PPV (clinical relevance) = no NPV = no Extra = yes |
| Cossette et al. (2019) 29 | A | Clinical relevance assessment by two pharmacists based on clinical experience |
Clinical relevance: 41% (34/83 alerts) 42% (27/65 patients with ≥1 alert) |
+ | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Dalton et al. (2020) 27 | A | Independent analysis by pharmacist and physician (6‐point scale) |
Clinical relevance = 74% (681 relevant/925 alerts) |
+ | +++ | X | _ | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| DiPoto et al. (2015) 43 | A | Pharmacists assessed alerts (clinically relevant: pharmacists proposed change) | Clinical relevance: 40% (ICU, 90/226 alerts), 45% (general ward, 235/525 alerts) | + | +++ | _ | _ | X |
PPV (clinical relevance) = yes NPV = no Extra = yes |
| B | Pharmacist assessed causality trigger and adverse events (subgroup analysis of 161 triggers of 19 rules) | PPV (DRHCs) = 71% (115/161 triggers) | ++ | ++ | |||||
| Dormann et al. (2000) 48 | B | Team of pharmacologist, clinician and pharmacists assessed alerts and patient chart for ADR (Naranjo score) | PPV (ADRs) = 13% (63/501 alerts) | ++ | ++ | _ | X | _ |
PPV (clinical relevance) = no NPV = no Extra = yes |
| C | Compare ADR detected by CDSS with ADRs from spontaneous reporting |
Relative Sn = 74% (34 ADRs CDSS/46 [all] ADRs) Relative Sp = 75% |
+ | + | |||||
| Eppenga et al. (2012) 44 | A | Two pharmacists independently assessed alerts of two CDSSs for clinical relevance (of pharmacist would take action) |
PPV (clinical relevance) CDSS 1 = 6% (150/2607 alert), PPV (clinical relevance) CDSS 2 = 17% (384/2256 alerts) |
+ | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Ferrández et al. (2017) 45 | A | Alerts reviewed for clinical relevance (action needed) by pharmacists | Clinical relevance = 20% (2808/13 833 alerts) | + | +++ | _ | _ | X |
PPV (clinical relevance) = yes NPV = no Extra = yes |
| C | Compare DRPs detected by CDSS with pharmacist review | 79% (2808 DRPs by CDSS/3552 [all] DRPs) | +++ | + | |||||
| Fritz et al. (2012) 34 | A | Alerts reviewed for clinical relevance (of pharmacists would take action) by pharmacists AND sensitivity to detect all 33 relevant alerts (identified by pharmacist while reviewing alerts) |
PPV (clinical relevance) = 6%, 8%, 8% (3/53, 29/364, 25/328 alerts), respectively Sn = 9%, 88%, 76% (3/33, 29/33,25/33 relevant alerts), respectively |
++ | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = yes |
| Garcia‐Caballero et al. (2018) 28 | A | A physician and psychiatrist reviewed alerts | Relevance = 12% (140/1155 alerts) | + | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Hammar et al. (2015) 35 | A | A physician reviewed alerts for clinical relevance as part of medication review | Clinical relevance = 68% (502/740 alerts) | + | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Hedna et al. (2019) 24 | B | For each alert, it was determined whether it was related to symptoms | PPV = 0.20–0.25 (low risk: 150/776, intermediate risk: 93/460, high risk: 53/208 alerts) | ++ | ++ | X | _ | _ |
PPV (clinical relevance) = no NPV = yes Extra = yes |
| C | Pharmacists extracted symptoms associated with medications, checked by second reviewer |
Sn = 0.12–0.37 (high–low, patients' symptom & alert/patients' symptom from review) Sp = 0.78–0.95 (low–high) NPV = 0.89–0.90 (high–low) |
++ | + | |||||
| Hwang et al. (2008) 49 | B | Pharmacist reviewed alerts and charts for association alert with ADE | PPV (ADEs) = 21% (148/718) | ++ | ++ | X | − | − |
PPV (clinical relevance) = no NPV = no Extra = yes |
| C | Pharmacist (checked by five other pharmacists) reviewed charts for patients without alert for ADEs | Sn = 79% (148/187 ADEs) | +++ | + | |||||
| Ibáñez‐Garcia et al. (2019) 36 | A | Pharmacist reviewed alerts, and advised physician | 51% (554 with advice/1086 alerts) | + | +++ | − | X | − |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Jha et al. (1998) 50 | B | Reviewer analysed alerts, charts for ADE association (checked by physician) | PPV (ADEs) = 17% (450/2620 alerts) | ++ | ++ | − | X | − |
PPV (clinical relevance) = no NPV = no Extra = yes |
| C | Reviewers (blinded to CDSS) conducted ADE detection study (three methods) | ADEs detected by CDSS = 45% (275/675 [all] ADEs) | +++ | + | |||||
| Jha et al. (2008) 37 | A | Reviewer analysed 52% alerts, and contacted physician if necessary | Clinical relevance = 11% (30 with contact/266 alerts) | + | +++ | − | X | − |
PPV (clinical relevance) = yes NPV = no Extra = yes |
| B | Chart review to identify (potential) ADEs in a sample of patients (checked by physician) |
PPV (ADEs) = 23% PPV (pADEs) = 15% |
++ | ++ | |||||
| Levy et al. (1999) 51 | B | Analyses of signals | 18% (signals related to ADR (52)/all signals [295]) | ++ | ++ | − | X | − |
PPV (clinical relevance) = no NPV = no Extra = yes |
| C | Team reviewed charts for ADRs |
Sn = 62% (40 ADR admissions by tool/65 [all] ADR admissions) Sp = 42% (79/135 admission without ADR) |
+++ | + | |||||
| Miguel et al. (2013) 52 | B | ADRs detected by CDSS reviewed for true ADRs | PPV = 80% (65 true ADRs/81 all suggested ADRs) | ++ | ++ | X | _ | _ |
PPV (clinical relevance) = no NPV = no Extra = yes |
| C | Chart review and assessment by CDSS in population | 83% (10 ADR CDSS/12 ADRs in chart review) | + | ++ | |||||
| Peterson et al. (2014) 25 | A | Pharmacist reviewed patients on dashboard and advised physician |
12% (22 with intervention/179 patients) 6% (31 with interventions/485 alerts [PIMs]) |
+ | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Quintens et al. (2019) 30 | A | Pharmacist checked alerts for appropriateness (clinical relevance = electronic note or phone call to physician) | Clinical relevance = 8% (3205 with action/39 481 alerts) | + | +++ | X | _ | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Raschke et al. (1998) 38 | A | Pharmacist/radiology technicians evaluated alerts and advised physician | Relevance = 71% (794 with advice/1116 alerts) | + | +++ | − | X | − |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Rommers et al. (2011) 31 | A | Hospital pharmacists reviewed true positive alerts for clinical relevance (= started intervention) | Clinical relevance = 19% (14 with intervention/72 true positive alert) | + | +++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Rommers et al. (2013) 39 | A | Pharmacists reviewed alerts, contacted and advised physician/nurse |
PPV (clinical relevance) = 8% (204 with advice/2650 alerts) |
+ | +++ | − | X | − |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Roten et al. (2010) 33 | C | Pharmacists conducted medication review (blinded to CDSS) to identify DRPs |
324 patients (65%) with alert Sn = 85% (235 patients by CDSS/276 [all] patients with DRP) Sp = 60% (136/225 [all] patients without DRP) |
+++ | + | − | X | − |
PPV (clinical relevance) = no NPV = no Extra = yes |
| Schiff et al. (2017) 23 | A | Chart of patients with an alert were reviewed for accuracy and clinical validity |
126 alerts: Accuracy = 93% (based on data) Clinical validity (clinical relevance) = 75% |
+ | ++ | X | _ | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Silverman et al. (2004) 32 | A | Pharmacists reviewed alerts, and advised physician (3× with different ADE rules) |
Rule effectiveness (clinical relevance) = 5%, 6%, 13% (169/3117, 452/7390, 792/6136 alerts), respectively |
+ | +++ | _ | _ | X |
PPV (clinical relevance) = yes NPV = no Extra = no |
| Segal et al. (2019) 22 | A | Biweekly interviews to manually review alerts |
315 alerts: Accuracy = 89% (no data issues) Clinical validity = 85% (no justification for medication) Clinical usefulness (clinical relevance) = 80% |
+ | ++ | _ | X | _ |
PPV (clinical relevance) = yes NPV = no Extra = no |
| de Wit et al. (2015) 40 | A | Pharmacists reviewed alerts for clinical relevance (= advised physician) | Efficiency (clinical relevance) = 4% (147/4065 alerts) | + | +++ | _ | _ | X |
PPV (clinical relevance) = yes NPV = no Extra = no |
| de Wit et al. (2016) 41 | A | Pharmacist and geriatrician independently checked DRPs by CDSS |
Clinical relevance = 12% (70/574 alerts) Sn = 72.9% (51 relevant alerts also classified as relevant by CDSS/70 relevant alerts) Sp = 98.6% (497 irrelevant alerts also classified as irrelevant by CDSS/504 irrelevant alerts) |
+ | ++ | X | _ | _ |
PPV (clinical relevance) = yes NPV = no Extra = yes |
| C | Geronto‐pharmacology meeting discussed DRPs from a medication review (blinded to CDSS) | 20% (44 DRPs CDSS/223 DRPs medication review) 28% (70 DRPs CDSS/249 [all] DRPs) | ++ | + | |||||
ADE, adverse drug event; ADR, adverse drug reaction; CDSS, clinical decision support system; DRHC, drug‐related hazardous conditions; DRP, drug‐related problems; Group A, studying clinical relevance of CDSS's output; Group B, CDSS's output and actual occurrence of DRPs (patients with alert); Group C, CDSS's output and chart/medication review in whole population; PIM, potentially inappropriate medication; PPV, positive predictive value; Sn, sensitivity; Sp, specificity.