Systematic review of predictive risk models for adverse drug events in hospitalized patients

Abstract

Aim

An emerging approach to reducing hospital adverse drug events is the use of predictive risk scores. The aim of this systematic review was to critically appraise models developed for predicting adverse drug event risk in inpatients.

Methods

Embase, PubMed, CINAHL and Scopus databases were used to identify studies of predictive risk models for hospitalized adult inpatients. Studies had to have used multivariable logistic regression for model development, resulting in a score or rule with two or more variables, to predict the likelihood of inpatient adverse drug events. The Checklist for the critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) was used to critically appraise eligible studies.

Results

Eleven studies met the inclusion criteria and were included in the review. Ten described the development of a new model, whilst one study revalidated and updated an existing score. Studies used different definitions for outcome but were synonymous with or closely related to adverse drug events. Four studies undertook external validation, five internally validated and two studies did not validate their model. No studies evaluated impact of risk scores on patient outcomes.

Conclusion

Adverse drug event risk prediction is a complex endeavour but could help to improve patient safety and hospital resource management. Studies in this review had some limitations in their methods for model development, reporting and validation. Two studies, the BADRI and Trivalle's risk scores, used better model development and validation methods and reported reasonable performance, and so could be considered for further research.

Introduction

Adverse drug events (ADEs) are a significant cause of morbidity and mortality in hospitalized patients 1, 2, 3, 4, 5. Several international studies have quantified ADE rates, ranging from 2.4% to 30% of inpatients 1, 6, 7, 8, 9, 10, 11. In the Australian context, the Quality in Healthcare study identified that 16.6% of admissions experienced an adverse event, with half deemed as highly preventable. Approximately 10% of these events were medication related 12. The high costs of ADEs are widely acknowledged 2, 3, 13, 14, 15. Preventable adverse events, including ADEs, have been reported to cost the British National Health Service in excess of £2.5 billion each year 14. In the USA, an estimated $3.5 billion dollars is spent annually on medical costs associated with ADEs 15.

Traditional efforts to reduce inpatient ADEs have focused primarily on systems measures 16, 17. Implementation of electronic medication management systems, including electronic prescribing and automated dispensing, have been shown to significantly reduce medication errors 18, 19, 20 and a number of Australian hospitals are adopting these systems in accordance with the government's E‐Health strategy 18, 19, 20, 21. However, such systems are expensive and, if not well implemented and maintained, can introduce new errors with potential for ADEs 22. Amongst other key strategies known to prevent ADEs and to reduce patient harm are clinical pharmacist services such as medication reconciliation, clinical appraisal of medication regimens and effective discharge coordination to primary care 16, 23. However, the high patient throughput and diminishing resources in many hospitals often limit the implementation of such services 16, 24. Therefore, methods to prioritize patients at‐risk of ADEs would be beneficial.

An approach to focus clinicians on patients at high risk is to use an ADE algorithm or predictive risk model. Given that up to 80% of ADEs are thought to be predictable 25 and 50% are estimated to be preventable 26, 27, a validated model incorporating significant ADE‐related risk factors should help identify at‐risk inpatients 7, 28, 29, 30, 31, 32, 33. Risk scores, which combine influential patient and medical variables to help clinicians rank patients, are becoming increasingly popular to assist in healthcare decisions 34, 35, 36. Risk prediction for ADEs can alert the multidisciplinary team to high risk inpatients and facilitate pharmacovigilance and targeted interventions. A recent Australian study evaluating adverse drug reaction (ADR)‐related hospital admissions in the elderly (PADR‐EC Score) developed a predictive risk score, to identify high risk patients for community‐based interventions 37. Similarly, the use of risk scores for hospital inpatients can enable systematic risk stratification, to help with the timely delivery of interventions, as advocated by medication safety bodies 38. In addition to benefits to patients, a predictive risk score can facilitate more efficient service delivery 34.

Several predictive risk scores are currently in routine clinical use in specific disciplines. These include the Patient at Re‐admission Risk (PARR) tool 39, cardiovascular disease management risk scores, such as the EuroSCORE 40 and the Framingham Risk Score 41 and the Ottawa Ankle Rule in orthopaedics 42. However, to the best of our knowledge, no risk scores are in routine clinical use to predict the risk of inpatient ADEs. A recent systematic review by Stevenson et al., identified and evaluated four ADR predictive risk models for hospitalized older patients, defined as 65 years or older 43. They found that risk scores had deficiencies in model development and further research with a focus on external validity of the risk scores is needed prior to impact studies and implementation in routine clinical use. In our review, we expand on Stevenson's research by evaluating studies of patients aged 15 years or older, and include a broad range of hospital settings and definitions of ADEs, to ensure all available models were appraised.

Aim

The aim of this systematic review was to critically appraise models developed for predicting ADE risk in adult inpatients. Specifically, the objectives of the review were to: identify relevant ADE predictive risk models; evaluate the development and performance of these models; assess their validation methodology; and identify any impact evaluations, to help guide suitability for clinical use.

Method

This review was conducted and reported according to the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) guidelines and follows the methods described in our protocol which was registered with PROSPERO, International Prospective Register of Systematic Reviews and can be accessed at: http://www.crd.york.ac.uk/PROSPERO/displayrecord.asp?ID=CRD42016045619.

Definitions

The following are frequently used terminology and their respective definitions relating to ADEs. One of the earliest and most frequently used definitions of ADEs is by the World Health Organization (WHO). They defined ADEs as follows:

“Any untoward occurrence that may present during treatment with a pharmaceutical product but that does not necessarily have a causal relation to the treatment.” The WHO definition of ADEs includes harm from medication errors as well as ADRs. A major subset of ADEs are ADRs, which were defined by the WHO as: "a response to a drug which is noxious and unintended and which occurs at doses normally used in man for prophylaxis, diagnosis, or therapy of disease, or the modification of physiological function" 44.

ADRs have also been defined as “harm directly caused by use of a drug at normal doses” 45. ADRs can occur despite therapeutically appropriate prescribing and administration but are generally not inclusive of harm from medication errors. ADRs can be divided in to type A reactions, which are predictable and dose dependent or the less common type B reactions which are unpredictable 25. Where an ADR has occurred, a direct causal relationship with a medication can be established.

Drug‐related problems (DRPs) have been defined as “an event or circumstance involving drug therapy that actually or potentially interferes with desired health outcomes" 46. Medication errors were defined as “any error in the process of ordering or delivering a mediation regardless of whether an injury occurred or the potential for injury was present” 47.

Inclusion criteria

All primary studies of predictive risk models developed for use in hospitalized adult inpatients were included, irrespective of hospital department or specialty. Studies were included if they used multivariable logistic regression for model development and resulted in a score or rule with two or more risk factors, which in combination could be used to predict the likelihood of inpatient ADEs, to help guide clinical decisions. For studies of models without validation we required that at a minimum the internal performance of the final model was reported.

As the concept of ADEs can include any events that place patients at high risk (potential ADEs) or expose them to medication harm, this review included all studies where the primary outcome measure was synonymous with, or closely related to ADE risk. This included ADRs, DRPs and medication errors. By keeping our inclusion criteria broad, we aimed to ensure that all relevant risk models could be evaluated.

Exclusion criteria

Studies of risk models for paediatric patients and patients in the ambulatory setting were excluded. Studies that did not include development of a predictive risk model using multivariable logistic regression were excluded. Aetiological studies that used multivariable modelling to identify causal risk factors for ADEs, ADRs, DRPs or medication errors were excluded. Studies of risk prediction models for specific medications (e.g. digoxin or lithium) or a medication class (e.g. ACE‐inhibitors) were also excluded. Abstracts of studies describing the development or validation of predictive risk models were excluded if the full text article could not be located, as they provide too little information to allow meaningful evaluation and quality appraisal.

Information sources and search strategy

The literature search was undertaken using Medical Subject Headings and words related to hospital adverse drug events and risk prediction scores. Search terms were used to search Embase, PubMed, CINAHL and Scopus. Google Scholar was also searched to identify any literature not found through the databases using the following terms: Predictive risk score adverse drug events, predictive risk score adverse drug reactions, predictive risk model(s) adverse drug events and predictive risk model(s) adverse drug reactions. Hand search of citations and bibliography lists of key studies was also undertaken. Please see Appendix 1 for our Embase search strategy. Full text original research articles available in English were included to 31 December 2016.

Study selection

The first author (N.F.) reviewed all studies retrieved for eligibility. To ensure the search strategy was accurate and reproducible, a co‐author (N.C.) independently undertook the Embase search and reviewed the search output. All three authors read a short list of potentially eligible studies, extracted data independently and discussed findings.

Data collection and quality and risk of bias assessment

Data from each eligible study were extracted with a standard checklist and appraised using the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) 48. The checklist was used to form the review question and assess the adherence of studies to recommended best practice methods for predictive risk model development and validation.

Because of the nature of the studies, due to variations in outcome measures and study results, a meta‐analysis was not conducted. We undertook a qualitative assessment, with focus on study characteristics, development, and performance of the models. For models that were validated, validation methodology and results were also appraised. The final models/risk scores and their potential for application in the clinical setting were evaluated.

Results

Study selection

A total of 12 795 studies were identified through the database search. An additional 70 studies were identified through hand search of citations and Google Scholar, providing a total of 12 865 publications. After removal of duplicates, 9904 remained, with 458 of these related to adult inpatient ADEs. The abstracts of these studies were reviewed and 436 removed. The remaining 22 articles described a predictive risk tool for ADEs. There were eleven studies excluded as shown in Figure 1, which resulted in 11 studies in the final review.

Figure 1.

Article selection flow diagram (modelled according to PRISMA guidelines)

Study characteristics

The characteristics of the 11 studies are presented in Table 1. Ten studies developed new risk models and one study re‐validated and updated an existing model. The studies by Tangiisuran et al. (BADRI risk score) 49, Onder et al. (GerontoNet ADR risk score) 50, Trivalle et al. 51, McElnay et al. 52, and Passarelli and Filho 53 developed models for use in older inpatients, in rehabilitation and/or acute hospital settings. The studies by Sakuma et al. 54, Urbina et al. 55, Zopf et al. 56 and Nguyen et al. 57 developed models for use in acute medical and surgical adult inpatients and Sharif‐Askari et al. 58 developed a model specifically for renal inpatients. O'Connor et al. externally validated and updated the GerontoNet risk score 59.

Table 1.

Model development

First author (year) [ref[]	Data	Participants		Outcome				Predictor variables			Modelling
	Design and dates of study	1. Sample size (n) 2. Age 3. Clear inclusion and exclusions?	1. Nonbiased selection? 2. Clinical setting 3. Followed to discharge?	Was outcome clearly defined?	Was appropriate method used to measure outcome?	Was causality assessed?	Was investigator blinded to predictor variables?	Was method and timing of predictor variable measurement appropriate?	1. Were all appropriate variables included? 2. How were variables handled?	Was sample size adequate? (based on EPV ≥ 10)	Appropriate method for selection of predictors for testing in MV modelling?	Appropriate method for selection of predictors during MV modelling?
Nguyen (2014) 57	Prospective Cohort 1st to 31st April 2014	1. n = 1408 patients 2. >17 years (median age 68 years) 3. No exclusions not clear	1. Yes Consecutive admissions (nonselective) 2. surgical and medical wards of a French hospital 3. Yes	Yes UI disc. and prescribing errors Clinically significant MEs = 25.9%	Yes MEs reviewed by independent panel using NCC MERP grading categories	N/A	Yes extracted using electronic medical records	Partly Predictor variables measured via electronic reporting which has higher chance of missing data	1. Partly ‐ lab tests, comorbidities and social risk factorsa NR 2. Correctly analysed continuous variables as continuous	Yes EPV = 20 475 MEs (365 patients with MEs) With 23 variables (including all ATC levels)	Partly UVA variables with P < 0.5 taken forward to MVA	Yes – MVLR with backwards elimination (P = 0.3) and forwards selection
Tangiisuran (2014) 49	Prospective cohort Jan to Mar 2007 and 2008, Sept to Feb 2009	1. n = 690 patients 2. > 65 years (mean age 84.3 years) 3. Yes	1. Yes Consecutive admissions (nonselective) 2. two geriatric and stroke wards of a UK hospital 3. Yes	Yes Edwards and Aronson definition of ADR ADR = 12.5%	Yes AM used and potential ADEs reviewed by panel with independent experts	Yes Hallas algorithm and Likert scale Possible, probable and definite ADRs included	NR Unlikely as potential ADRs measured by a PI	Partly AM used but vital signs and labs only measured once within 48 h of admission	1. Yes 2. NoM and LOS dichotomized. Variables in ≤5% of study population excluded Multi‐collinearity assessed	No EPV < 10 95 ADRs (86 patients with ADRs) with 40+ variables	Partly UVA Variables with P < 0.05 taken forward to MVA. Investigator choice with variables with P > 0.05 and < 0.25 also included	Yes – MVLR with backwards elimination (P = 0.1) and forwards selection
Sharif‐Askari (2014) 58	Prospective cohort Jan to Dec 2012	1. n = 512 2. ≥18 years (mean age 60 years) 3. Yes	1. Yes Consecutive admissions (nonselective) 2. renal ward of hospital in the UAE 3. Yes	Yes Edwards and Aronson definition of ADR ADRs = 12.1%	Partly Potential ADRs reviewed by two study authors. Frequency and method of data collection NR	Yes Naranjo algorithm Probable and definite ADRs included	NR	Partly AM used variables only measured once at admission by PI.	1. Mostly, hospital utilizationb and social risk factors NR 2. Age, labs, NoM, dichotomized. Variables >10% MD exc. Other MD imputed using MIs Multi‐collinearity assessed	No EPV = 2.7 62 patients with at least one ADR with 23 variables	Partly Investigator choice with variables from literature and preselected using UVA, variables with P < 0.05 taken forward to MVA	Yes – MVLR forwards selection, variables with P > 0.10 exc. model with lowest AIC and best C‐statistic selected as final model Adjusted for age, sex, eGFR
Urbina (2014) 55	Prospective cohort Jan to August 2009	1. n = 7202 patients 2. >18 years (mean age 58.9 years) 3. Yes	1. Yes Consecutive admissions (nonselective) 2. acute medical and surgical wards of a Spanish hospital 3. Yes	Yes PCNE definition of DRPs DRPs = 29.9%	Yes Daily review of DRPs from CPOE and Clinically significant (CS) DRPs included (Independent assessment of CS NR).	N/A Type of DRP classified according to PCNE (but no outcomes Assessed)	NR	Partly Exact timing of measurement NR – variables measured prospectively	1. Partly‐full list of lab tests and social risk factorsa NR 2. Age and NoM dichotomized	Yes EPV = 89 2425 DRPs with 27 variables	Partly UVA variables with P < 0.1 taken forward to MVA	Yes – MVLR stepwise selection, P value cut‐off NR Final formula reported
Sakuma (2012) 54	Prospective cohort Jan to June 2004	1. n = 1729 2. ≥15 years 3. Yes	1. Yes Consecutive admission (nonselective) 2. 15 randomly selected acute medical, surgical wards and ICU of three Japanese Hospitals 3. Yes	Yes ADEs (including ADRs and harm from errors) defined by Morimoto et al. ADEs = 21%	Yes AM used and potential ADEs reviewed by panel with independent experts	Partly Inter‐rater reliability for classification of incidents, severity and preventability assessed using Kappa Statistics.	NR	Partly AM used, Variables collected daily by nursing staff or students but unclear if variables updated at time of ADE detection	1. Partly, hospital utilisation, social risk factors NR– unclear which lab tests (other than renal) included 2. Age and Creatinine levels dichotomized	No EPV = 6.6 376 patients with at least one ADE and 57 variables	Partly UVA variables with P ≤ 0.2 taken forward to MVA	Yes – MVLR Stepwise selection, variables with P ≤ 0.01 were retained in model Final formula reported
O'Connor (2012) 59	Prospective cohort (re‐validation and updating of GerontoNet score) July to Oct 2010	1. n = 513 2. ≥65 years (Median age 77 years) 3. Yes	1. Consecutive admissions (nonselective) 2. acute medical and surgical wards of hospital in Northern Ireland 3. Yes	Yes WHO definition of ADR ADRs = 26%	Yes AM used and potential ADEs measured by PI on days 5 and 10, reviewed by expert panel	Yes WHO UMC Criteria Probable and definite ADRs included	NR Unlikely as ADRs identified by PI	Yes AM used Variables collected at admission, days 5 and 10 of study by PI	1. Mostly, Hospital utilization and social risk factors NR 2. Age, renal function and help with ADLs categorized. NoM and PIMs entered as continuous using block entry	N/A See table 4	Partly UVA, variables with P < 0.05 taken forward to MVA	Unclear MVLR but selection criteria NR
Trivalle (2011) 51	Prospective cohort Dates not specified	1. n = 526 2. ≥65 years (mean age 83.6 years) 3. Unsure Clear inclusions but 54 patients excluded and reason not provided	1. No – patients excluded without clear reasons 2. 16 French rehab. Hospitals for a four‐week period 3. No – ADEs recorded for 4 weeks of study	Yes modified WHO definition of ADR ADEs = 39%	Yes AM with a 32‐item checklist used by nursing staff and weekly review by a PI Potential ADEs reviewed by expert panel	Yes Doucet and colleagues (classifies ADRs as probable or not) Probable ADRs included	NR	Yes AM used variables collected at admission and updated at time of ADE detection by PI	1. Partly, only medications and medical comorbidities included 2.NoM categorized Variables present < 5% of study population exc. Multicollinearity assessed	Not Clear Unable to calculate EPV as full list of variables not provided	Partly UVA variables with P < 0.05 taken forward to MVA	Yes – MVLR stepwise selection, variables with P ≤ 0.05 retained in final model
Onder (2010) 50	Retrospective cohort using GIFA database May to June and Sept to Oct 1993, 1995 and 1997	1. n = 5936 patients 2. ≥65 years (mean age 83.6 years) 3. Yes	1. No ‐ patients with MD in GIFA database excluded 2. Italian hospitals 3. Yes	Yes WHO definition of ADR ADRs = 6.5%	Partly AM used but potential ADEs only assessed by a study physician	Yes Naranjo algorithm Probable and definite ADRs included	NR	Partly AM used variables collected by PI, unclear if variables (other than medicines) updated at time of ADE detection	1. Mostly, Hospital utilization NR 2. Multiple variables categorized, including age and NoM (3 groups with overlap between 2), body mass index, number of comorbidities, albumin	Yes EPV = 22.5 383 ADRs with 17 variables	Partly UVA variables with P ≤ 0.1 taken forward to MVA	Yes – MVLR stepwise selection, Variables with P ≤ 0.10 retained in final model
Zopf (2008) 56	Prospective Cohort Dates not specified	1. n = 907 patients 2. ≥ 16 years (mean age 60) 3. No	1. Yes Consecutive admissions (nonselective) 2. Medical wards of 2 German hospitals 3. Yes	Yes WHO definition of ADR ADRs = 38%	Yes AM used and potential ADEs reviewed by panel with independent experts	Yes Naranjo algorithm Possible, probable and definite ADRs included	NR	Yes AM used, Variables collected by PI, unclear if variables updated at time of ADE detection	1. Partly, hospital utilization, social risk factors and comorbidity score NR 2. NoM, alcohol use and smoking categorized	Yes EPV = 14.8 592 ADRs (345 patients with at least one ADR) with 40 variables	Partly UVA, P value cut‐off NR	Yes – MVLR selection criteria NR, models adjusted for age, heart rate, hospital stay, department and labs.
Passarelli (2007) 53	Prospective cohort Sept 2002 to May 2004	1. n = 186 patients 2. ≥60 years 3. Yes	1. Consecutive Admissions (nonselective) 2. Medical ward of a Brazilian Hospital 3. Yes	Yes Edwards and Aronson definition of ADR ADRs = 61.8%	Not clear AM used but unclear who measured and reviewed outcomes	Yes Naranjo algorithm (but NR in results)	NR	Yes AM used but unclear who collected data and if variables updated at time of ADE detection	1. Partly– labs, vital signs, hospital utilization NR 2. Age categorized into three groups	Not clear EPV likely <10 full list of variables NR, Total ADRs = 199 (115 patients with at least one ADR	Partly UVA (P value cut‐off NR)	Yes – MVLR stepwise selection, backwards elimination criteria NR
McElnay (1997) 52	Prospective cohort Dates not specified	1. n = 929 patients 2. ≥65 years 3. No – exclusions not clear	1. Yes Consecutive admissions (nonselective) 2. Medical, cardiac, surgical and geriatric wards of a Hospital in Northern Ireland 3. Yes	Not clear WHO ADR definition modified to include ineffective treatment, nonadherence and overdoses ADEs = 16%	NR	Yes Modified Naranjo algorithm Probable or definite ADEs included	NR	Partly AM used and approx. Half patient interviewed 72 hours of admission. Unclear who collected data and if variables updated at time of ADE detection	1. Yes 2. All continuous variables categorized variables present in <5% of study population exc. Multicollinearity assessed	Not clear EPV likely <10 Full list of variables NR, Total ADEs = 149	Partly UVA, variables with P < 0.25 taken forward to multivariable stage. All medications included irrespective of P values	Yes ‐ MVLR backwards elimination (using maximum likelihood method) – Initial criteria P = 0.15 and final criteria P = 0.05

ADLs, activities of daily living; ADRs, adverse drug reactions; ADEs, adverse drug events; CPOE, computerized physician order entry; DRPs, drug‐related problems; eGFR, estimated glomerular filtration rate; EPV, events‐per‐variable; exc., excluded; MD, missing data; MEs, medication errors; MI, multiple imputations; MVA, multivariable analysis; MVLR, multivariable logistic regression; NoM, number of medications; NR, not reported; PI, primary investigator; PIMs, potentially inappropriate medications; GIFA, Gruppo Italiano di Farmacoepidemiologia nell'Anziano; UAE, United Arab Emirates; UI disc, unintentional discrepancies; UVA, univariable analysis; WHO, World Health Organization; AM, appropriate methods: daily review of clinical notes, medication charts and other patient computer records (e.g. laboratory test results) by a trained researcher, and review of all incident reports

^aIncludes living situation, alcohol intake, cigarette smoking

^bIncludes length of stay, hospital re‐admission

The studies were conducted between 1997 and 2014, and two studies did not specify exact dates 52, 56. The study cohorts varied with respect to age (mean age: 58–85 years), sample size (186–7209 patients), country where the study was undertaken (eight studies originated from Europe, one study in three Japanese hospitals, and another in the United Arab Emirates, the hospital setting (rehabilitation wards vs. acute wards) and outcome measures. The description of study participants was reported in sufficient detail in all studies and seven of the 11 studies had clear inclusion and exclusion criteria. Zopf et al. 56 did not stipulate inclusion or exclusion criteria, whilst Nguyen et al. 57, McElnay et al. 52 and Trivalle et al. 51 only reported their inclusion criteria. All studies followed patients from admission to discharge, except for the study by Trivalle et al. 51, which collected data over a prespecified 4‐week period.

Model development

Study design

A prospective observational study, which is the ideal design for predictive risk model development (as it can facilitate comprehensive and accurate data collection and ADE identification), was used by all studies except for the GerontoNet risk score by Onder et al. 50, where a database [Gruppo Italiano di Farmacoepidemiologia nell'Anziano sample (GIFA)] was retrospectively accessed for model development. The GIFA data originated from surveys undertaken between 1993 to 1997, approximately a decade prior to the development and validation of the GerontoNET score 50.

Outcomes

Consistency in the definitions of outcome is important for the comparison and replication of studies. Studies in this review used several outcome measures and, in some instances, interpreted the same outcome with varying definitions. The study by Sakuma et al. 54 was the only one to measure all types of ADEs, including medication errors that resulted in harm 54. Trivalle et al. 51 reported ADEs, excluding therapeutic failures, poisoning and intentional overdose whilst McElnay et al. 52 included ADRs, ineffective treatment and overdose in their definitions of ADEs, however it was unclear if harm from medication errors were included. Of the remaining studies, six measured ADRs 49, 50, 53, 56, 58, 59, one study measured clinically significant medication errors and assessed if any errors resulted in harm 57, and another measured clinically significant DRPs (the majority of which were prescribing errors) but did not assess harm 55. As a result of the diverse definitions, outcome rates varied widely between studies, ranging from 6.5% in the study by Onder et al. 50, to as high as 61% in the study by Passarelli and Filho 53 (refer to Table 1).

Where ADEs and ADRs were measured, they were identified using appropriate methods, combining strategies to increase detection rates. These included regular, prospective review of inpatient charts, notes, computer records and examination of incident reports. In the study by Tangiisuran et al. 49, a trigger tool method was combined with incident analysis 49. In Nguyen et al.'s study 57, 15 clinical pharmacists followed patients during their admission and recorded and graded medication errors. Only medication errors with the potential for patient harm were included 57. The frequency of chart review was not reported. The study by Urbina et al. 55 identified DRPs using the hospital's computerized physician order entry software, which generated alerts that were evaluated by clinical pharmacists. Those judged to be of clinical significance were included in the study. The study did not evaluate subsequent outcomes of DRPs or report potential severity.

To ensure reliable classification of outcomes, a robust review and consensus by several independent experts, with extensive medical and pharmacological knowledge is needed. Teams of two or more reviewers were used by eight of the 11 model development studies to confirm if a suspected event met the outcome definition. It was unclear in three studies if a second reviewer was involved in assessing outcomes 52, 53, 55. A multidisciplinary panel of physicians, clinical pharmacologists and pharmacists was used by six studies 49, 51, 54, 56, 57, 59.

None of the studies reported blinding to predictor variables when screening for outcomes in the model development phase, although Onder et al. 50 and Tangiisuran et al. 49 did report blinding investigators in the validation phase. This is important for outcomes such as ADEs, which often manifest with nonspecific symptoms and are subject to interpretation, as an investigator's knowledge of a patient's pre‐existing risk factors (i.e. predictor variables) could bias their assessment of whether a patient has experienced a medication related incident.

Candidate predictor variables

A well‐defined, comprehensive list of clinically relevant candidate predictor variables is important in the development of the final multivariable model 60. Variables collected in the included studies are summarized in Table 1. Seven of the studies measured patient demographics, medications, medical conditions and a variety of laboratory tests, most commonly renal function, as predictor variables. The study by Trivalle et al. 51, only included medications and medical conditions. Urbina et al. 55, Nguyen et al. 57, and Passarelli and Filho 53 did not assess laboratory variables. Nguyen et al. 57 did not assess diagnoses or comorbidities or any social risk factors.

Tangiisuran et al. 49 and McElnay et al. 52 reported the most comprehensive list of candidate predictor variables, including social risk factors (e.g. marital status, living situation, smoking and alcohol history) and hospital re‐admission. Zopf et al. 56 reported the most comprehensive list of laboratory predictor variables, although this only included the first test results post admission. McElnay et al. 52 was the only study to report the inclusion of serum drug levels used for therapeutic drug monitoring. Variables were collected prospectively using standard checklists, early post admission (Table 1).

In some instances, there was lack of a clear definition for some of the variables collected, such as comorbidities, anaemia, diabetes, heart failure, liver disease/hepatic failure, ‘history of ADRs’ and ‘patient's belief that their medication was in some way responsible for their hospital admission’ 49, 50, 52, 53, 58. For example, Onder et al. 50 and Tangiisuran et al. 49 did not state what haemoglobin level was used to diagnose anaemia or what liver function result, such as, greater than twice upper limit of normal, would qualify as ‘liver disease’ 49, 50.

Handling of continuous variables

Continuous risk variables should be analysed and presented appropriately and nonlinear transformations used where indicated, or if categorized for practical purposes a rationale is provided for how risk categories were derived. However, despite this all studies, other than Nguyen et al. 57, categorized some continuous variables, most commonly age and number of medicines, in some instances without explanation of how thresholds were determined.

To comply with best statistical methodology, Nguyen et al. 57 analysed variables such as age and number of medications as continuous and conducted multivariable polynomial analysis to account for nonlinearity of continuous variables. Trivalle et al. 51, appropriately justified categorizing the number of medications with four groups chosen, based on quartiles of distribution, with an approximately constant increase in risk between adjacent groups 51. Urbina et al. 55 categorized age and number of medications using cut‐off points with the highest sensitivity and specificity 55. Zopf et al. 56 categorized smoking and alcohol consumption and used area under the receiver operative characteristic curve (AuROC) analysis, and Youden's index to determine an optimal cut‐off score for the number of medications 56. O'Connor et al. 59 also appropriately analysed the number of medications and potentially inappropriate medications as continuous variables using a block entry method, however renal impairment was dichotomized without explanation 59.

Onder et al. 50 categorized several variables, such as the number of medications into three groups, two of which were overlapping (0–5 and 5–7), making interpretation of the final model challenging 50.

Sample size

In the development of multivariable risk models, adequacy of sample size is assessed by calculating events per variable (EPV) and a minimum EPV of 10 is generally recommended. This is calculated by using the number of outcomes divided by the number of candidate predictor variables in the development cohort 48. Amongst the reviewed studies, insufficient sample size, as defined by an EPV of < 10, was seen in three studies 49, 54, 58. A further three studies, provided insufficient information to calculate the EPV 51, 52, 53. A low EPV can result in a poorly fitted model and a subsequent risk score that does not perform well in different patient populations.

Missing data

Reporting on the frequency and types of missing data was limited and only discussed by three studies 50, 56, 58. Onder et al. 50 used complete case analysis (excluding all missing data), with no information provided on number of patients, types or numbers of variables affected 50. Sharif‐Askari et al. 58 excluded patients with >10% missing data, also not specifying how many patients or which variables were affected, but appropriately used multiple imputation techniques to deal with remaining missing data 58. Zopf et al. 56 reported excluding alcohol as a risk variable in a second multivariable analysis due to high rate (26%) of missing data 56.

Modelling

Studies used binary logistic regression to develop models. All model development studies preselected candidate predictor variables using univariable analysis, despite recommendations against this approach (with a full model approach being preferable), as it can exclude variables that may later become significant after adjustments during multivariable modelling 48. The study by Tangiisuran et al. 49 also included variables found to be significantly associated with ADE risk from the literature, despite not being statistically significant in the univariable analysis. This is a recommended approach that ensures important predictor variables which might not have been adequately represented in the development cohort, and hence could not reach statistical significance, are not excluded too early 61. For selection of predictors, during multivariable modelling, studies used step‐wise methods with backwards elimination and/or forwards selection (Table 1).

Performance

A model's potential clinical usefulness is determined by its' discrimination between patients with or without an ADE. This can be measured using the area under the receiver operative characteristic curve (AuROC), also known as the concordance (C) statistic 62. The AuROC (or C‐statistic in one study) 58 was used to measure model discrimination in nine studies. Two studies reported model accuracy 52, 53.

Using AuROC allows quantification of the probability that “a patient with an ADE had a higher predicted probability than a patient without an ADE” 49, 62, 63. An AuROC of 1 represents a perfect model whilst 0.5 is random concordance (where 0.5–0.59 equals a failed model). The discrimination of the models in the development studies ranged from 0.63 (0.60–0.69 equals poor performance) to 0.813 (0.80–0.89 equals good performance) as per Table 2. Passarelli and Filho 53 and McElnay et al. 52 reported performance measures of accuracy of 70% and 63% respectively (Tables 2 and 4) 52, 53.

Table 2.

Model performance

First author (year) [ref]	Discrimination	Classification	Calibration
Nguyen (2017) 57	AuROC 0.718 (95% CI 0.689–0.748)	NR	Calibration plot Intercept = 0 Slope = 1 Hosmer–Lemeshow NR
Tangiisuran (2014) 49	AuROC 0.74 (95% CI 0.68–0.79)	Youden's index = 0.36 optimal risk score cut‐off >1 Sensitivity 80% Specificity 55%	Hosmer–Lemeshow P = 0.757
Urbina (2014) 55	AuROC 0.778 (95% CI 0.768–0.7891)	NR	Hosmer–Lemeshow P = 0.131
Sharif‐Askari (2014) 58	C‐statistic 0.813 (95% CI 0.760–0.867)	NR	Hosmer–Lemeshow P = 0.874
Sakuma (2012) 54	AuROC 0.63 ± 0.03	NR	NR
Trivalle (2011) 51	NR (reported for validation, Table 4)	NR	NR
Onder (2010) 50	AuROC 0.71 (95% CI 0.68–0.73)	Risk score cut‐off 3–4 Sensitivity 68% Specificity 65%	NR
Zopf (2008) 56	AuROC 0.78 (model 1 – without labs) AuROC 0.80 (model 2 – with labs) CIs = NR P = 0.0023	Youden's index = NR Model 1: sensitivity 65%, specificity 80% Model 2: sensitivity 64%, specificity 86%	NR
Passarelli (2007) 53	Accuracy 70.4%	ADR risk greater than P = 0.5 Sensitivity = 88.7% Specificity = 40.8%	Hosmer‐Lemeshow P = 0.348
McElnay (1997) 52	NR (reported for validation, Table 4)	NR	NR

AuROC, area under the receiver operative characteristic curve; labs, laboratory test results; NR, not reported

Table 4.

Model validation

First author (year) [ref]	Validation method	Outcome and power	Performance
Nguyen (2017) 59	Design: internal validation (bootstrap) n = 500 resamples Dates: 1–31 April 2014	NA	AuROC 0.707 (95% CIs NR) Calibration plot: Intercept = −0.069 slope = 0.926 Odds ratios corrected by shrinkage factor = 0.926
Tangiisuran (2014) 49	Design: external (geographical) validation. Prospective observational Setting: geriatric and medical wards of four European hospitals n = 483 (mean age 80 years) Dates: September–December 2008	56 patients with ADRs: 11.6% (vs.12.5% development study)   Insufficiently powered	For cut‐off score > 1 Youden's Index = 0.26 AuROC 0.73 (95% CI 0.66–0.80) Sensitivity 80% Specificity 46%
Urbina (2014) 55	Design: external (temporal) validation. Prospective observational Setting: acute medical and surgical wards of Spanish hospital n: 3598 (mean age 59 years) Dates: September–December 2009	876 patients with at least 1 DRP: 21.6% (vs. 29.9% development study)   Adequately powered	AuROC 0.776 (95% CI 0.759–0.792)
Sharif‐Askari (2014) 58	Design: internal validation (bootstrap) n = 1000 resamples Dates: January–December 2012	NA	Bootstrapping did not change significant variables with similar SEs and CIs reported
Sakuma (2012) 54	Design: internal validation (split‐model with 1:1 random allocation) Prospective observational Setting: acute medical and surgical wards of three Japanese Hospitals n = 1730 of total 3459 (mean age reported as similar to development cohort) Dates: January–June 2004	350 patients with at least 1 ADE: 20% (vs. 21% development study)   Adequately powered	AuROC 0.63 ± 0.03
O'Connor (2012) 59	Design: re‐validation of model by Onder (GerontoNet ADR risk score) Prospective observational Setting: acute medical and surgical wards hospital in Northern Ireland n = 513 (median age 77 years) Dates: July–October 2010	135 patients with at least 1 ADR (178 ADRs total): 26%   Adequately powered	AuROC 0.62 (95% CI 0.57–0.68) admission AuROC 0.51 (95% CI: 0.46–0.57) on Day 5 AuROC 0.55 (95% CI: 0.47–0.62) on Day 10
Onder (2010) 50	Design: external (geographical) validation. Prospective observational Setting: geriatric and medical wards of four European hospitals n = 483 (mean age 80 years) Dates: September–December 2008	56 patients with ADRs: 11.6% (vs. 6.5% development study)   Insufficiently powered	AuROC 0.70 (95%CI, 0.63–0.78)
Trivalle (2011) 51	Design: internal validation (bootstrap) n = Number of resamples not reported Dates: NR	NR	AuROC 0.70 (95% CI 0.65–0.74)
McElnay (1997) 52	Design: internal validation (split model). Prospective observational n = 181 (mean age reported at similar to development cohort) Dates: NR	37 Patients with ADE in validation cohort: 20%   Insufficiently powered	Using optimal cut point of ADE P ≥ 0.3 (0 = no risk of ADE and 1 = highest risk) at risk score ≥ 4 Accuracy 63% Sensitivity 40.5% Specificity 65%

NA, not applicable; NR, not reported; SE, standard error; Geographical validation, type of external validation, new individuals from different countries, considered gold standard to assess score generalizability; Temporal validation, type of external validation using new individuals, often within the same hospital but at a later time; Split sample method, type of internal validation, initial sample of patients are randomly divided in to two groups, one for development and one for validation; Bootstrap method, type of internal validation, preferred method to split‐sample as it adjusts for over‐optimism from apparent performance of model

Another key measure of predictive performance is a model's calibration which is the agreement between the probability of developing an ADE as predicted by a model and actual outcomes 64. Because the AuROC only shows if patients were classified correctly as high or low risk and does not account for the degree of accuracy of the predicted probabilities, it is also important to ensure reporting of model calibration. Only five studies, reported calibration, four using the Hosmer–Lemeshow statistic, which that showed the models had an adequate fit to observed events 49, 53, 55, 58 and one study presented a calibration plot that also showed reasonable calibration 57.

Tangiisuran et al. 49 was the only study to report Nagelkerke R², recommended in multivariable analysis to measure the strength of association of the predictor variables in explaining the variance between model prediction and the outcome measure. Low values suggest that other variables may predict risk of ADR more accurately. From the R² of 0.16, the percentage of variance that the BADRI Score accounted for was only 16% 49.

Classification measures of sensitivity and specificity, which can be useful for determining if a risk score can correctly classify high/low risk patients, were reported by four studies using an optimal risk score cut‐off 49, 50, 53, 56. Sensitivity and specificity values ranged from 64% 56 to 88.7% 53 and 40.8% 53 to 86% 56, respectively (Table 2).

Model presentation

All studies presented a final model of significant variables generated from logistic regression analysis (Table 3). The number of variables in the final models ranged from three to 14. ‘Number of medications’ featured in nine risk scores, except in models by Sakuma et al. 54 and McElnay et al. 52. Renal impairment 50, 58, 59, older age 50, 55, 58, 59, some comorbid conditions 50, 53, 55 also featured in several models. Several laboratory‐related variables were present in the final models; however, there was no common theme except for renal impairment, as per Table 3. The study by Sakuma et al. 54 included junior prescribers and preoperative patients as a variable in their final models – both of which were unique predictor variables only included in the JADE study 54. In the study by Nguyen et al. 57 age was analysed as a continuous variable and found to be significant in the final model, with maximal risk at 75 years; however, the question of how best to apply this variable remains, as clinical application of the model is difficult without defined risk categories. Predictor variable weights were either kept at the estimated odds ratios or simplified to allocate a score for ease of calculation.

Table 3.

Final multivariable model

First author (year) [ref]	Final model		Score development
	Significant variables	OR (95% CI)
Nguyen (2017) 57	Increasing age (max risk 75 years) Number of medications Treatment initiated before admission Best possible medication history	4.26 (3.56–5.09)   1.16 (1.10–1.23) per medication 5.64 (2.38–13.36)   0.5 (0.37–0.67)	Outcome: MEs 4 variables in the final model. Weighting of variables kept at OR Age not categorised and maximal risk at 75 years Best possible medication history reduced ME risk
Tangiisuran (2014) 49	Hyperlipidaemia Number of medications (≥8) Length of stay (≥12 days) Use of anti‐diabetic agents High WCC on admission	3.316 (1.811–6.072) 3.300 (1.927–5.651) 2.269 (1.345–3.826) 1.906 (1.040–3.493) 1.548 (0.940–2.548)	Outcome: ADRs 5 variables in risk score Simplified scoring with 1 point per variable Risk score range: 0–5
Urbina (2014) 55	Number of drugs >10 ATC S: sensory organs ATC V: various MDC mskl/connective tissue ATC J: syst anti‐infectives MDC circulatory system MDC kidney and urinary tract ATC C: cardiovascular system MDC 0: Others MDC nervous system MDC digestive system CCI = 2 ATC H: hormone therapy Age >60 years	3.335 (2.956–3.763) 2.559 (1.717–3.814) 2.181 (1.679–2.834) 1.937 (1.432–2.619) 1.913 (1.696–2.157) 1.892 (1.400–2.557) 1.616 (1.169–2.235) 1.546 (1.352–1.769)   1.393 (1.056–1.838) 1.393 (1.002–1.937) 1.393 (1.042–1.863) 1.332 (1.183–1.499) 1.198 (1.050–1.367) 1.197 (1.051–1.364)	Outcome: DRPs 14 variables in risk score with points as per OR (i.e. OR 1–1.99 = score of 1, OR 2–2.99 = score of 2, etc.) Risk score range: 0–18
Sharif‐Askari (2014) 58	≥8 medications >10 mg l–1 C‐reactive proteins ESRID, conservative management   Vascular disease Serum albumin <3.5 g/dL Female sex Age ≥65 years	4.64 (2.51–8.59) 2.41 (1.33–4.37) 2.39 (1.21–4.14)   2.36 (1.24–4.46) 2.24 (1.21–4.14) 1.33 (0.73–2.41) 1.16 (0.62–2.17)	Outcome: ADRs 7 variables in risk score with points as per OR (i.e. OR 1–1.99 = score of 1, OR 2–2.99 = score of 2, etc.) Risk score range: 0–14
Sakuma (2012) 54	Burden of illness (CCI) Dementia Hemiplegia Cancer Consciousness (clear) Preadmit meds (laxatives) Dyspnoea Preoperative patient Junior doctors (<3 years)	2.8 (2.0–4.1) 1.8 (1.2–2.8) 1.6 (1.2–2.0) 2.6 (1.6–4.2) 1.7 (1.3–2.2) 1.7 (1.3–2.3) 1.6 (1.2–2.1) 1.5 (1.2–2.0)	Outcome: ADEs 8 variables in final model Weighting of variables kept at OR   Risk groups derived using average probability of ADEs in the development cohort: P = 0.22 (376/1729) 15% high risk (P ≥ 0.3) = 43% ADE rate 36% medium risk (0.2 ≤ P < 0.3) = 27% ADE rate 54% low risk (P < 0.2) = 15% ADE rate
O'Connor (2012) 59	Age (years) 65–74 75–84 ≥85 Number of STOPP medications Liver disease Renal Failure (eGFR ≤60 ml/min) Number of medications Assistance with ≥1 ADLs	2.12 (1.23–3.70) 2.22 (1.68–4.23) 2.40 (1.25–4.50)   1.86 (0.90–3.84) 1.81 (1.12–2.92)   1.09 1.02–1.17) 0.75 (0.45–1.26)	Outcome: ADRs Additional finding to study by Onder et al. that each additional PIM was associated with double the risk of an ADR (OR: 2.40, 95% CI: 1.26–4.59) Weighting of variables kept at OR   Older patients (≥85) with renal impairment and IP were nine times more likely to have an ADR. Patients who needed assistance with ≥1 ADLs had a lower risk of ADRs.
Onder (2010) 50	Number of medicines ≤5 5–7 ≥8 Previous ADR Heart failure Liver disease ≥4 comorbid conditions Renal failure	1 [reference] 1.90 (1.35–2.68) 4.07 (2.93–5.65) 2.41 (1.79–3.23) 1.79 (1.39–2.30) 1.36 (1.06–1.74) 1.31 (1.04–1.64) 1.21 (0.96–1.51)	Outcome: ADRs Six variables in final score with points as per OR (i.e. OR 1–1.99 = score of 1, OR 2–2.99 = score of 2, etc.) Risk score range: 0–10
Zopf (2008) 56	Raised body temperature Female sex No. of drugs (> 10) Low erythrocyte count Low thrombocyte count	1.659 (1.172–2.348) 1.562 (1.001–2.438) 1.120 (1.079–1.162) 0.445 (0.230–0.863) 0.773 (0.617–0.967)	Outcome: ADRs Five variables in final model and weighting kept at OR
Passarelli (2007) 53	Use of inappropriate drugs Number of diagnoses Number of drugs	2.32 (1.17–4.58) 1.41 (1.06–1.86) 1.10 (1.03–1.17)	Outcome: ADRs   A risk stratification rule was developed using three variables in final model ‐ giving ADR probability above 0.5   Patients were assigned to risk groups based on their number of drugs, inappropriate drugs and number of diagnoses
Trivalle (2011) 51	Antipsychotics Recent anticoagulation   Number of medications	2.5 (1.5–4.1) 2.0 (1.1–3.7)   1.9 (1.6–2.3)	Outcome: ADEs   Final risk score: Number of medications (four groups): <7 (0), 7–9 (+6), 10–12 (+12), ≥13 (+18) Antipsychotic use (+9) Recent anticoagulant use (+7) Risk score range: 0–34   Risk Groups (based on total risk score): ≤6 risk = 12% (95% CI: 8–15%) 7–12 risk = 28% (95% CI 19–36%) 13–18 risk = 35% (95% CI: 28–43%) > 18 risk = 52% (95% CI: 40–62%)
McElnay (1997) 52	Antidepressants ‘Thinks drug responsible for hospitalization’ Potassium (< 3.6 or >5.2 mmol/L) COAD GI problems Digoxin Angina	5.7942 (2.12–15.85) 4.2103 (2.18–8.14)   2.5740 (1.35–4.91)   2.4057 (1.06–5.44) 2.1606 (1.13–4.15) 1.9905 (1.05–2.33) 0.1676 (0.07–0.42)	Outcome: ADEs Seven variable risk model for ADE risk score ≥ 4 (P ≥ 0.3) Weighting kept at OR

ADEs, adverse drug events; ADL, activities of daily living; ADRs, adverse drug reactions; ATC, anatomical therapeutic chemical; COAD, chronic obstructive airway disease; DRPs, drug related problems; eGFR, estimated glomerular filtration rate; GI, gastrointestinal problems; MDC, major diagnostic category; MEs, medication errors; OR, odds ratio; PIM, potentially inappropriate medication; STOPP, Screening Tool of Older Person Prescriptions

Model validation

Of the 11 studies, four undertook external validation in a different patient population to the development study 49, 50, 55, 59. The studies by Onder et al. 50 and Tangiisuran et al. 49 were underpowered with less than the minimum 100 recommended events 48. Three of the four studies reported only a small difference (≤0.01) in their AuROC from external validation compared with performance in the development studies 49, 50, 55. The study by O'Connor et al. 59 revalidated and updated the GerontoNet ADR risk score using a cohort of acutely ill elderly patients in an Irish hospital. This study described sample size rational and was adequately powered with 178 ADRs in 135 patients. The AuROC (0.62, 95% CI 0.57–0.68) was significantly lower compared to its original development study (0.71, 95% CI 0.68–0.73). This study also identified additional predictors of age ≥75 years and potentially inappropriate medicines PIMs (Table 4) 59.

Five studies were internally validated. Three used bootstrap methods 51, 57, 58. Nguyen et al. 57 was the only study to present a calibration plot and apply a shrinkage factor to correct regression coefficients based on findings of over‐fitting from bootstrapping. Furthermore, the study used simulated experiments (n = 5000) to theoretically test the effect of their model on identification of medication errors when compared with using age or number of medications alone to predict high risk patients. The simulations demonstrated that use of the PRISMOR model in a low clinical pharmacist coverage scenario (10% cover) would result in a 17.5% mean improvement over age‐based selection alone (P < 0.05) in 83.2% of simulations 57.

Two studies by Sakuma et al. 54 and McElnay et al. 52 adopted a split sample method, where the development group was randomized to two cohorts at enrolment, and one used as the validation group. The studies by Passarelli and Filho 53 and Zopf et al. 56, did not undertake a validation process.

Discussion

In this systematic review, 11 studies were identified that developed or updated a predictive risk model for inpatient ADEs. We used the CHARMS Checklist to appraise study quality, and found some limitations in development, validation and reporting methods 48. As outlined by CHARMS, a clinically useful model should accurately discriminate high‐ from low‐risk patients, be well calibrated and ideally externally validated to assess performance in different populations. The subsequent risk score should also be impact tested to evaluate its ability to change patient outcomes in a real patient population. Whilst one study attempted to test the impact of their model, it simulated potential impact rather than evaluate actual patient outcomes 57. From the remaining studies, three risk scores were externally validated 49, 50, 55, one of which was developed using better model development and reporting methods 49, 55. Tangiisuran et al.'s BADRI risk score demonstrated fair discrimination, was well calibrated and externally validated in four European hospitals 49. The reasonable performance, from the external validation of the BADRI model, suggests that the risk score could be further evaluated and has the potential for prioritizing high risk patients for targeted interventions in a local hospital setting. Urbina et al.'s model also performed well in external validation, however it did not assess outcomes of DRPs and its' potential use for ADE prediction is unknown 55. Trivalle et al. 51 developed a practical risk score with three variables and a comprehensive scoring approach (based on probability of ADE risk for different risk groups), and had reasonable performance in the internal validation of the score; however, their risk score was not externally validated 51.

Recently Petrovic and colleagues externally validated the GerontoNet ADR risk score 65. Although the discrimination of the GerontoNet was better than in O′Connor et al.'s study 59, it was still significantly lower than that the AuROC in the original study (0.64 vs. 0.71 respectively). The score's discrimination improved when only predicting type A ADRs (AuROC 0.69). When subpopulations were studied the score had fair discrimination (AuROC range 0.70–0.79) in certain age groups (<70 or ≥80 years), medical conditions (heart failure or diabetes) and with a history of prior ADR. It had good discrimination for patients with Type A ADRs who had body mass index <18.5 kg m⁻², mini‐mental state examination score >24 and osteoarthritis. However, due to the small sample size of the subpopulations extrapolation of these results to other patients would be less meaningful.

One of the challenges when developing a predictive risk model for ADEs is defining the study outcome. Similar to the inconsistencies in definitions in safety literature, with ADEs and ADRs used interchangeably, there was variation in definitions amongst the reviewed studies. ADEs were measured by three studies using different definitions, six studies measured ADRs, one study measured DRPs and another measured medication errors. In the ideal study, all medication events that result in patient harm, or have a high potential for harm, should be included to better facilitate the design and targeting of interventions to mitigate ADEs. Whilst DRPs and medication errors are attractive outcomes, as they are common and easy to measure, most do not result in patient harm 47, 66. ADRs form a significant proportion of medication harm events but they do not account for all ADEs. Harm from medication errors and therapeutic failures contribute to approximately 25% of ADEs 47. Given that such harm is often preventable, early identification of high risk patients can better enable implementation of preventative measures. Furthermore, a well‐defined and consistent outcome, which encompasses all subsets of ADEs, will make comparison between studies more meaningful and is likely to improve the generalisability of the risk score.

Our review found limitations in the timing of measurement and analysis of predictor variables, which play critical roles in determining the final model. Most studies measured variables at hospital admission. Whilst this approach is suitable for variables that do not change during a patient's hospital stay, such as age or ethnicity, time‐dependant variables, for example, laboratory results, should not be treated as fixed values – unless the risk score is only intended for use on admission. For model development, such variables should be evaluated at regular intervals throughout admission, ideally near the time of ADE detection, to enable the relationship between the variable and the outcome to be quantified. This is particularly important if a score is to be used later in the admission such as when a patient transfers specialty, or before discharge. Of the studies in our review, two reported daily assessment of variables, however it was unclear how changes in variables were analysed (for example, if the values of variables used in their analysis aligned with the time of ADE detection) 53, 54. Another study reported that variables were updated with ADE detection but their model was based on medications and did not assess laboratory variables 51. The study by O'Connor et al. 59, which validated the GerontoNet ADR risk score, recorded predictor variables on admission, day five and day 10 and was the only study that reported a separate analysis of model performance at specific time points. Interestingly 25% of patients had higher risk scores at discharge, highlighting that patient risk changes during the course of hospitalization and that depending on the time when the prediction model is to be used, a variable only measured at one time point may not be accurate for model development.

Other issues in the handling of predictor variables included a lack of blinding to variables when measuring outcomes and poorly defined variables, such as liver impairment, anaemia, comorbidities and diabetes, amongst others 49, 50, 52, 53, 54, 58. All studies categorized certain variables, commonly age and the number of medications, a method of rounding that risks loss of important information, loss of power, and may lead to poor model fit 48, 67. However, it may be argued that for clinical use, categorization of certain variables is necessary to develop a user‐friendly risk score.

The type and handling of missing data was frequently not reported. Where it was reported, missing data were excluded from the analysis in all but one study 58. Rather than excluding missing variables, multiple imputations may be used. Inadequate sample size based on EPV < 10, or inability to assess sample size, due to incomplete lists of original candidate predictor variables, was also a shortcoming. Sample size rational (for model development) was reported in three studies 49, 57, 58. Two appear to have used the number of predictor variables in the final multivariable model for their calculation, despite the recommended approach to use the number of original candidate predictor variables 49, 58. Insufficient sample size can cause model over‐fitting resulting in a model with a poor predictive ability, and subsequent lack of generalizability of the score in new patient populations.

Preselection of variables using univariable analysis based solely on statistical significance was used by all studies; however, this method is not optimal as correlations between variables may not be identified or variables may be prematurely rejected 68. Variables were appropriately selected in multivariable modelling using stepwise methods of forward selection and/or backward elimination 68. The addition of backward elimination, which starts with a full list of variables, is preferred and was used in the majority of studies.

Disappointingly there were deficiencies in meeting assumptions for multivariable modelling. In addition to insufficient sample size, the assessment of multicollinearity was only reported by four studies 49, 51, 56, 58. Only McElnay et al. 52 reported how variable interactions were assessed, whilst no study specified the coding of variables, both of which are necessary for accurate replication of a model. Similar issues have been reported in systematic reviews of other risk scores, highlighting the complexities of predictive risk modelling in healthcare 35, 68, 69, 70. The recently published Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement, should help raise the quality and reporting standards of these studies, as previously no consensus guidelines existed 71.

Irrespective of how a model is developed, clinical implementation of the score depends predominantly on how well it performs with respect to discrimination and calibration. In this review, all studies reported some form of model discrimination, most commonly the AuROC. However, calibration results were only reported in five studies 49, 53, 55, 57, 58. Similarly, only six studies (development and/or validation phase) reported classification measures of sensitivity and specificity. Limited reporting of these measures may be due to the requirement of predefined thresholds with a model's sensitivity and specificity changing at different risk scores 48. Often a model with higher sensitivity will report lower specificity and vice versa depending on the chosen cut‐off score. For an ADE risk score, sensitivity is most important as misclassification of a high‐risk patient could result in patient harm. However, a risk score with poor specificity would be less useful in a resource constrained hospital setting as it could incorrectly flag low risk patients for intervention. The goal of an ADE risk score should be to optimize detection of high risk individuals (sensitivity) with an acceptable number of false alarms (1 – specificity). Overall, from the studies that reported classification measures three had reasonable sensitivity (80% or greater) but poor specificity 49, 53, 65.

Whilst some studies presented their final model using simplified scoring or a rule, others kept variable weights as odds ratios. An example of a practical score, allowing simple calculation, was the BADRI risk score, which allocated one point to each of the final five variables 49. Although it may be argued that such simplification might make the score less discriminatory, user friendliness is also an important consideration. For a score to be applied at the bedside it should include easily identifiable variables with practical scoring to allow quick calculation and risk stratification. Four studies did not proceed to simplify their model into a score 52, 54, 56, 57. Practicality of the score is also likely to influence clinician uptake and face validity. Alternatively, with increasing availability of electronic health records, a pragmatic solution that mitigates the need for cumbersome calculations could be a real‐time risk score, which can provide up‐to‐date and clinically informative data throughout hospitalisation, and guide clinical interventions based on changes in variables. A similar approach was used by a New Zealand hospital which developed software for identifying patients at high risk of ADEs by extracting risk variables from the hospital's databases 72.

Several health care organizations have developed and implemented tools, mainly using consensus methods, to help identify inpatients at risk of medication harm 72, 73, 74, 75. If such tools are not developed and validated with sufficient rigour, there could be implications for patient safety. To reduce harm systematically, we must iteratively improve our ability to predict high risk circumstances, provide mechanisms for timely interventions, feedback and evaluation. A well‐developed model and validated risk score, is one potential medication safety solution.

In their systematic review of ADE risk models in older patients, Stevenson and colleagues concluded that none of the models reviewed were yet suitable for implementation 43. Our review concurs with, and adds to, these findings that unfortunately no perfect model was identified and that studies did not meet all requirements for best statistical approach to developing, validating and reporting on a predictive risk model. We found one study, the BADRI risk score, used reasonable methodologies and their model performed well in both development and external validation studies and so could be considered for further research in other hospitals. The risk score developed by Trivalle et al. 51, also demonstrated reasonable performance with a practical final score that has potential for external validation and potential updating, in hospitalized patients. To address limitations and better tailor these models to new patient populations, these risk scores should be externally validated in an adequately powered study. Based on these findings, we aim to externally validate the BADRI and Trivalle risk scores in an Australian hospital inpatient setting. We hope that our findings will help determine the effectiveness of the two scores in predicting risk of ADEs in a new geographical setting and establish if impact studies would be warranted.

Limitations

This systematic review only included studies of predictive risk models for ADEs in hospital inpatients, developed using multivariable logistic regression analysis. As a consequence, we did not review ADE risk tools developed by other methods. This was deemed appropriate as predictive risk modelling uses an informed statistical approach which is systematic, reproducible and evidence based 36, 68. Also, due to the heterogeneity in outcomes of the included studies, and the reporting of different performance measures, a quantitative analysis was not possible, but we attempted to thoroughly review included studies using a qualitative approach. Our quality assessment of each study is subject to interpretation; however, to minimize subjectivity a standardized quality assessment checklist was used and all three study authors reviewed the extracted data.

Conclusion

ADE risk prediction is a complex endeavour but has the potential to contribute to reducing ADEs, improving patient safety and optimizing hospital resources. For research in this area to progress, efforts to update and re‐validate existing models, using recognized statistical methods and transparent reporting are required. In addition, clinical impact studies, to measure effectiveness of a risk score in hospitals, are needed. This can determine if risk scores are fit for clinical use and wider implementation.

Competing Interests

The authors have no conflicts of interest to declare.

The authors gratefully acknowledge the contribution of Christine Dalais, Liaison Librarian, at The University of Queensland.

Embase search query:

Search terms, their order and combination as outlined below:

Date restricted to 31 December 2016.

Exp = explode all trees

Risk model:

1. ‘Risk prediction*’

2. ‘Score’

3. ‘Clinical tool*’

4. ‘Risk prediction model*’

5. ‘Risk assessment’ / exp OR ‘risk assessment*’

6. ‘Risk analysis’

7. ‘Risk prediction score*’

8. ‘Prediction rule*’

9. ‘Decision support technique*’

10. ‘Decision support’

11. ‘Decision support system’/exp OR ‘decision support system*’

12. ‘Risk factor’/exp OR ‘risk factor*’

13. ‘Risk management’/exp OR ‘risk management’

14. ‘Risk score’/exp OR ‘risk score*’

15. ‘Prediction model’/exp OR ‘prediction model*’

1 OR 2 OR OR 3 OR 4 OR 5 OR 6 OR 7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13 OR OR 14 OR 15

Adverse drug events:

1. ‘Adverse drug reaction*’: ti,ab (title and abstract only)

2. ‘Adverse drug reaction’/mj [as a subject heading limited to major focus (/mj)]

3. ‘Adverse drug effect*’

4. ‘Drug‐related side effect*’

5. ‘Medication error’/ exp OR ‘Medication error*’

6. ‘Adverse drug event’/exp OR “adverse drug event*’

7. ‘Medication related problem*’

8. ‘Medication harm’

9. ‘Drug related problem*’

10. ‘Drug administration error*’

11. ‘High risk medication*’

12. ‘High risk drug*’

1 OR 2 OR 3 OR 4 OR 5 OR 6 OR 7 OR 8 OR 9 OR 10 OR 11 OR 12

Hospitalized patients:

1. ‘Hospital patient”/exp OR “hospital patient*’

2. ‘Inpatient*’ [searched for as a key word not as a subject heading]

3. ‘Hospitalised patient*’

4. ‘Hospitalized patient*’

5. ‘In‐hospital patient’

6. ‘Hospitalized’

7. ‘Hospitalised’

8. ‘Hospital’/exp

1 OR 2 OR 3 OR 4 OR 5 OR 6 OR 7 OR 8.

The final searches from Risk model, Adverse drug events and Hospitalized patients were combined with AND. The search was limited to English language and humans.

Falconer, N. , Barras, M. , and Cottrell, N. (2018) Systematic review of predictive risk models for adverse drug events in hospitalized patients. Br J Clin Pharmacol, 84: 846–864. doi: 10.1111/bcp.13514.