Abstract
Aims
Adverse drug reactions (ADRs) have major impacts on patients and the hospital system. Methods identifying ADRs from selected International Classification of Diseases–10th revision (ICD‐10) diagnosis and external cause codes can be applied to population‐level hospital admissions data, enabling the study of rare, yet serious ADRs. The present study aimed to use ICD10‐based methods to identify four types of serious idiosyncratic ADRs in Australia, and to assess changes in incidence and their impact on length of stay (LOS), readmission and in‐hospital mortality.
Methods
The study used a census of hospital admission data from New South Wales between July 2000 and June 2012. Changes in incidence rates over time relative to a control group were estimated using log‐linear regression. To assess impacts on LOS, readmission and mortality, each ADR case was matched with five controls, and cases were compared with controls via generalized linear models appropriate to each outcome.
Results
The incidence of three ADR types showed a significant increase over time relative to controls, while the fourth type showed no evidence of change. All ADR types were significantly associated with an increase in LOS of between 22% and 328%. Significant increases in risk of readmission or death were only observed for some ADR types.
Conclusions
Reducing the incidence of idiosyncratic ADRs is challenging. ICD10‐based methods support population‐level analyses that can provide important insights into the effects and changes in ADRs over time. This, combined with strategies related to both patient care and drug monitoring pre‐ and post‐commercial release, provides ways forward.
Keywords: adverse drug reactions, ICD‐10, population‐based
What is Already known about this Subject
Adverse drug reactions (ADRs) have major impacts on patients and hospitals.
The International Classification of Diseases–10th revision‐based methods to identify ADRs at a population level can be used to study rare but serious ADRs.
Detailed population‐level information on serious ADRs is important for reducing their incidence and impact.
What this Study Adds
The relative incidence of serious ADRs was at best nondecreasing, and in some cases significantly increased over the study period.
Serious ADRs were associated with a significantly longer length of hospital stay but only with a significant increase in the risk of readmission or in‐hospital mortality for certain ADR types.
These results highlight the disproportionate impact of rare serious ADRs.
Introduction
Adverse drug reactions (ADRs) can have major impacts on both patients and the health system. They are a significant cause of hospital admissions, with several meta‐analyses reporting around 5% of admissions due to ADRs 1, 2, and a major cause of mortality 1, 2, 3. It has also been estimated that ADRs occur during 6–15% of hospital admissions 3, 4, 5, with higher rates associated with older age, female gender, number of concurrent medications and length of stay (LOS) 4, 5, 6. These in‐hospital ADRs can also increase morbidity, LOS and risk of death 4.
Although relatively rare, certain ADRs are particularly serious. Clear examples include Stevens–Johnson syndrome and toxic epidermal necrolysis (SJS/TEN), and reactions involving the bone marrow and blood (aplastic anaemia, thrombocytopenia, agranulocytosis), liver (toxic liver disease) or kidneys (interstitial nephropathy). These serious ADRs have been associated with high case‐fatality rates and reduced long‐term survival relative to both the population and to patients with ADRs in general 7, 8, 9, 10. The majority of SJS/TEN cases are associated with reactions to specific drugs, including antibiotics, anticonvulsants, nonsteroidal anti‐inflammatory drugs (NSAIDs) and allopurinol 11, 12. Aplastic anaemia, thrombocytopenia and agranulocytosis are largely disorders of the marrow, sometimes drug induced, in which blood cell production is reduced. Somewhat predictably, the drugs implicated in these conditions include antineoplastic agents, while unexpected idiosyncratic reactions have been associated with a range of drugs, including allopurinol, antibiotics, antithyroid drugs and antiviral agents 13, 14, 15. Drug‐induced toxic liver disease is often associated with acetaminophen (paracetamol), anti‐infective drugs and immunosuppressive agents 16 and less commonly with medicines causing autoimmune hepatitis. Contrast media, analgesics, aminoglycosides and diuretics are frequently implicated in drug‐induced nephropathy 17 but less commonly there are serious idiosyncratic reactions to, e.g., proton pump inhibitors.
There is scarce documentation of the impacts of these serious ADRs on patients and the health system at a population level. With the availability of jurisdiction‐wide hospital admissions data, combined with International Classification of Diseases–10th revision (ICD10)‐based approaches for identifying ADRs, it is increasingly possible to evaluate rare ADRs. While limitations with ICD‐based methods have been noted 18, there is considerable scope for population‐level data on ADRs to quantify their burden and identify areas in need of closer investigation. The aim of the present study was to examine the incidence and impact of serious ADRs at a population level using a census of hospital admissions data. In particular, we aimed to examine changes in incidence over time relative to controls, and to assess the effect of ADRs on LOS and the risk of both readmission and in‐hospital mortality.
Methods
Data
The New South Wales (NSW) Admitted Patients Data Collection (APDC) is a census of hospital admissions from all public and private hospitals and day procedure centres across the state of NSW, Australia, which has a population of 7.5 million. The study used records of all admissions occurring between 1 July 2000 and 30 June 2012. ‘Admission’ in the present study refers to an episode of care that ends in discharge, transfer, death or change in type of care (e.g. from acute to rehabilitation). The APDC data records one row per admission; however, when a patient has multiple adjoining, overlapping or nested admissions, these records represent a single hospital stay. The APDC is an administrative dataset containing coded information on health procedures, diagnoses (55 fields), external causes (eight fields), facilities, costs and patient demographics. Clinical information is coded using the Australian modification to the ICD–10 (ICD‐10‐AM). Changes in classification over time were taken into account using mappings between ICD‐10‐AM versions. The NSW Registry of Births, Deaths and Marriages records the date of death for all deaths in NSW. Probabilistic data linkage was carried out by the NSW Centre for Health Record Linkage (CHeReL) to identify multiple admissions belonging to the same person and to link death records with hospital records 19. Ethics approval was provided by the New South Wales Population and Health Services Research Ethics Committee (reference: HREC/09/CIPHS/69).
Defining adverse drug reactions
The term ‘adverse drug reaction’ has been defined in varying ways. The World Health Organization definition, often used as the basis of study specific definitions, is: ‘A response to a drug which is noxious and unintended, and which occurs at doses normally used in man for the prophylaxis, diagnosis or therapy of disease, or for the modifications of physiological function’ 20. Edwards and Aronson built on this, to give a more precise discussion of what may or may not come under the umbrella of ADRs. They distinguish ADRs from medication errors, and suggest that most ADRs are toxic effects, negative side effects, or immunological or idiosyncratic reactions 21. An adverse drug event (ADE) is a broader category that encompasses ADRs and which is defined as: ‘Any untoward medical occurrence that may present during treatment with a pharmaceutical product but which does not necessarily have a causal relationship with this treatment’ 22. In addition to ADRs, ADEs also include medication errors (both omission and commission) and adverse events that are indirectly drug related, such as a fall in an elderly person taking a sedating drug.
Attributing a patient's reaction to a particular drug or combination of drugs is not trivial. Traditionally, manual record review by experts was used to identify ADEs, including ADRs; however, this can have low inter‐rater reliability (IRR) and is time intensive 23, 24, 25, 26. In order to address the IRR issue and capture the causal uncertainty, many methods have been developed that assess the likelihood that a particular patient's symptoms represent an ADR. An early example is the Naranjo algorithm, developed to improve agreement between clinicians' assessments of potential ADRs 23. Bayesian methods can be applied to detailed clinical data using a computational approach to generate a probability that a certain drug is implicated in a given adverse event 27. This can have good discrimination but requires sufficiently detailed clinical information. More recently, researchers have applied electronic health record (EHR) phenotyping algorithms to identify selected ADRs, particularly drug‐induced liver injury 28. Increasingly, algorithms based on ICD codes are being applied to jurisdiction‐wide hospital data, hence opening the way for population‐level assessment of ADRs 29, 30, 31; however, to date, most studies using this approach assess ADEs or ADRs overall rather than examining specific conditions resulting from ADRs.
There has been much variation in the way that researchers use specific ICD codes to identify ADRs and/or ADEs 31. For example, Stausberg and Hasford looked at ADEs and defined seven groups of ICD‐10–German modification (ICD‐10‐GM) diagnosis codes, ranked in order of the likelihood of representing an actual ADE 30. In comparison, Australian researchers developed the ICD‐10‐AM‐based classification of hospital‐acquired diagnoses (CHADx) algorithm to identify hospital‐acquired adverse events, including ADRs, through a combinations of diagnosis and external cause codes 29. This requires an indicator of whether each condition was present on admission to identify hospital‐acquired adverse events 32. Owing to our focus on specific conditions rather than ADRs overall, we have necessarily developed our own ICD10‐based approach to identify four groups of serious ADRs.
ICD‐based definition
The present study aimed to identify idiosyncratic ADRs – i.e. those that are both difficult to predict (neither a feature of the known pharmacology nor dose related) and rare. In the ICD‐10‐based definitions described below, we have attempted to exclude known dose‐related reactions, such as cytotoxic agents for marrow‐related ADRs, but we must acknowledge that there may still be dose involvement in some of our definitions, even if there is not a dose–response relationship within normal therapeutic dose ranges.
As with many ICD‐based studies of ADRs, an ADR in the present study was defined by the concurrent appearance of selected diagnosis codes, accompanied by the presence of selected external cause codes. Table 1 lists the code combinations used to define each ADR type examined in the present study. Specifically, we defined drug‐induced SJS/TEN by a diagnosis of L51.1 or L51.2 with the concurrent appearance of an external cause code in the range Y40–Y59, excluding Y42.0 glucocorticoids and synthetic analogues, and Y44.6 natural blood and blood products. Glucocorticoids were excluded as these may be used in the treatment of ADRs, so their appearance as an external cause may indicate a reaction subsequent to the main ADR or simply the treatment for the ADR. Similarly, drug‐induced toxic liver disease was defined where external cause codes Y40‐Y59 (with the same exclusions as above) appeared with a diagnosis of K71. Diagnoses of D61.1, D61.3, D61.8, D61.9, D69.5, D69.6 and D70 were used to define marrow‐related ADRs but with additional exclusions of external cause codes Y43.1–Y43.4. These exclusions aimed to omit predictable reactions to cytotoxic drugs, so that the definition better captured idiosyncratic cases. For kidney‐related ADRs (including interstitial nephritis), analgesic nephropathy was defined as a diagnosis code of N14.0 in combination with an external cause code for analgesics (Y45). Other forms of nephropathy (N14.1, N14.2) were considered in combination with all other external causes related to drugs in therapeutic use (Y40‐Y44 and Y46‐Y59, excluding Y42.0 and Y44.6). X‐ray contrast media (Y57.5) were also excluded as an external cause of other nephropathy as reactions to these are not considered idiosyncratic.
Table 1.
Australian modification to the International Classification of Diseases–10th revision (ICD‐10‐AM) codes used to define ADR cases
| Type of reaction | Diagnosis code | Diagnosis description | External cause code |
|---|---|---|---|
| Skin | L51.1 | Bullous erythema multiforme: Stevens–Johnson syndrome | Y40–Y59 (excl. Y42.0, Y44.6)a |
| L51.2 | Toxic epidermal necrolysis | Y40–Y59 (excl. Y42.0, Y44.6)a | |
| Blood/marrow | D61.1 | Drug‐induced aplastic anaemia | Y40–Y59 (excl. Y42.0, Y44.6, Y43.1–Y43.4) |
| D61.3 | Idiopathic aplastic anaemia | Y40–Y59 (excl. Y42.0, Y44.6, Y43.1–Y43.4) | |
| D61.8 | Other specified aplastic anaemias | Y40–Y59 (excl. Y42.0, Y44.6, Y43.1–Y43.4) | |
| D61.9 | Aplastic anaemia, unspecified | Y40–Y59 (excl. Y42.0, Y44.6, Y43.1‐Y43.4) | |
| D69.5 | Secondary thrombocytopenia | Y40–Y59 (excl. Y42.0, Y44.6, Y43.1‐Y43.4) | |
| D69.6 | Thrombocytopenia, unspecified | Y40–Y59 (excl. Y42.0, Y44.6, Y43.1‐Y43.4) | |
| D70 | Agranulocytosis | Y40–Y59 (excl. Y42.0, Y44.6, Y43.1‐Y43.4) | |
| Liver | K71.0 | Toxic liver disease with cholestasis | Y40–Y59 (excl. Y42.0, Y44.6) |
| K71.1 | Toxic liver disease with hepatic necrosis | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71.2 | Toxic liver disease with acute hepatitis | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71.3 | Toxic liver disease with chronic persistent hepatitis | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71,4 | Toxic liver disease with chronic lobular hepatitis | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71.5 | Toxic liver disease with chronic active hepatitis | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71.6 | Toxic liver disease with hepatitis, not elsewhere classified | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71.7 | Toxic liver disease with fibrosis and cirrhosis of liver | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71.8 | Toxic liver disease with other disorders of liver | Y40–Y59 (excl. Y42.0, Y44.6) | |
| K71.9 | Toxic liver disease, unspecified | Y40–Y59 (excl. Y42.0, Y44.6) | |
| Kidneys | N14.0 | Analgesic nephropathy | Y45 |
| N14.1 | Nephropathy induced by other drugs, medicaments and biological substances | Y40–Y59 (excl. Y42.0, Y44.6, Y45, Y57.5) | |
| N14.2 | Nephropathy induced by unspecified drug, medicament or biological substance | Y40–Y59 (excl. Y42.0, Y44.6, Y45, Y57.5) | |
| N14.4 | Toxic nephropathy, not elsewhere classified | Y40–Y59 (excl. Y42.0, Y44.6, Y45, Y57.5) |
Patients with an epilepsy diagnosis recorded in any field were assumed to be on anticonvulsants
Previous studies have differentiated ADRs prompting hospital admission 33 from those occurring during admission 34. In the APDC, the principal diagnosis field represents the diagnosis, established after study, to be chiefly responsible for occasioning the patient's episode of admitted patient care 35. Prior to 2006–07, the second diagnosis field was the ‘stay’ diagnosis, and indicated the condition responsible for the LOS. It was often, but not always, the same as the principal diagnosis. The third and subsequent diagnosis fields record additional diagnoses encompassing comorbidities present on admission and conditions arising during the admission 35. From 2006–07 onwards, the second diagnosis field was also used for additional diagnoses. In the APCD, it was not possible to separate ADRs occurring in hospital from those that were pre‐existing comorbidities as there was no indicator of whether or not additional diagnoses were present on admission. It was only possible to separate those ADRs that were the principal reason for admission (principally diagnosed) from all other ADRs (additionally diagnosed).
A principally diagnosed ADR was flagged when an ADR code combination appeared in the first diagnosis field and the first external cause field on the first record of a hospital stay (Figure 1A). If the same type of ADR was coded as both principal and additional on the same hospital stay, it was treated as principal only. An additionally diagnosed ADR was flagged when an ADR code combination appeared in the additional diagnosis fields with any external cause field on the first record of a hospital stay (Figures 1B and 1C); or when it appeared in any diagnosis field or external cause field on the second or subsequent record of a stay (Figure 1D). To account for the change in the definition of the second diagnosis field in 2006–07, this field was excluded as an additional diagnosis if the code was the same as the principal diagnosis (Figure 1B) but was included otherwise (Figure 1C). In general terms, principally diagnosed cases represent ADRs resulting in hospitalization, whereas additionally diagnosed cases correspond to a mixture of ADRs comorbid with the principal diagnosis at the time of admission and ADRs occurring during the hospital stay.
Figure 1.

Illustration of the diagnosis and external cause fields used to define principally and additionally diagnosed adverse drug reactions (ADRs)
The inclusion of external cause codes in addition to diagnosis codes when defining ADRs aims to maximize the proportion of true cases in the group classified as ADR cases. This may result in reduced sensitivity as the stricter criteria would cause more false negatives. However, as the number of true noncases vastly outweighs the number of true cases, the additional false negatives are a tiny proportion of the group classified as noncases. The intended result is then two groups with a high proportion of correctly classified records – i.e. high positive and negative predictive values.
Transfers and exclusions
When multiple admissions were adjacent or nested, these were combined into a single hospital stay where the principal diagnosis was taken from the first diagnosis field on the first admission record. Additional diagnoses were considered from all records during a given stay. Admissions for extracorporeal dialysis were excluded (10.5% of admissions) as these patients have many short admissions which would dominate an analysis at the level of hospital stays. Duplicate records were excluded (<0.1% of admissions). Hospital stays during which a patient died were excluded from analyses of readmission and LOS.
Statistical methods
Rates
Analysing trends in ICD‐defined conditions may be confounded by changes in coding practices. To address this issue, we used a comparative time‐series approach, whereby changes in the rate of ADRs were assessed relative to an appropriate comparison group. For each type of ADR, a comparison group consisted of patients with the same diagnosis but with no corresponding external cause code in the range Y40–Y59. A negative binomial model was fitted to annual counts, with year treated as a continuous variable, a binary variable distinguishing the case and comparison groups, plus the interaction between these two variables. A generalized estimating equations (GEE) approach was applied to account for correlation within the same year between the two groups 36. A significant interaction term indicates a change in ADR incidence over and above background trends or any changes in diagnosis coding.
Assessing the impact of ADRs using matched data
Owing to the rarity of the conditions of interest and the poor performance of logistic regression on such data, we avoided propensity score matching. Rather, a nearest neighbour approach was applied, whereby potential controls were selected that matched initially on gender, age within 5 years, comorbidity group, day stay status and admission date within 6 months. The three comorbidity groups were defined by a Charlson–Deyo comorbidity score 37, 38 of 1 or less, 2, or 3 or more. For the set of potential controls for each case, five were randomly selected that also matched on the greatest number of additional criteria comprising emergency department (ED) referral status, diagnosis‐related group (DRG) (surgical, medical, other), number of transfers during the hospital stay, time of day (four 6‐h blocks) and weekday of admission.
The three outcomes of interest were LOS, readmission and death in hospital. LOS was defined as the difference between the dates and times of discharge and admission for a given hospital stay. A readmission was defined as the existence of any hospital admission record within 28 days of separation from an ADR‐related hospital stay. Death in hospital was defined when the date of death was the same as the separation date for an ADR‐related stay. The effect of an ADR on each of these outcomes was assessed via univariate regression on the matched data, with binary ADR status as the only independent variable. For both readmission and mortality, logistic regression models were used, while for LOS linear regression was applied to the log‐transformed data. When modelling readmission and LOS, patients who died in hospital were excluded as they could not be readmitted and their LOS may be truncated. Correlation within clusters of matched patients was accounted for by the GEE approach, implemented with SAS PROC GENMOD (SAS Institute Inc., Cary, NC, USA). This takes into account the dependence between outcomes within each matched group and adjusts the standard error (and hence significance) estimates accordingly. Owing to the abundance of potential controls and the need to minimize unmeasured bias, the process of randomly selecting controls and applying models was repeated 100 times. The 100 estimates were combined using SAS PROC MIANALYZE to compute an average point estimate and to incorporate the between‐model variation into the overall standard error estimate 39.
Results
Out of 22 371 808 admissions in the final dataset, 21 053 455 hospital stays were identified, representing 6 607 817 patients. There was a higher proportion of females among patients with additionally diagnosed kidney‐related ADRs (75.1%) compared with the other ADR groups (47–59%) (Table 2). Kidney‐related ADR patients tended to be older, with median ages of 69 and 74 years, compared with medians of 48 and 62 years for the other conditions investigated. Among patients admitted for ADRs, the mortality rate ranged from 1.5% to 5.0% of hospital stays, compared with 1.3% in the general hospital population, while patients with an ADR as an additional diagnosis had higher rates, ranging from 8.8% to 13.4%. Crude readmission rates ranged from 15.6% to 36.8% in the ADR groups, compared with 19.4% in the general hospital population. LOS was longer, on average, for skin‐related ADRs and for additionally diagnosed cases.
Table 2.
Characteristics of adverse drug reaction (ADR) cases
| Skin | Blood/Marrow | Liver | Kidneys | |||||
|---|---|---|---|---|---|---|---|---|
| Principal | Additional | Principal | Additional | Principal | Additional | Principal | Additional | |
| ADR episodes, n | 162 | 172 | 831 | 6254 | 500 | 830 | 132 | 1643 |
| Females, n (%) | 94 (58.0) | 89 (51.7) | 431 (51.9) | 2902 (46.4) | 296 (59.2) | 436 (52.5) | 79 (59.9) | 1234 (75.1) |
| Age, median (IQR) | 47 (32–64) | 62 (42–76) | 61 (44–73) | 65 (50–77) | 62 (46–75) | 62 (46–76) | 69 (57–82) | 74 (67–80) |
| Referred from ED, n (%) | 128 (79.0) | 115 (66.9) | 569 (68.5) | 3834 (61.3) | 367 (73.4) | 553 (66.6) | 79 (59.9) | 847 (51.5) |
| DRG group, n (%) | ||||||||
| Surgical | 9 (85.6) | 40 (23.3) | 25 (3.0) | 1697 (27.1) | 9 (1.8) | 158 (19.0) | 8 (6.0) | 357 (21.7) |
| Other | 0 | 2 (01.2) | 0 | 202 (3.2) | 10 (2.0) | 28 (3.4) | 4 (3.0) | 62 (3.8) |
| Medical | 153 (94.4) | 130 (75.6) | 806 (97.0) | 4355 (69.6) | 481 (96.2) | 64 (77.6) | 120 (90.9) | 1224 (74.5) |
| Comorbidity score, n (%) | ||||||||
| 1 | 118 (72.8) | 86 (50.0) | 406 (48.9) | 2525 (40.4) | 332 (66.4) | 428 (51.6) | 67 (50.8) | 279 (17.0) |
| 2 | 40 (24.7) | 79 (45.9) | 404 (48.6) | 3100 (49.6) | 155 (31.0) | 344 (41.5) | 61 (46.2) | 1122 (68.3) |
| 3+ | 4 (2.5) | 7 (4.0) | 21 (2.5) | 629 (10.1) | 13 (2.6) | 58 (7.0) | 4 (3.0) | 242 (14.7) |
| Readmitted within 28 days, n (%) a | 24 (15.6) | 36 (24.2) | 284 (35.2) | 2013 (36.8) | 114 (24.0) | 219 (28.9) | 41 (31.5) | 458 (31.0) |
| Died in hospital, n (%) | 8 (4.9) | 23 (13.4) | 25 (3.0) | 776 (12.4) | 25 (5.0) | 73 (8.8) | 2 (1.5) | 164 (10.0) |
| Length of stay, median days (IQR) a | 7.2 (5.1–13.7) | 14.8 (7.0–33.2) | 5.8 (3.0–9.5) | 20.1 (8.7–34.3) | 5.2 (2.9–9.2) | 12.9 (6.9–31.3) | 4.5 (1.9–7.9) | 8.2 (3.7–18.2) |
DRG, Diagnosis‐related group; ED, emergency department; IQR, interquartile range.
Excluding those who died in hospital
There was no evidence of a change in SJS/TEN incidence in general over the study period, relative to the control group (Table 3). A small increasing trend was observed for marrow‐related ADR incidence, relative to controls. This was borderline significant for principally diagnosed cases (P = 0.05) and significant for additionally diagnosed cases (P < 0.001). For principally diagnosed liver‐related ADRs, there was a significant change of 12% per year [95% confidence interval (CI) 5%, 20%] relative to an almost flat trend in the control group. Kidney‐related ADRs showed the strongest increase in relative incidence, with principally diagnosed cases increasing by 62% per year (95% CI 32%, 97%) over and above the control group trend and additionally diagnosed cases increasing by 27% (95% CI 7%, 52%) relative to controls.
Table 3.
Estimated annual change in incidence relative to control group
| Type of reaction | Diagnosis type | N | Annual change relative to control group, rate ratio (95 % CI) |
|---|---|---|---|
| Skin | Principal | 162 | 0.95 (0.90,1.01) |
| Additional | 172 | 0.99 (0.90,1.09) | |
| Blood/marrow | Principal | 831 | 1.03 (1.00,1.06) |
| Additional | 6254 | 1.04 (1.02,1.05) | |
| Liver | Principal | 500 | 1.12 (1.05,1.20) |
| Additional | 830 | 1.04 (0.99,1.09) | |
| Kidneys | Principal | 132 | 1.62 (1.32,1.97) |
| Additional | 1643 | 1.27 (1.07,1.52) |
CI, confidence interval
On the whole, the impact of ADRs was greater among additionally diagnosed than principally diagnosed cases (Table 4), particularly for LOS. When the ADR involved the principal diagnosis, only marrow‐related cases showed significantly higher odds of readmission [odds ratio (OR) 1.74, 95% CI 1.46, 2.07) compared with matched controls. For additionally diagnosed ADRs, all except SJS/TEN showed a significantly increased odds of readmission relative to controls, with marrow‐related ADRs having the highest (OR 1.77, 95% CI 1.66, 1.89). The odds of death in hospital was significantly increased for ADRs listed as additional diagnoses (ORs between 1.22 and 3.49), with SJS/TEN showing the greatest increase in odds (OR 3.49, 95% CI 1.89, 6.42). When marrow‐ or kidney‐related ADRs were the primary diagnosis, the odds of death in hospital were reduced relative to controls (marrow OR 0.76, 95% CI 0.48, 1.18; kidney OR 0.30, 95% CI 0.07, 1.34), although neither effect was significant. LOS increased significantly for all ADRs, with increases ranging from 22% to 328% compared with controls.
Table 4.
Impact of adverse drug reactions on readmission within 28 days, in‐hospital mortality and length of stay
| 28‐day readmission | Death in hospital | Length of stay | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Type of reaction | Diagnosis field | Odds ratio | 95% CI | P‐value | Odds ratio | 95% CI | P‐value | Mean ratio | 95% CI | P‐value |
| Skin | Principal | 0.78 | 0.48, 1.26 | 0.30 | 2.02 | 0.81, 5.04 | 0.13 | 2.73 | 2.34, 3.18 | <0.001 |
| Additional | 1.09 | 0.70, 1.70 | 0.70 | 3.49 | 1.89, 6.42 | <0.001 | 3.89 | 3.15, 4.81 | <0.001 | |
| Blood/marrow | Principal | 1.74 | 1.46, 2.07 | <0.001 | 0.76 | 0.48, 1.18 | 0.22 | 1.63 | 1.50, 1.77 | <0.001 |
| Additional | 1.77 | 1.66, 1.89 | <0.001 | 2.30 | 2.08, 2.54 | <0.001 | 4.28 | 4.14, 4.43 | <0.001 | |
| Liver | Principal | 1.14 | 0.89, 1.46 | 0.30 | 1.59 | 0.97, 2.60 | 0.06 | 1.64 | 1.49, 1.82 | <0.001 |
| Additional | 1.32 | 1.10, 1.60 | 0.004 | 1.97 | 1.45, 2.69 | <0.001 | 3.61 | 3.30, 3.94 | <0.001 | |
| Kidneys | Principal | 1.31 | 0.84, 2.04 | 0.23 | 0.30 | 0.07, 1.34 | 0.11 | 1.22 | 1.01, 1.47 | 0.04 |
| Additional | 1.14 | 1.00, 1.30 | 0.05 | 1.24 | 1.03, 1.51 | 0.03 | 1.52 | 1.42, 1.63 | <0.001 | |
CI, confidence interval
Discussion
The present study provided a comprehensive, population‐level assessment of the impact of, and relative trends in, the incidence of serious ADRs. We found evidence of significant relative increases in incidence over time for several ADRs, while there was no evidence of a decrease for any ADR types. There was also a significant and consistent augmentation of LOS and the odds of both readmission and death for most ADR groups, particularly those that were additionally diagnosed. Principally diagnosed marrow‐ and kidney‐related ADRs were associated with a reduced odds of death in hospital.
There was no evidence of a change in SJS/TEN incidence over time relative to the control group. By contrast, all other estimates suggested an increase in ADR cases relative to controls, with a higher gradient than the respective comparison groups. For both definitions of marrow‐related ADRs, there was some evidence of a small relative increase over the study period. Closer examination of kidney‐related ADRs revealed that the significant trend in principally diagnosed cases was influenced by a pronounced jump in cases from around 2007–08 onwards, from fewer than 10 cases per year to more than 30, something that did not also occur among controls. For additionally diagnosed kidney‐related ADRs, a marked drop in the control series between 2000–01 and 2002–03 seems to have had a strong influence on the significance of the relative trend estimate. Reanalysing the data without the initial 3 years resulted in a nonsignificant interaction term (P = 0.97), suggesting no evidence of a relative trend since 2003–04. There was a significant relative change in the incidence of principally diagnosed liver‐related ADRs, driven by the case series slightly increasing relative to a decreasing control series. The fact that significant trends have been observed, even after accounting for background influences through the use of control series, highlights areas requiring further investigation, particularly given the scarcity of recent studies on ADR incidence trends.
Some of the commonly implicated drugs in Table 5 concur with those known to be associated with certain ADRs – e.g. NSAIDs and analgesic nephropathy. However, these must be interpreted with caution as individual external cause codes cannot be linked to specific diagnosis codes, as described above. External cause codes involving ‘other’ or ‘unspecified’ classifications feature frequently in Table 5 across several ADRs, with drug or medicament, unspecified being the most frequent code for agranulocytosis. This indicates uncertainty in determining the cause of the ADR at the stage of coding, rather than issues with the coding itself, and this underscores the challenge of establishing the causality of ADRs, as discussed above.
Table 5.
The five most commonly implicated drugs by principally diagnosed adverse drug reaction type
| ADR | ICD‐10 code | Description | % |
|---|---|---|---|
| SJS/TEN | Y46.2 | Hydantoin derivatives | 14.6 |
| Y46.4 | Iminostilbenes | 13.9 | |
| Y40.0 | Penicillins | 12.7 | |
| Y54.8 | Agents affecting uric acid metabolism | 12.0 | |
| Y41.0 | Sulfonamides | 9.5 | |
| Aplastic anaemia | Y59.3 | Immunoglobulin | 24.2 |
| Y41.5 | Antiviral drugs | 10.6 | |
| Y57.9 | Drug or medicament, unspecified | 9.8 | |
| Y44.1 | Vitamin B12, folic acid and other antimegaloblastic–anaemic preparations | 5.3 | |
| Y54.8 | Agents affecting uric acid metabolism | 4.5 | |
| Thrombocytopenia | Y41.2 | Antimalarial agents and drugs acting on other blood protozoa | 34.9 |
| Y44.2 | Anticoagulants | 11.2 | |
| Y57.9 | Drug or medicament, unspecified | 8.4 | |
| Y44.4 | Antithrombotic drugs [platelet‐aggregation inhibitors] | 3.3 | |
| Y41.5 | Antiviral drugs | 3.3 | |
| Agranulocytosis | Y57.9 | Drug or medicament, unspecified | 11.6 |
| Y41.0 | Sulfonamides | 6.6 | |
| Y42.2 | Antithyroid drugs | 6.6 | |
| Y40.0 | Penicillins | 6.0 | |
| Y40.8 | Other systemic antibiotics | 5.2 | |
| Toxic liver disease | Y40.0 | Penicillins | 14.2 |
| Y57.9 | Drug or medicament, unspecified | 8.4 | |
| Y52.6 | Antihyperlipidaemic and antiarteriosclerotic drugs | 6.2 | |
| Y45.5 | 4‐Aminophenol derivatives | 4.8 | |
| Y43.1 | Antineoplastic antimetabolites | 4.6 | |
| Analgesic nephropathy | Y45.8 | Other analgesics and antipyretics | 39.4 |
| Y45.3 | Other nonsteroidal anti‐inflammatory drugs [NSAID] | 21.2 | |
| Y45.9 | Analgesic, antipyretic and anti‐inflammatory drug, unspecified | 21.2 | |
| Y45.2 | Propionic acid derivatives | 12.1 | |
| Y45.1 | Salicylates | 6.1 | |
| Other nephropathy | Y43.4 | Immunosuppressive agents | 15.5 |
| Y57.5 | X‐ray contrast media | 14.7 | |
| Y54.4 | Loop [high‐ceiling] diuretics | 7.8 | |
| Y52.4 | Angiotensin‐converting‐enzyme inhibitors | 6.9 | |
| Y43.3 | Other antineoplastic drugs | 6.0 |
ICD‐10, International Classification of Diseases–10th revision; SJS/TEN, Stevens–Johnson syndrome and toxic epidermal necrolysis
Whether additionally diagnosed ADRs were pre‐existing or occurred in hospital, they were effectively comorbid with at least the principal condition. The median number of additional diagnosis codes ranged from three to four among principally diagnosed ADR cases, in comparison with 6.5 to eight among additionally diagnosed cases, indicating considerably higher comorbidity in the latter group. Hence, this may explain most of the consistently augmented impacts of ADRs among additionally diagnosed cases. It is also possible that some conditions coded alongside ADRs were actually caused by the ADR – e.g. aplastic anaemia can result in infection, which is a major cause of death in such cases 40. Among those admitted for a marrow or kidney‐related ADR, the odds of dying in hospital were lower than for matched control patients. Although these effects were not statistically significant, the point estimates were not small and were opposite in direction to all other estimates of mortality. A possible explanation is that marrow‐related conditions often resolve once the offending drug is ceased. Similarly, acute drug‐induced renal injury can often be reversed with discontinuation of the causal medication 17.
Whether an ADR is preventable depends on the definition of preventability and the way an ADR is defined. The criteria used by some authors to define preventable ADRs includes the use of an inappropriate drug for a particular condition; the use of an inappropriate dose, route or frequency; and the occurrence of an avoidable drug interaction 34, 41. However, these could just as easily come under the category of prescribing error and would fall outside the ADR definitions used in many studies. The ADRs we have attempted to capture are largely idiosyncratic and hence difficult to predict – e.g. interstitial nephritis leading to kidney failure after taking omeprazole within the recommended dose range. Prevention of such cases is challenging but a combination of several approaches can contribute to reducing ADR incidence. At a clinical level, using enhanced monitoring of patients when they are in a high risk group or have a history of ADR may help to detect early signs, which may prevent progression to a serious ADR in some cases. There is also a role for computerized decision support in monitoring both drug dosage and clinical warning signs 42. While at a broader level, adequate surveillance systems for monitoring ADRs and associated drugs can minimize the possibility that dangerous drugs remain on the market. To this end, ICD‐based approaches, as used in the present study, can contribute timely population‐level epidemiological evidence to the surveillance of drug‐induced conditions. Given the data limitations encountered in the present analysis, these techniques could be improved through collection of more reliable and detailed data, including drug exposure information. There is also a role for emerging machine learning methods for predicting a drug's potential to cause an ADR 43, 44. These techniques use information on a drug's molecular structure and can help to identify high‐risk drugs before they reach the market.
Limitations
The use of ICD‐10 codes to identify potential ADRs has a number of limitations. Firstly, it relies on the accuracy of coding, particularly in the way putative drug‐induced conditions are determined. The recording of ICD codes in EDs can be poor 18, and in NSW is done by medical staff without training in clinical coding 45. By contrast, coding in the APDC is performed by trained clinical information managers, and coding quality is monitored at the level both of individual hospitals and the Ministry of Health 46. However, this does not give any concrete indication of the level of coding accuracy. Moreover, our ICD‐10 based definitions of ADRs have not, as yet, been validated, so the sensitivity and specificity are unknown. While validation is important for such methods, it is challenging to carry out with jurisdiction‐wide hospital data, and this is reflected by the general lack of validation among studies applying ICD‐based algorithms to administrative data 31.
Further uncertainty is introduced into case definitions by the inability to associate external causes with their corresponding diagnoses. When diagnosis codes are specifically for drug‐induced conditions (i.e. D61.1, K71, N14.0), including external cause codes in the definition may result in missed cases. For the other diagnoses, having to include all external cause fields in the case definition increases the possibility of including false positives.
Misclassification resulting from the ICD‐based definitions, whether false positive or negative, would bias assessments of impact on LOS, readmission and mortality towards the null, suggesting that the result on ADR impact reported herein are, if anything, underestimates. The matching of cases with controls used a set of available variables that were considered to be potential confounders of the relationship between ADR status and outcomes. While other factors, such as the number of medications or a measure of patient frailty, may also be confounders, such variables were not available. In addition, the use of a relatively crude measure of comorbidity does not capture the severity of individual diagnosed conditions. Although not all potential confounders could be accounted for by matching, it has been noted that a random‐order nearest available matching approach, as used in the present study, removes a large proportion of bias in estimated effects 47. Residual bias not accounted for by matching is mitigated by repeating the random selection of matched controls many times and combining the estimates from each iteration into a single result.
Conclusions
The present study has provided new insights into the relative incidence and impacts of serious ADRs at a population level. We have shown the potential for interrogating population‐level hospital data to generate important information on rare events, including serious ADRs. We have also demonstrated the use of statistical techniques to address the analytical challenges of such studies. Although these ADR cases are relatively rare, they represent a considerable cost both to patients and the health system. Furthermore, when these conditions are concurrent with other diagnoses, their impact is all the higher. Reducing the incidence of idiosyncratic ADRs is a challenging task but there are strategies related to both patient care and drug monitoring, both pre‐ and post‐commercial release, that provide potential ways forward.
Competing Interests
All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and report no support from any organization for the submitted work, no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.
The authors would like to thank NSW Health for providing the hospital admissions data, and Hanna Noworytko, Julie Rust and Anne Elsworthy for their advice on the data and the finer points of ICD‐10‐AM coding. This work was supported by Australian NHMRC grants 1055498 and 1054146.
Contributors
All authors contributed to the conceptualization of the study. S.W. and R.D. developed the ICD‐based definitions. B.G. obtained the data and provided advice on the analysis. S.W. performed the analysis and led the drafting of the manuscript. All other authors contributed to the drafting process. All authors read and approved the final version of the manuscript.
Walter, S. R. , Day, R. O. , Gallego, B. , and Westbrook, J. I. (2017) The impact of serious adverse drug reactions: a population‐based study of a decade of hospital admissions in New South Wales, Australia. Br J Clin Pharmacol, 83: 416–426. doi: 10.1111/bcp.13124.
References
- 1. Beijerm HJM, de Blaey CJ. Hospitalisations caused by adverse drug reactions (ADR): a meta‐analysis of observational studies. Pham World Sci 2008; 24: 46–54. [DOI] [PubMed] [Google Scholar]
- 2. Kongkaew C, Noyce PR, Ashcroft DM. Hospital admissions associated with adverse drug reactions: A systematic review of prospective observational studies. Ann Pharmacother 2008; 42: 1017–1025. [DOI] [PubMed] [Google Scholar]
- 3. Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients. JAMA 1998; 279: 1200–1205. [DOI] [PubMed] [Google Scholar]
- 4. Davies EC, Green CF, Taylor S, Williamson PR, Mottram DR, Pirmohamed M. Adverse drug reactions in hospital in‐patients: A prospective analysis of 3695 patient‐episodes. PLoS One 2009; 4: e4439. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Moore N, Lecointre D, Noblet C, Mabille M. Frequency and cost of serious adverse drug reactions in a department of general medicine. Br J Clin Pharmacol 1998; 45: 301–308. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. van der Hooft CS, Sturkenboom MCJM, van Grootheest K, Kingma HJ, Stricker BHC. Adverse drug reaction‐related hospitalisations: a nationwide study in the Netherlands. Drug Saf 2006; 29: 161–168. [DOI] [PubMed] [Google Scholar]
- 7. Harr T, French LE. Toxic epidermal necrolysis and Stevens‐Johnson syndrome. Orphanet J Rare Dis 2010; 5. doi:10.1186/1750-1172-5-39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Andrade RJ, Lucena MI, Fernández MC, Pelaez G, Pachkoria K, García–Ruiz E, et al. Drug induced liver injury: an analysis of 461 incidences submitted to the Spanish Registry over a 10‐year period. Gastroenterology 2005; 129: 512–521. [DOI] [PubMed] [Google Scholar]
- 9. Chalasani N, Fontana RJ, Bonkovsky HL, Watkins PB, Davern T, Serrano J, et al. Causes, clinical features and outcomes from a prospective study of drug‐induced liver injury in the United States. Gastroenterology 2008; 135: 1924–1934. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Finn WF. The clinical and renal consequences of contrast‐induced nephropathy. Nephrol Dial Transplant 2006; 21: i2–i10. [DOI] [PubMed] [Google Scholar]
- 11. Fritsch PO, Sidoroff A. Drug‐induced Stevens‐Johnson syndrome/toxic epidermal necrolysis. Am J Clin Dermatol 2000; 1: 349–360. [DOI] [PubMed] [Google Scholar]
- 12. Barvaliya M, Sanmukhani J, Patel T, Paliwal N, Shah H, Tripathi C. Drug‐induced Stevens‐Johnson syndrome (SJS), toxic epidermal necrolysis (TEN), and SJS‐TEN overlap: a multicentric retrospective study. J Postgrad Med 2011; 57: 115–119. [DOI] [PubMed] [Google Scholar]
- 13. Montané E, Ibáñez L, Vidal X, Ballarín E, Puig R, García N, et al. Epidemiology of aplastic anemia: a prospective multicenter study. Haematologica 2008; 93: 518–523. [DOI] [PubMed] [Google Scholar]
- 14. Ibáñez L, Vidal X, Ballarín E, Laporte JR. Population‐based drug‐induced agranulocytosis. Arch Intern Med 2005; 165: 869–874. [DOI] [PubMed] [Google Scholar]
- 15. Kenney B, Stack G. Drug‐induced thrombocytopenia. Arch Pathol Lab Med 2009; 133: 309–314. [DOI] [PubMed] [Google Scholar]
- 16. Bjornsson ES, Bergmann OM, Bjornsson HK, Kvaran RB, Olafsson F. Incidence, presentation and, outcomes in patients with drug‐induced liver injury in the general population of Iceland. Gastroenterology 2013; 144: 1419–1425. [DOI] [PubMed] [Google Scholar]
- 17. Choudhury D, Ahmed Z. Drug‐associated renal dysfunction and injury. Nat Clin Pract Nephrol 2006; 2: 80–91. [DOI] [PubMed] [Google Scholar]
- 18. Hohl CM, Kuramoto L, Yu E, Rogula B, Stausberg J, Sololev B. Evaluating adverse drug event reporting in administrative data from emergency departments: a validation study. BMC Health Serv Res 2013; 13: 473. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Lawrence G, Dinh I, Taylor L. The Centre for Health Record Linkage: a new resource for health services research and evaluation. Health Inform Manag J 2008; 37: 60–62. [DOI] [PubMed] [Google Scholar]
- 20. World Health Organization . International drug monitoring – the role of the national centres. Technical report series No. 498, Geneva, 1972. [PubMed]
- 21. Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis and management. Lancet 2000; 356: 1255–1259. [DOI] [PubMed] [Google Scholar]
- 22. World Health Organization . Glossary of terms in pharmacovigilance, 2011. [online]. Available at: www.who‐umc.org/Graphics/24729.pdf (last accessed 7 October 2015).
- 23. Naranjo CA, Busto U, Sellers EM, Sandor P, Ruiz I, Roberts EA, et al. A method for estimating the probability of adverse drug reactions. Clin Pharamacol Ther 1981; 30: 239–245. [DOI] [PubMed] [Google Scholar]
- 24. Jha AK, Kuperman GJ, Teich JM, Leape L, Shea B, Rittenberg E, et al. Identifying adverse drug events: development of a computer‐based monitor and comparison with chart review and simulated voluntary report. J Am Med Inform Assoc 1998; 5: 305–314. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Localio AR, Weaver SL, Landis JR, Lawthers AG, Brenhan TA, Hebert L, et al. Identifying adverse events caused by medical care: degree of physician agreement in a retrospective chart review. Ann Intern Med 1996; 125: 457–464. [DOI] [PubMed] [Google Scholar]
- 26. Agbabiaka TB, Savovic J, Ernst E. Methods for causality assessment of adverse drug events. A systematic review. Drug Saf 2008; 31: 21–37. [DOI] [PubMed] [Google Scholar]
- 27. Lanctôt KL, Naranjo CA. Comparison of the Bayesian approach and a simple algorithm for assessment of adverse drug events. Clin Pharmacol Ther 1995; 58: 692–698. [DOI] [PubMed] [Google Scholar]
- 28. Overby CL, Pathak J, Gottesman O, Haerian K, Perotte A, Murphy S, et al. A collaborative approach to developing an electronic health record phenotyping algorithm for drug‐induced liver injury. J Am Med Inform Assoc 2013; 20: e243–52. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Jackson TJ, Michel JL, Roberts RF, Jorm CM, Wakefiled JG. A classification of hospital‐acquired diagnoses for use with routine hospital data. Med J Aust 2009; 191: 544–548. [DOI] [PubMed] [Google Scholar]
- 30. Stausberg J, Hasford J. Drug‐related admissions and hospital‐acquired adverse drug events in Germany: a longitudinal analysis from 2003 to 2007 of ICD‐10‐coded routine data. BMC Health Serv Res 2011; 11: 134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Hohl CM, Karpov A, Reddekopp L, Stausberg J. ICD‐10 codes used to identify adverse drug events in administrative data: a systematic review. J Am Med Inform Assoc 2014; 21: 547–557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Utz M, Johnson T, Halech R. A review of the classification of hospital‐acquired diagnoses (CHADx). Technical report No. 12. Queensland Health, Brisbane, 2012.
- 33. Pirmohamed M, James S, Meakin S, Green C, Scott AK, Walley TJ, et al. Adverse drug reactions as cause of admission to hospital prospective analysis of 18 820 patients. BMJ 2004; 329: 15–19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Gholami K, Shalviri G. Factors associated with preventability, predictability, and severity of adverse drug reactions. Ann Pharmacother 1999; 33: 236–240. [DOI] [PubMed] [Google Scholar]
- 35. Australian Institute of Health and Welfare . Australian Hospital Statistics 2011‐12. Health Services Series No. 50, Cat. No. 134. Canberra, 2013.
- 36. Liang KY, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika 1986; 73: 13–22. [Google Scholar]
- 37. Charlson ME, Pompei P, Ales KL, Makenzie CR. A new method of classifying co‐morbidity in longitudinal studies – development and validation. J Chronic Dis 1987; 40: 373–383. [DOI] [PubMed] [Google Scholar]
- 38. Deyo RA, Cherkin DC, Cio MA. Adapting a clinical comorbidity index for use with ICD‐9‐CM administrative databases. J Clin Epidemiol 1992; 45: 613–619. [DOI] [PubMed] [Google Scholar]
- 39. Rubin DB. Multiple Imputation for Nonresponse in Surveys, 1st edn. New York, NY: John Wiley & Sons, 1987. [Google Scholar]
- 40. Torres HA, Bodey GP, Rolston KV, Kantarjian HM, Raad II, Kontoyiannis DP. Infections in patients with aplastic anemia: experience at a tertiary care cancer center. Cancer 2003; 98: 86–93. [DOI] [PubMed] [Google Scholar]
- 41. McDonnell PJ, Jacobs MR. Hospital admissions resulting from preventable adverse drug reactions. Ann Pharmacother 2002; 36: 1331–1336. [DOI] [PubMed] [Google Scholar]
- 42. Coleman JJ, Ferner RE, Evans SJW. Monitoring for adverse drug reactions. Br J Clin Pharmacol 2006; 61: 371–378. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Kuang Q, Wang M, Li R, Dong Y, Li Y, Li M. A systematic investigation of computational models for predicting adverse drug reactions (ADRs). PLoS One 2014; 9: e105889. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Xu Y, Dai Z, Chen F, Gao S, Pei J, Lai L. Deep learning for drug‐induced liver injury. J Chem Inf Model 2015; 55: 2085–2093. [DOI] [PubMed] [Google Scholar]
- 45. New South Wales Centre for Health Record Linkage . Emergency Department Data Collection: data dictionary, 2015. [online]. Available at http://www.cherel.org.au/media/31700/eddc_data_dictionary_sep2015‐a.docx (last accessed 17 August 2016).
- 46. Australian Institute of Health and Welfare . Australian hospital statistics 2012‐13. Health services Series No. 54, Cat. No. HSE 145. Canberra, 2014.
- 47. Rubin DB. Matching to remove bias in observational studies. Biometrics 1973; 29: 159–183. [Google Scholar]
