Skip to main content
NCBI home page
Nephrology Dialysis Transplantation logoLink to Nephrology Dialysis Transplantation
. 2019 Apr 26;34(7):1145–1154. doi: 10.1093/ndt/gfz062

Nonsteroidal anti-inflammatory drug use and risk of acute kidney injury and hyperkalemia in older adults: a population-based study

Danielle M Nash 1,2,✉, Maureen Markle-Reid 1,3, Kenneth S Brimble 4,5, Eric McArthur 2, Pavel S Roshanov 5, Jeffrey C Fink 6, Matthew A Weir 2,7, Amit X Garg 2,7
PMCID: PMC6603365  PMID: 31264694

Abstract

Background

Clinical guidelines caution against nonsteroidal anti-inflammatory drug (NSAID) use in older adults. The study objective was to quantify the 30-day risk of acute kidney injury (AKI) and hyperkalemia in older adults after NSAID initiation and to develop a model to predict these outcomes.

Methods

We conducted a population-based retrospective cohort study in Ontario, Canada from 2007 to 2015 of patients ≥66 years. We matched 46 107 new NSAID users with 46 107 nonusers with similar baseline health. The primary outcome was 30-day risk of AKI and secondary outcomes were hyperkalemia and all-cause mortality.

Results

NSAID use versus nonuse was associated with a higher 30-day risk of AKI {380 [0.82%] versus 272 [0.59%]; odds ratio (OR) 1.41 [95% confidence interval (CI) 1.20–1.65]} and hyperkalemia [184 (0.40%) versus 123 (0.27%); OR 1.50 (95% CI 1.20–1.89); risk difference 0.23% (95% CI 0.13–0.34)]. There was no association between NSAID use and all-cause mortality. A prediction model incorporated six predictors of AKI or hyperkalemia: older age, male gender, lower baseline estimated glomerular filtration rate, higher baseline serum potassium, angiotensin-converting enzyme inhibitor or angiotensin receptor blocker use or diuretic use. This model had moderate discrimination [C-statistic 0.72 (95% CI 0.70–0.74)] and good calibration.

Conclusions

In older adults, new NSAID use compared with nonuse was associated with a higher 30-day risk of AKI and hyperkalemia but not all-cause mortality. Prescription NSAID use among many older adults may be safe, but providers should use caution and assess individual risk.

Keywords: acute kidney injury, hyperkalemia, nonsteroidal anti-inflammatory drugs, prediction model

ADDITIONAL CONTENT

An author video to accompany this article is available at: https://academic.oup.com/ndt/pages/author_videos.

INTRODUCTION

Nonsteroidal anti-inflammatory drugs (NSAIDs) are commonly used to treat pain and inflammation. Health Canada along with many clinical guidelines caution against NSAID use in older adults or patients with chronic kidney disease (CKD) due to the risk of adverse events, including acute kidney injury (AKI) (Supplementary data, Table S1) [1–11]. Over the last decade, 43% of older adults received at least one NSAID prescription in Ontario, Canada (Supplementary data, Methods 1). Furthermore, from 2006 to 2011, 16% of patients with CKD in Ontario received at least one NSAID prescription for >14 days [12]. While there is evidence from a meta-analysis of case–control studies that NSAID use is associated with a higher risk of AKI [13], studies that have assessed the risk of hyperkalemia have yielded conflicting results (Supplementary data, Table S2) [14–16]. Therefore we designed this large population-based cohort study to quantify the 30-day risk of AKI and hyperkalemia among older adults in Ontario, Canada who were dispensed a >14-day supply for NSAIDs compared with older adults without prescription NSAID use. Because of the utility of risk prediction tools for NSAID-induced gastrointestinal complications [17–20], we also sought to develop and internally validate a prediction model to quantify a patient’s 30-day risk of developing AKI or hyperkalemia after NSAID initiation based on patient characteristics at the time of prescribing.

MATERIALS AND METHODS

Study design and research setting

Ontario has >13 million residents, 17% of whom are ≥65 years of age [21]. Health care services in Ontario are publically funded, with the exception of outpatient medications, which are only funded for individuals ≥65 years of age, and other special populations [22]. These health care encounters are recorded in administrative databases held at the Institute for Clinical Evaluative Sciences (ICES). All databases are linked using unique, encoded identifiers.

We conducted a population-based, retrospective cohort study using healthcare data at ICES. The use of ICES data in this project was authorized under Section 45 of Ontario’s Personal Health Information Protection Act, which does not require research ethics board review. We followed reporting guidelines for observational and prediction studies (Supplementary data, Tables S3 and S4) [23, 24].

Data sources

We used the Ontario Drug Benefit database to identify prescriptions for individuals ≥65 years of age. This database contains accurate records of all dispensed outpatient prescriptions [25]. We used the Ontario Laboratories Information System database to identify laboratory values and estimated glomerular filtration rates (eGFRs) using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation [26]. We had no information on race and assumed all patients to be nonblack for the CKD-EPI equation (<5% of the Ontario population is of black race) [27]. We used seven additional linked databases held at ICES to ascertain information on hospitalizations and emergency department visits (Canadian Institute for Health Information’s Discharge Abstract Database and National Ambulatory Care Reporting System), physician billings (Ontario Health Insurance Plan database), prescribing physicians (ICES Physician database, Client Agency Program Enrolment), treatment for end-stage kidney disease (the Ontario portion of the Canadian Organ Replacement Register) and demographics (Registered Persons Database).

We used the 10th edition of the Canadian Modified International Classification of Disease system to define comorbidities and the Canadian Classification for Health Interventions to define health care procedures. We also used the Johns Hopkins Aggregated Diagnosis Groups and the Expanded Diagnosis Clusters to assess baseline comorbidity (Johns Hopkins ACG System version 10.0) [28].

Our data sources were complete for all study variables except for NSAID prescriber data (15% missing), rural status (<0.005% missing) and income quintile (<0.5% missing). Emigration from Ontario is very low (0.1%/year) and was the only reason for lost study follow-up [29].

Cohort assembly and exposure categorization

We identified geographical areas across time where residents would likely visit a hospital with linked laboratory data (referred to as the laboratory catchment area). We included Ontarians who resided within this catchment area between 1 April 2007 and 30 September 2015. In the exposed group, we identified patients with an NSAID prescription filled between 1 April 2007 and 31 August 2015 with a >14-day supply. For patients with multiple eligible NSAID prescriptions after exclusions, we restricted use to the first one. See Supplementary data, Table S5 for a list of eligible drug names. The date of NSAID dispensing was referred to as the index date, also called the cohort entry date. For patients in our comparison group without an NSAID prescription, we randomly assigned an index date based on the distribution of index dates in the NSAID user group.

We excluded the following patients: (i) those in their first year of provincial drug coverage (between 65 and 66 years of age) to avoid incomplete medication history; (ii) those with an NSAID prescription in the prior 6 months to ensure new users only; (iii) those discharged from a hospital in the 2 days prior to the index date, to ensure new outpatient prescriptions since patients who initiate treatment in the hospital fill ongoing prescriptions on the hospital discharge date or the day after; (iv) those without at least one outpatient value for both serum creatinine and serum potassium in the prior year, to assess baseline kidney function and potassium levels; (v) those with baseline serum potassium values >5.5 mEq/L (5.5 mmol/L) or potassium binder prescriptions in the past 6 months; (vi) those with a baseline eGFR >150 mL/min/1.73 m2, which was likely a data error; (vii) those not in the laboratory catchment area on their index date, to reliably assess laboratory results in follow-up; and (viii) patients with end-stage kidney disease as an outcome of AKI. For patients in the nonuser group, we excluded anyone without at least one health care encounter in our physician claims or drug benefit databases in the past 30 days, to ensure that patients were accessing the Ontario health care system.

Outcomes

Our primary outcome was AKI, defined by the 2012 Kidney Disease: Improving Global Outcomes (KDIGO) thresholds: compared with baseline, a serum creatinine increase ≥50% or an absolute increase of at least 0.3 mg/dL (26.5 µmol/L) [30]. We selected an outpatient serum creatinine value within the past year that was closest to the index date. We compared this baseline value to the highest serum creatinine value in the following 30 days, whether the measurement was done in the community, emergency department or during a hospitalization. In additional analyses, we assessed more severe AKI, defined according to KDIGO staging thresholds (Supplementary data, Table S6) [30]. We assessed all outcomes in the 30 days following the index date since AKI onset generally occurs within 2 weeks of NSAID initiation [31].

Our secondary outcomes were hyperkalemia, all-cause mortality and a composite of AKI or hyperkalemia. We defined hyperkalemia as a serum potassium value in an outpatient, emergency department or inpatient setting that was ≥5.5 mEq/L. In additional analyses, we assessed outcomes of more severe hyperkalemia, defined as values of 6.0 and ≥6.5 mEq/L.

Statistical analyses

We conducted all analyses using SAS version 9.4 (SAS Institute, Cary, NC, USA). We compared baseline characteristics between NSAID users and nonusers using standardized mean differences, where a difference ≥10% was considered significant [32]. We calculated a propensity score for the probability of receiving an NSAID prescription using a multivariable logistic regression model that incorporated >150 baseline characteristics (including indications for NSAID use and risk factors for AKI and hyperkalemia; Supplementary data, Table S7). We matched NSAID users to nonusers 1:1 using greedy matching without replacement [33]. Matching criteria included eGFR (±5 mL/min/1.73 m2), serum potassium (±0.5 mEq/L), angiotensin-converting enzyme inhibitor (ACEi) or angiotensin receptor blocker (ARB) use, diuretic use [categorized as (i) potassium sparing or (ii) loop or thiazide] and the logit of the propensity score (±0.2 standard deviation).

We performed conditional logistic regression analyses to estimate odds ratios (ORs) and 95% confidence intervals (CIs). Given the low incidences observed, we approximated all ORs as relative risks. We estimated the risk difference between the groups and the number needed to harm [34]. This latter measure indicates how many patients need to receive an NSAID prescription for one patient to have an adverse event who would not have experienced this event in the absence of an NSAID prescription. For AKI, we performed subgroup analyses on baseline eGFR, ACEi/ARB use, diuretic use and NSAID dose measured as the percentage of the maximum daily dose (for this subgroup, nonusers followed their matched users). For hyperkalemia, we performed the same subgroup analyses with the addition of baseline serum potassium. We performed an additional analysis for the primary AKI outcome where we restricted the analysis to patients who received at least one serum creatinine test within the 30-day follow-up. We did not account for matching in this additional analysis since the matched pairs were separated. We interpreted two-tailed P-values <0.05 as statistically significant in all analyses.

We developed a prediction model of 30-day AKI or hyperkalemia risk among all NSAID users and followed relevant reporting guidelines (Supplementary data, Table S4) [24]. For the potential predictors, we identified the following variables a priori that are easily ascertained in practice and were associated with a higher risk of AKI or hyperkalemia in the literature: older age [35–37], female sex [36, 38], lower baseline eGFR [35, 36, 38], higher baseline serum potassium concentration [39], higher NSAID dose [40, 41], ACEi/ARB use [35, 42] and use of loop, potassium-sparing or thiazide-type diuretics [35, 37, 42]. We kept continuous variables continuous and assumed that their relationship with the outcome was linear.

We performed stepwise multivariable logistic regression using a P < 0.05 threshold to retain variables. We assessed model discrimination using the C-statistic, which describes the ability of our model to assign higher predicted risks to those with the outcome versus those without [43]. We assessed model calibration by plotting the predicted probabilities versus the observed risk of the outcome using locally weighted scatterplot smoothing [44]. We validated the model internally using 200 bootstrap samples selected with replacement from the NSAID users [45]. We repeated stepwise logistic regression for each of these 200 samples and used Harrell’s optimism for the C-statistic to estimate how the model might perform in new data [45]. Finally, we created an online calculator for ease of use.

RESULTS

See Figure 1 for cohort assembly. We identified 9 131 660 individuals in the laboratory catchment area, where 428 825 received at least one NSAID prescription for >14 days. After exclusions, we had 61 219 NSAID users and 156 589 nonusers. We matched 46 107 NSAID users to 46 107 nonusers and matched pairs were similar across >150 baseline characteristics (see Table 1 for select baseline characteristics and Supplementary data, Table S7 for full baseline table). The mean age was 74 years and 58% were women.

FIGURE 1.

FIGURE 1

Open in a new tab

Cohort assembly for patients in the NSAID user group and the matched nonuser group. Exclusions 11 and 12 were only applied to the nonuser group. N/A, not applicable.

Table 1.

Select baseline characteristics of the cohort before and after matching

Variable Baselines before matching
 Baselines after matching
NSAID users (n = 61 219) Nonusers (n = 156 589) Standard difference (%) NSAID users (n = 46 107) Nonusers (n = 46 107) Standard difference (%)
Demographics
 Age (years), mean (SD) 74.1 (6.4) 76.0 (8.1) 27 74.2 (6.5) 74.2 (7.1) 1
 Female sex, n (%) 36 016 (58.8) 84 355 (53.9) 10 26 654 (57.8) 26 552 (57.6) 0
 Rural status, n (%)a 3489 (5.7) 8431 (5.4) 1 2298 (5.0) 2334 (5.1) 0
 Income quintile, n (%)b
  1 (lowest) 11 523 (18.8) 26 116 (16.7) 5 8337 (18.1) 8234 (17.9) 1
  2 12 365 (20.2) 29 469 (18.8) 4 9215 (20.0) 9055 (19.6) 1
  3 12 823 (20.9) 31 049 (19.8) 3 9510 (20.6) 9457 (20.5) 0
  4 12 653 (20.7) 33 721 (21.5) 2 9622 (20.9) 9680 (21.0) 0
  5 (highest) 11 855 (19.4) 36 234 (23.1) 9 9423 (20.4) 9681 (21.0) 1
Index prescription information
 Index NSAID is a COX2 inhibitor, n (%) 30 741 (50.2) 23 208 (50.3)
 Percentage of maximum daily dosage, mean (SD) 61.9 (28.3) 61.5 (28.2)
Prescribing physician characteristics
 Age (years), mean (SD) 55.6 (11.3) 55.8 (11.3)
  Missing, n (%) 3256 (5.3) 2395 (5.2)
 Female sex, n (%) 15 972 (26.1) 12 019 (26.1)
  Missing, n (%) 2620 (4.3) 1862 (4.0)
 Rural status, n (%) 2719 (4.4) 1778 (3.9)
  Missing, n (%) 2574 (4.2) 1827 (4.0)
 Medical specialty, n (%)
  Primary care 52 772 (86.2) 40 040 (86.8)
  Nephrologist 54 (0.1) 38 (0.1)
  Cardiologist 127 (0.2) 107 (0.2)
  Other 5694 (9.3) 4097 (8.9)
  Missing 2572 (4.2) 1825 (4.0)
Health care visits in prior year, n
 Hospitalizations, mean (SD) 0.2 (0.5) 0.2 (0.6) 11 0.1 (0.5) 0.1 (0.5) 0
 Emergency department, mean (SD) 0.5 (1.2) 0.5 (1.1) 3 0.5 (1.2) 0.5 (1.1) 1
 Family physician, mean (SD) 9.2 (7.4) 8.9 (9.4) 4 8.8 (7.2) 8.7 (8.0) 2
 Nephrologist, mean (SD) 0.1 (0.6) 0.2 (1.0) 15 0.1 (0.6) 0.1 (0.5) 0
Comorbidity in past 5 years, n (%)
 Charlson comorbidity score
  0 or no hospitalization 46 909 (76.6) 104 469 (66.7) 22 35 055 (76.0) 34 918 (75.7) 1
  1 6668 (10.9) 19 989 (12.8) 6 5017 (10.9) 5089 (11.0) 0
  2 4426 (7.2) 15 417 (9.8) 9 3442 (7.5) 3493 (7.6) 0
  ≥3 3216 (5.3) 16 714 (10.7) 20 2593 (5.6) 2607 (5.7) 0
 John Hopkins ADG score, mean (SD) 12.4 (4.0) 11.6 (4.2) 18 12.1 (4.0) 12.1 (4.0) 1
 Ischemic heart disease 15 950 (26.1) 49 501 (31.6) 12 12 150 (26.4) 12 265 (26.6) 0
 Congestive heart failure 4105 (6.7) 21 268 (13.6) 23 3178 (6.9) 3269 (7.1) 1
 Ventricular arrhythmia 91 (0.1) 791 (0.5) 7 71 (0.2) 76 (0.2) 0
 Myocardial infarction 2119 (3.5) 8828 (5.6) 10 1634 (3.5) 1686 (3.7) 1
 Peripheral vascular disease 2320 (3.8) 7572 (4.8) 5 1752 (3.8) 1736 (3.8) 0
 Atrial fibrillation 1548 (2.5) 12 489 (8.0) 25 1277 (2.8) 1407 (3.1) 2
 Coronary artery bypass graft 570 (0.9) 2 301 (1.5) 6 450 (1.0) 479 (1.0) 0
 Percutaneous coronary intervention 1360 (2.2) 4957 (3.2) 6 1091 (2.4) 1093 (2.4) 0
 Diabetes 22 177 (36.2) 57 397 (36.7) 1 16 653 (36.1) 16 610 (36.0) 0
 Hypertension 46 410 (75.8) 116 542 (74.4) 3 34 526 (74.9) 34 391 (74.6) 1
 Chronic liver disease 642 (1.0) 2455 (1.6) 5 509 (1.1) 519 (1.1) 0
 Chronic lung disease 14 709 (24.0) 34 723 (22.2) 4 10 473 (22.7) 10 403 (22.6) 0
 Malignancy 7407 (12.1) 21 501 (13.7) 5 5669 (12.3) 5657 (12.3) 0
 Osteoporosis 10 193 (16.7) 23 428 (15.0) 5 7570 (16.4) 7519 (16.3) 0
 Joint disease 34 669 (56.6) 44 363 (28.3) 60 23 064 (50.0) 22 681 (49.2) 2
 Joint disorder 7378 (12.1) 9779 (6.2) 21 4706 (10.2) 4572 (9.9) 1
 Bursitis 18 345 (30.0) 23 268 (14.9) 37 11 902 (25.8) 11 626 (25.2) 1
 Fibromyalgia 4417 (7.2) 5305 (3.4) 17 2686 (5.8) 2514 (5.5) 1
 Fracture 7630 (12.5) 20 413 (13.0) 1 5634 (12.2) 5558 (12.1) 0
 Hip fracture 531 (0.9) 3297 (2.1) 10 436 (0.9) 454 (1.0) 1
 Back pain 30 486 (49.8) 41 164 (26.3) 50 20 383 (44.2) 20 051 (43.5) 1
 Back surgery 790 (1.3) 836 (0.5) 8 454 (1.0) 428 (0.9) 1
 Knee surgery 3829 (6.3) 3755 (2.4) 19 2221 (4.8) 2130 (4.6) 1
 Hip surgery 1612 (2.6) 2604 (1.7) 6 1091 (2.4) 1091 (2.4) 0
 Cerebrovascular disease 4783 (7.8) 20 211 (12.9) 17 3765 (8.2) 3778 (8.2) 0
 Dementia 3729 (6.1) 24 123 (15.4) 30 3142 (6.8) 3106 (6.7) 0
 Migraine 2649 (4.3) 4491 (2.9) 8 1801 (3.9) 1804 (3.9) 0
 Gout 4985 (8.1) 4581 (2.9) 23 2804 (6.1) 2426 (5.3) 3
 Rheumatoid arthritis 3994 (6.5) 4517 (2.9) 17 2436 (5.3) 2320 (5.0) 1
 Osteoarthritis 7031 (11.5) 7861 (5.0) 24 4 245 (9.2) 4163 (9.0) 1
 Sciatica 21 449 (35.0) 26 131 (16.7) 43 13 872 (30.1) 13 479 (29.2) 2
 Pain 32 285 (52.7) 54 717 (34.9) 36 22 548 (48.9) 22 434 (48.7) 0
 Schizophrenia 704 (1.1) 3766 (2.4) 10 576 (1.2) 580 (1.3) 1
 Depression 5161 (8.4) 13 267 (8.5) 0 3729 (8.1) 3766 (8.2) 0
Health care procedures in prior year, n (%)
 Carotid ultrasound 2812 (4.6) 8492 (5.4) 4 2179 (4.7) 2230 (4.8) 0
 Coronary angiogram 1005 (1.6) 3557 (2.3) 5 785 (1.7) 797 (1.7) 0
 Coronary revascularization 415 (0.7) 1731 (1.1) 4 344 (0.7) 359 (0.8) 1
 Echocardiogram 12 466 (20.4) 37 234 (23.8) 8 9526 (20.7) 9649 (20.9) 0
 Holter test 4684 (7.7) 14 820 (9.5) 6 3616 (7.8) 3675 (8.0) 1
 Stress test 8472 (13.8) 19 822 (12.7) 3 6223 (13.5) 6213 (13.5) 0
 Cardiac catheterization 1032 (1.7) 3 699 (2.4) 5 807 (1.8) 819 (1.8) 0
 Back X-ray 29 048 (47.4) 44 677 (28.5) 40 19 685 (42.7) 19 347 (42.0) 1
 At least one albumin:creatinine ratio test 35 803 (58.5) 79 468 (50.7) 16 26 481 (57.4) 26 330 (57.1) 1
Laboratory tests in prior year
 Time from baseline serum creatinine to index date (days), median (IQR) 94 (33–192) 88 (34–179) 94 (32–193) 95 (37–187)
 Baseline serum creatinine value (mg/dL), mean (SD) 0.9 (0.3) 1.0 (0.4) 27 0.9 (0.3) 0.9 (0.3) 0
 Baseline eGFR value (mL/min/1.73 m2), mean (SD) 73.0 (16.0) 69.1 (18.6) 23 73.0 (16.0) 73.0 (16.0) 0
 Baseline potassium value (mEq/L), mean (SD) 4.4 (0.4) 4.4 (0.4) 2 4.4 (0.4) 4.4 (0.4) 0
Prescriptions in prior 120 days, n (%)
 Aspirin 1549 (2.5) 2532 (1.6) 6 1018 (2.2) 959 (2.1) 1
 Tylenol 2939 (4.8) 2879 (1.8) 17 1664 (3.6) 1473 (3.2) 2
 Opiates 12 499 (20.4) 15 521 (9.9) 30 7697 (16.7) 7166 (15.5) 3
 ACEi 15 898 (26.0) 48 917 (31.2) 12 12 190 (26.4) 12 190 (26.4) 0
 ARB 16 618 (27.1) 37 556 (24.0) 7 12 139 (26.3) 12 139 (26.3) 0
 Statin 32 296 (52.8) 84 994 (54.3) 3 24 343 (52.8) 24 269 (52.6) 0
 Diabetes drug 13 159 (21.5) 37 247 (23.8) 5 10 097 (21.9) 10 087 (21.9) 0
 Calcium channel blocker 16 386 (26.8) 45 357 (29.0) 5 12 400 (26.9) 12 244 (26.6) 1
 β-blocker 14 136 (23.1) 48 987 (31.3) 19 10 921 (23.7) 11 045 (24.0) 1
 Proton pump inhibitor 24 196 (39.5) 40 148 (25.6) 30 16 246 (35.2) 15 737 (34.1) 2
 Thiazide diuretic 8871 (14.5) 22 768 (14.5) 0 6461 (14.0) 6466 (14.0) 0
 Loop diuretic 3137 (5.1) 16 247 (10.4) 20 2270 (4.9) 2213 (4.8) 0
 Potassium-sparing diuretic 1877 (3.1) 6000 (3.8) 4 1251 (2.7) 1230 (2.7) 0
Open in a new tab

SI conversion factors: to convert serum creatinine from mg/dL to μmol/L, multiply by 88.4; to convert serum potassium from mEq/L to mmol/L, multiply by 1.0.

a

Missing rural status was categorized as not rural.

b

Missing income quintile was imputed into the third quintile.

ADG, aggregated diagnostic group; IQR, interquartile range.

Primary outcome: AKI

NSAID use was associated with a higher 30-day risk of AKI: 380 (0.82%) versus 272 (0.59%) events, respectively; OR 1.41 (95% CI 1.20–1.65); risk difference 0.23% (95% CI 0.13–0.34%). This association was consistent across AKI stages (Table 2). Baseline eGFR, ACEi or ARB use, diuretic use and NSAID dose did not significantly modify the association between NSAID use and AKI (Figure 2).

Table 2.

30-day primary and secondary outcomes of prescription NSAID users compared with nonusers

Outcome NSAID users, number of events (%) (n = 46 107) Nonusers, number of events (%) (n = 46 107) OR (95% CI) Risk difference (95% CI) Number needed to harm (95% CI)
AKIa,b 380 (0.82) 272 (0.59) 1.41 (1.20–1.65) 0.23 (0.13–0.34) 427 (292–787)
AKI Stage 2 or morec 60 (0.13) 40 (0.09) 1.50 (1.01–2.24) 0.04 (0.00–0.09) 2305 (1164–114  916)
AKI Stage 3d 25 (0.05) 12 (0.03) 2.08 (1.05–4.15) 0.03 (0.002–0.05) 3547 (1848–43 478)
Hyperkalemiae (≥5.5 mEq/L) 184 (0.40) 123 (0.27) 1.50 (1.20–1.89) 0.13 (0.06–0.2) 756 (485–1715)
Hyperkalemia (≥6.0 mEq/L) 38 (0.08) 30 (0.07) 1.27 (0.79–2.04) 0.02 (−0.02–0.05) N/A
Hyperkalemia (≥6.5 mEq/L) 16 (0.03) 30 (0.02) 2.29 (0.94–5.56) 0.02 (0.00–0.04) N/A
Composite of AKIa or hyperkalemia (≥5.5 mEq/L) 510 (1.11) 370 (0.80) 1.39 (1.21–1.59) 0.30 (0.18–0.43) 329 (234–558)
All-cause mortality 66 (0.14) 79 (0.17) 0.83 (0.60–1.16) −0.03 (−0.08–0.02) N/A
AKI additional analysisa,f 380 (4.87) 272 (3.76) 1.31 (1.12–1.53) 1.11 (0.46–1.75) 90 (57–217)
Open in a new tab

SI conversion factor: to convert serum potassium from mEq/L to mmol/L, multiply by 1.0.

a

The primary definition of AKI was a serum creatinine increase ≥50% or an absolute increase of at least 0.3 mg/dL (26.5 µmol/L) compared with baseline.

b

Among all patients with an AKI event, 37% were in the hospital at the time of the serum creatinine test (241/652 patients); 87% of these patients in the hospital had the laboratory test done within the first 3 days of hospital admission (209/241 patients).

c

A serum creatinine increase of 100% but <200% from baseline.

d

A serum creatinine increase of at least 200% from baseline, an absolute value of 4.0 mg/dL (353.6 µmol/L) or receipt of acute dialysis.

e

Among all patients with a hyperkalemia event, 23% were in the hospital at the time of the potassium test (71/307 patients); 76% of these patients in the hospital had the laboratory test done on the day of or after the admission date (54/71 patients).

f

Analysis performed among 15 030 people with 30-day follow-up serum creatinine tests (7804 NSAID users and 7226 nonusers).

N/A, not applicable.

FIGURE 2.

FIGURE 2

Open in a new tab

Subgroup analyses for the outcome of AKI from prescription NSAID use. N/A, not applicable. aWe defined the percentage of maximum NSAID daily dose by the NSAID group, with the nonuser group following their matched NSAID user.

Secondary outcomes: hyperkalemia and all-cause mortality

NSAID use was associated with a higher 30-day risk of hyperkalemia: 184 (0.40%) versus 123 (0.27%) events, respectively [OR 1.50 (95% CI 1.20–1.89)]. We did not observe a statistically significant association between NSAID use and higher thresholds of hyperkalemia (serum potassium ≥6.0 or ≥6.5 mEq/L); there were few events and estimates were imprecise (Table 2). Baseline serum potassium, eGFR, ACEi or ARB use, diuretic use and NSAID dose did not significantly modify the association between NSAID use and hyperkalemia (Supplementary data, Figure S1).

NSAID use was associated with a higher 30-day risk of AKI or hyperkalemia: 510 (1.11%) versus 370 (0.80%) events, respectively [OR 1.39 (95% CI 1.21–1.59)]. NSAID use versus nonuse was not significantly associated with all-cause mortality: 66 (0.14%) versus 79 (0.17%) events, respectively [OR 0.83 (95% CI 0.60–1.16)].

Additional analysis

Over a 30-day follow-up, 16% (15 030) and 14% (12 519) of patients had at least one serum creatinine and serum potassium test completed, respectively. Among those with a follow-up serum creatinine test, 17% (7804) were NSAID users and 16% (7226) were nonusers. We found similar risk estimates when we looked at the association between NSAID use and AKI only among the 15 030 people with follow-up serum creatinine tests (Table 2).

Prediction model

Among 61 219 NSAID users, 701 (1.15%) developed AKI or hyperkalemia in the 30 days following prescription. Our model included six predictors of AKI or hyperkalemia: older age, male gender, lower baseline eGFR, higher baseline serum potassium, ACEi or ARB prescription and diuretic prescription (Supplementary data, Table S8).

Predicted risk ranged from 0.05 to 22.6% (see Supplementary data, Table S9 for distribution). The optimism-adjusted C-statistic was 0.72 (95% CI 0.70–0.74), indicating moderate discrimination (Supplementary data, Figure S2) and the model had good calibration (Figure 3). Supplementary data, Table S10 demonstrates the clinical utility of the model to identify high-risk patients based on predicted risk thresholds >1, >5 and >10%. The sensitivity ranged from 2.6 to 67.8% and the specificity from 64.1 to 99.8%. This model is available as an online calculator: https://qxmd.com/calculate-by-qxmd.

FIGURE 3.

FIGURE 3

Open in a new tab

Calibration plot of predicted probabilities versus observed events of AKI or hyperkalemia among patients in the NSAID user group.

DISCUSSION

In this large population-based cohort study of older adults, we found that receiving a new NSAID prescription (with a supply >14 days) was associated with a higher 30-day risk of AKI and hyperkalemia compared with no prescription NSAID use. However, absolute 30-day risks of AKI and hyperkalemia after NSAID initiation were low (<1%). Since only 15% of people received serum creatinine and potassium tests within a 30-day follow-up, the true incidence of AKI and hyperkalemia in this population is likely higher. We also found that NSAID users did not have a higher risk of 30-day mortality. This was likely because the majority of the adverse outcomes were mild: 79% of AKI events were stage 1 and 63% of people with hyperkalemia had serum potassium values from 5.5  to 6 mEq/L. Therefore, prescription NSAID use may be safe for many older adults, but given the higher relative risk, providers should still use caution and assess individual risk when considering NSAID prescriptions for older adults.

A systematic review and meta-analysis by Zhang et al. [13], examining 1.6 million people from 10 case–control studies, showed that NSAID use versus nonuse was associated with 70% greater odds of developing AKI [13]. Similarly, we found a 40% greater relative risk of AKI with NSAID use compared with nonuse. A recent population-based retrospective cohort study from Ontario, Canada of older patients with CKD, congestive heart failure or hypertension found a new NSAID prescription compared with no such prescription was not associated with a higher risk of hospitalization with AKI, hospitalization with hyperkalemia or death within 30 days of prescription date [46]. However, these authors used diagnosis codes to define hospitalization with AKI and hyperkalemia, which are known to have low sensitivity compared with laboratory data [47, 48].

The association between NSAID use and hyperkalemia is less consistent in the literature. Two large case–control studies comparing NSAID use versus no use found opposing results [14, 16]. Our large population-based cohort study helps remove some of the uncertainty of hyperkalemia risk, as we found that NSAID use was associated with a 50% increased risk of developing hyperkalemia.

Clinical guidelines recommend that patients with CKD avoid NSAIDs [2–6]. In our study, baseline eGFR did not significantly modify the relative association between NSAID use and AKI risk. Consistent with our findings, Zhang et al. [13] also found that patients with CKD had a similar relative risk of AKI with NSAID use compared with the general population. However, patients with lower baseline eGFR have the highest absolute increase in AKI risk with NSAID use.

We developed a prediction model that may be useful to discriminate between people at low versus high risk of AKI or hyperkalemia if they initiated NSAID treatment. Our model predicted patients’ risk of experiencing AKI or hyperkalemia within 30 days of NSAID initiation with acceptable accuracy. Using a predicted risk threshold of >5%, we showed that our model had high specificity, which means we can be confident that people with a predicted risk >5% are truly at high risk. Using our model, clinicians can identify high-risk patients who should either receive serum creatinine and potassium monitoring after initiating NSAIDs or should avoid taking NSAIDs altogether. Our calibration plot shows that it may slightly underestimate risk for individuals at higher levels of risk. This model should be externally validated in other datasets and populations before it is used in practice.

Study strengths and limitations

We completed a large population-based study to assess the association of an uncommon, yet important complication of NSAID use among older adults. This is the first study to use a cohort design with laboratory data to quantify the absolute risk of AKI and hyperkalemia with NSAID use. This is also the first study to develop a prediction model to estimate patients’ risk of developing these outcomes after NSAID prescription. Given the observational study design, we cannot infer causality. Comparing NSAID users to nonusers (rather than people prescribed other drugs) may have introduced some confounding by indication bias. Although we cannot completely eliminate residual confounding, we attempted to reduce it by using propensity scores to balance patients on >150 baseline characteristics.

We used laboratory data to ascertain AKI and hyperkalemia events because these events are underrepresented in administrative databases [47, 48]. This also allowed us to examine associations among subgroups with varying baseline risk of our outcomes, as everyone required a baseline serum creatinine and potassium value. However, we only captured patients who received laboratory tests as part of routine care (∼15% of cohort). There may have been a tendency for NSAID users to receive testing compared with nonusers (17% versus 16%); however, we do not expect this small difference to bias our results since there was a similar association between NSAID use and AKI in the additional analysis restricted to patients with follow-up serum creatinine testing.

Another limitation is that we only had data on NSAID prescriptions and are unaware of over-the-counter NSAID use. However, researchers have shown that not accounting for over-the-counter medications likely contributes only a small amount of bias [49]. Another limitation is that we could only identify prescriptions dispensed by a pharmacy, but we do not know if patients were taking their medications. Similarly, patients may take NSAIDs on an as-needed basis rather than daily. Both these limitations—the possibility of the nonuser group taking over-the-counter NSAIDs and the NSAID users not taking their medication as prescribed—would likely attenuate the estimated effect of NSAID use on AKI and hyperkalemia.

We only included patients >66 years of age, but our findings were consistent with other studies that included adults of all ages [14, 16].

CONCLUSIONS

In summary, we found that older adults prescribed NSAIDs for >14 days are at greater risk for AKI and hyperkalemia in the following 30 days compared with similar patients not prescribed NSAIDs; however, this did not translate into an increased short-term mortality risk. Therefore, prescription NSAID use among many older adults may be safe, but providers should still use caution and assess individual patient risk.

Supplementary Material

gfz062_Supplementary_Material
Click here for additional data file. (288.5KB, docx)

FUNDING

This study was supported by the ICES Western site. ICES is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). Core funding for ICES Western is provided by the Academic Medical Organization of Southwestern Ontario (AMOSO), the Schulich School of Medicine and Dentistry (SSMD), Western University and the Lawson Health Research Institute (LHRI). This research was conducted by members of the ICES Kidney, Dialysis and Transplantation team at the ICES Western facility, who are supported by a grant from the Canadian Institutes of Health Research (CIHR). Parts of this material are based on data and/or information compiled and provided by the Canadian Institute for Health Information (CIHI). The opinions, results and conclusions are those of the authors and are independent from the funding and data sources. No endorsement by the ICES, AMOSO, SSMD, LHRI, CIHR, CIHI or MOHLTC is intended or should be inferred. D.M.N.’s training is supported by a CIHR Doctoral Scholarship. A.X.G. was supported by the Dr Adam Linton Chair in Kidney Health Analytics and a Clinician Investigator Award from the CIHR. This research was undertaken, in part, thanks to funding from the Canada Research Chairs program for M.M.-R.

AUTHORS’ CONTRIBUTIONS

D.M.N. and A.X.G. developed the initial study and analysis plan. K.S.B., M.M.-R., E.M., J.C.F., P.S.R. and M.A.W. provided input and approved the study and analysis plan. D.M.N. completed all data programming and statistical analyses. All authors helped with interpretation of the results. P.S.R. created the online calculator. D.M.N. drafted the initial manuscript with oversight by A.X.G. and all other authors critically reviewed and revised the manuscript. All authors reviewed and approved the final manuscript.

CONFLICT OF INTEREST STATEMENT

None declared.

(See related article by Novick and Grams. Safely treating pain in older adults. Nephrol Dial Transplant 2019; 34: 1075--1077)

REFERENCES

  • 1.Health Canada. Notice: Basic Product Monograph Information for Nonsteroidal Anti-Inflammatory Drugs (NSAIDs) Ottawa, ON, Canada: Health Canada, 2006
  • 2.Kidney Disease: Improving Global Outcomes CKD Work Group. KDIGO clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int Suppl 2012; 3: 1–150 [Google Scholar]
  • 3.National Kidney Foundation. K/DOQI Clinical Practice Guidelines for Chronic Kidney Disease: Evaluation, Clasification and Stratification. American Journal of Kidney Diseases 2002; 39: S1-S266. [PubMed] [Google Scholar]
  • 4. Ickowicz E. Pharmacological management of persistent pain in older persons. J Am Geriatr Soc 2009; 57: 1331–1346 [DOI] [PubMed] [Google Scholar]
  • 5. Henrich WL, Agodoa LE, Barrett B. et al. Analgesics and the kidney: summary and recommendations to the Scientific Advisory Board of the National Kidney Foundation from an ad hoc committee of the National Kidney Foundation. Am J Kidney Dis 1996; 27: 162–165 [DOI] [PubMed] [Google Scholar]
  • 6. Launay-Vacher V, Karie S, Fau JB. et al. Treatment of pain in patients with renal insufficiency: the World Health Organization three-step ladder adapted. J Pain 2005; 6: 137–148 [DOI] [PubMed] [Google Scholar]
  • 7.Hypertension Canada. Annual Update: 2016 Hypertension Canada CHEP Guidelines for the Management of Hypertension Markham, ON, Canada: Hypertension Canada, 2016
  • 8. Weber MA, Schiffrin EL, White WB. et al. Clinical practice guidelines for the management of hypertension in the community a statement by the American Society of Hypertension and the International Society of Hypertension. J Hypertens 2014; 32: 3–15 [DOI] [PubMed] [Google Scholar]
  • 9. Canadian Diabetes Association Clinical Practice Guidelines Expert Committee, Cheng A. Canadian Diabetes Association 2013 clinical practice guidelines for the prevention and management of diabetes in Canada. Introduction. Can J Diabetes 2013; 37(Suppl 1): S1–S3 [DOI] [PubMed] [Google Scholar]
  • 10. Yancy CW, Jessup M, Bozkurt B. et al. ACCF/AHA guideline for the management of heart failure: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 2013; 62: e147–e239 [DOI] [PubMed] [Google Scholar]
  • 11. Rostom A, Moayyedi P, Hunt R.. Canadian consensus guidelines on long-term nonsteroidal anti-inflammatory drug therapy and the need for gastroprotection: benefits versus risks. Aliment Pharmacol Ther 2009; 29: 481–496 [DOI] [PubMed] [Google Scholar]
  • 12. Nash DM, Brimble S, Markle-Reid M. et al. Quality of care for patients with chronic kidney disease in the primary care setting: a retrospective cohort study from Ontario, Canada. Can J Kidney Health Dis 2017; 4: 1–14 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Zhang X, Donnan PT, Bell S. et al. Non-steroidal anti-inflammatory drug induced acute kidney injury in the community dwelling general population and people with chronic kidney disease: systematic review and meta-analysis. BMC Nephrol 2017; 18: 1–12 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Michel A, Martín-Pérez M, Ruigómez A. et al. Risk factors for hyperkalaemia in a cohort of patients with newly diagnosed heart failure: a nested case–control study in UK general practice. Eur J Heart Fail 2015; 17: 205–213 [DOI] [PubMed] [Google Scholar]
  • 15. Aljadhey H, Tu W, Hansen RA. et al. Risk of hyperkalemia associated with selective COX-2 inhibitors. Pharmacoepidemiol Drug Saf 2010; 19: 1194–1198 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Lafrance J-P, Miller DR.. Dispensed selective and nonselective nonsteroidal anti-inflammatory drugs and the risk of moderate to severe hyperkalemia: a nested case-control study. Am J Kidney Dis 2012; 60: 82–89 [DOI] [PubMed] [Google Scholar]
  • 17. Romio SA, Vergouwe Y, Valkhoff V. et al. Risk models for predicting stroke and upper gastrointestinal complications in NSAID users: the SOS Project Silvana. Pharmacoepidemiol Drug Saf 2012; 21: 1–481 [Google Scholar]
  • 18. Sweet BV, Townsend KA, Tsai CY.. Risk assessment of NSAID-induced gastrointestinal toxicity in ambulatory care patients. Am J Health Syst Pharm 2004; 61: 1917–1921 [DOI] [PubMed] [Google Scholar]
  • 19. Cheetham TC, Levy G, Spence M.. Predicting the risk of gastrointestinal bleeding due to nonsteroidal antiinflammatory drugs: NSAID electronic assessment of risk. J Rheumatol 2003; 30: 2241–2244 [PubMed] [Google Scholar]
  • 20. Petersen J, Hallas J, Dall M. et al. A model to assess the risk for ASA/NSAID-related ulcer bleeding for the individual patient based on the number of risk factors. Sa1879. Gastroenterology 2014; 146(5 Suppl 1): S-319 [Google Scholar]
  • 21.Ontario Ministry of Finance. 2016 Census Highlights: Factsheet 3 https://www.fin.gov.on.ca/en/economy/demographics/census/cenhi16-3.html (22 March 2019, date last accessed)
  • 22.Ontario Ministry of Health and Long-term Care. Drugs and Devices Division, Ontario Public Drug Programs http://www.health.gov.on.ca/en/public/programs/drugs/ (22 March 2019, date last accessed)
  • 23. Benchimol EI, Smeeth L, Guttmann A. et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement. PLoS Med 2015; 12: 1–22 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Collins GS, Reitsma JB, Altman DG. et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med 2015; 162: 55–63 [DOI] [PubMed] [Google Scholar]
  • 25. Levy AR, O’Brien BJ, Sellors C. et al. Coding accuracy of administrative drug claims in the Ontario Drug Benefit database. Can J Clin Pharmacol 2003; 10: 67–71 [PubMed] [Google Scholar]
  • 26. Levey AS, Stevens LA.. Estimating GFR using the CKD Epidemiology Collaboration (CKD-EPI) creatinine equation: more accurate GFR estimates, lower CKD prevalence. Estimates, and better risk predictions. Am J Kidney Dis 2010; 55: 622–627 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Statistics Canada. National Household Survey 2011.http://www12.statcan.gc.ca (22 March 2019, date last accessed)
  • 28.Johns Hopkins Bloomberg School of Public Health. Health Services Research & Development Center. The Johns Hopkins ACG© Case-Mix System Version 10.0 Release Notes Baltimore: Johns Hopkins University. 2011
  • 29.Ontario Ministry of Finance. Ontario Demographic Quarterly: Highlights of Fourth Quarter 2017 https://www.fin.gov.on.ca/en/economy/demographics/quarterly/dhiq4.html (22 March 2019, date last accessed)
  • 30. Kellum J. A, Lameire N, Aspelin P. et al. KDIGO clinical practice guideline for acute kidney injury. Kidney Int Suppl 2012; 2: 1–138 [Google Scholar]
  • 31. Copley-Merriman C, Olanrewaju O, Hogue S.. Systematic literature review of the temporal relationship between NSAID use and gastrointestinal, cardiovascular, or renal events. Value Health 2013; 16: A110 [Google Scholar]
  • 32. Austin PC. Using the standardized difference to compare the prevalence of a binary variable between two groups in observational research. Commun Stat 2009; 38: 1228–1234 [Google Scholar]
  • 33. Rosenbaum PR, Rubin DB.. The bias due to incomplete matching. Biometrics 1985; 41: 103–116 [PubMed] [Google Scholar]
  • 34. Laupacis A, Sackett DL, Roberts RS.. An assessment of clinically useful measures of the consequences of treatment. N Engl J Med 1988; 318: 1728–1733 [DOI] [PubMed] [Google Scholar]
  • 35. Dreischulte T, Morales DR, Bell S. et al. Combined use of nonsteroidal anti-inflammatory drugs with diuretics and/or renin-angiotensin system inhibitors in the community increases the risk of acute kidney injury. Kidney Int 2015; 88: 396–403 [DOI] [PubMed] [Google Scholar]
  • 36. Ng SY, Sanagou M, Wolfe R. et al. Prediction of acute kidney injury within 30 days of cardiac surgery. J Thorac Cardiovasc Surg 2014; 147: 1875–1883, 1883.e1 [DOI] [PubMed] [Google Scholar]
  • 37. Lombardi R, Ferreiro A.. Risk factors profile for acute kidney injury after cardiac surgery is different according to the level of baseline renal function. Ren Fail 2008; 30: 155–160 [DOI] [PubMed] [Google Scholar]
  • 38. Thakar CV, Arrigain S, Worley S. et al. A clinical score to predict acute renal failure after cardiac surgery. J Am Soc Nephrol 2005; 16: 162–168 [DOI] [PubMed] [Google Scholar]
  • 39. Park S, Baek SH, Lee SW. et al. Elevated baseline potassium level within reference range is associated with worse clinical outcomes in hospitalised patients. Sci Rep 2017; 7: 2402. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Nakyung J, Winterstein AG.. Addition of nonsteroidal anti-inflammatory drugs (NSAIDs) and risk of acute kidney injury (AKI) in hospitalized patients receiving vancomycin. Pharmacoepidemiol Drug Saf 2015; 24: 1–58726395592 [Google Scholar]
  • 41. Schneider V, Lévesque LE, Zhang B. et al. Association of selective and conventional nonsteroidal antiinflammatory drugs with acute renal failure: a population-based, nested case-control analysis. Am J Epidemiol 2006; 164: 881–889 [DOI] [PubMed] [Google Scholar]
  • 42. Fournier JP, Sommet A, Durrieu G. et al. More on the “Triple Whammy”: antihypertensive drugs, non-steroidal anti-inflammatory agents and acute kidney injury–a case/non-case study in the French pharmacovigilance database. Ren Fail 2014; 36: 1166–1168 [DOI] [PubMed] [Google Scholar]
  • 43. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating (Statistics for Biology and Health). New York: Springer, 2009 [Google Scholar]
  • 44. Austin PC, Steyerberg EW.. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014; 33: 517–535 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Steyerberg EW, Harrell FE, Borsboom GJJM. et al. Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001; 54: 774–781 [DOI] [PubMed] [Google Scholar]
  • 46. Bouck Z, Mecredy GC, Ivers NM. et al. Frequency and associations of prescription nonsteroidal anti-inflammatory drug use among patients with a musculoskeletal disorder and hypertension, heart failure, or chronic kidney disease. JAMA Intern Med, 2018; 178: 1516–1525 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Hwang YJ, Shariff SZ, Gandhi S. et al. Validity of the International Classification of Diseases, Tenth Revision code for acute kidney injury in elderly patients at presentation to the emergency department and at hospital admission. BMJ Open 2012; 2: e001821. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48. Fleet JL, Shariff SZ, Gandhi S. et al. Validity of the International Classification of Diseases 10th revision code for hyperkalaemia in elderly patients at presentation to an emergency department and at hospital admission. BMJ Open 2012; 2: e002011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49. Yood MU, Campbell UB, Rothman KJ. et al. Using prescription claims data for drugs available over-the-counter (OTC). Pharmacoepidemiol Drug Saf 2007; 16: 961–968 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

gfz062_Supplementary_Material
Click here for additional data file. (288.5KB, docx)

Articles from Nephrology Dialysis Transplantation are provided here courtesy of Oxford University Press

RESOURCES