Predictive validity of the Beers and STOPP Criteria to detect adverse drug events, hospitalizations, and emergency department visits in the United States

Abstract

OBJECTIVES

To compare the predictive validity of the 2003 Beers, 2012 Beers, and STOPP Criteria.

DESIGN

Retrospective cohort.

SETTING

Managed care administrative claims data from 2006 to 2009

PARTICIPANTS

174,275 commercially insured persons 65 and older in the United States.

MEASUREMENTS

Association between adverse drug events, emergency department (ED) visits, and hospitalization outcomes and inappropriate medications using time-varying Cox proportional hazard models. Measures of model discrimination (c-index) and hazard ratios (HR) were calculated to compare unadjusted and adjusted models for associations.

RESULTS

The prevalence of inappropriate prescribing was 34.1%, 32.2%, and 27.6% for the 2012 Beers, 2003 Beers, and the STOPP Criteria. Each criteria modestly discriminated ADEs in unadjusted analyses: STOPP Criteria (HR=2.89 [2.68–3.12]; C-index=0.607), 2012 Beers Criteria (HR=2.51 [2.33–2.70]; C-index=0.603), 2003 Beers Criteria (HR=2.65 [2.46–2.85]; C-index=0.605). Similar results were observed for ED visits and hospitalizations. Adjusted analyses increased the c-indices to between 0.65 and 0.70. The kappa for agreement between criteria was 0.80 for the 2003 and 2012 Beers Criteria, 0.58 for the 2012 Beers and STOPP Criteria, and 0.59 for the 2003 Beers and STOPP Criteria. For the three outcomes, the 2012 Beers Criteria had the highest sensitivity (61.2%–71.2%) and the lowest specificity (41.2%–70.7%) while the STOPP Criteria had the lowest sensitivity (53.8%–64.7%) but the highest specificity (47.8%–78.1%).

CONCLUSIONS

All three criteria were modestly prognostic for ADEs, EDs, and hospitalizations with the STOPP Criteria slightly outperforming both Beers Criteria. With low sensitivity, low specificity, as well as low agreement between the criteria, these criteria can be used in a complementary fashion to enhance sensitivity of detecting ADEs.

INTRODUCTION

A potentially inappropriate medication (PIM) exists when the risk of adverse events due to treatment outweighs the clinical benefit.¹ PIMs are associated with adverse health and economic outcomes, ^2–11 making detection and prevention a primary goal of clinicians, payers, and policymakers. Since its development in 1991,¹² the Beers Criteria has become the most widely used and recognized explicit criteria for the detection of PIMs in older adults.^8,13,14 The criteria were updated in 1997¹⁵, 2003¹⁶, and again in 2012 by an American Geriatrics Society (AGS) expert panel and includes drugs to always avoid, drugs to use with caution, and drug-disease interactions.¹⁷

The Screening Tool of Older Persons’ Prescriptions (STOPP) Criteria is an alternative criteria developed in 2008 by a European consensus group.¹ The STOPP Criteria is organized by physiological system and includes drugs to avoid, drug-drug and drug-disease interactions, and therapeutic duplication to define PIMs. It is purported to be more effective in a European population where many of the medications considered inappropriate by the Beers Criteria are not available.^1,18 As a result, the STOPP Criteria has modest overlap with the 2012 Beers Criteria – 55% of the 65 criteria are not found in the 2012 Beers Criteria.¹⁹

The STOPP and 2003 Beers Criteria have been compared in European populations where the STOPP Criteria identified more PIMs and increased the odds of having a serious adverse drug event (ADE) by 85%. ^20–26 A study conducted in Spain compared the 2003 Beers and STOPP Criteria along with the updated 2012 Beers Criteria.²⁷ The PIM prevalence was 24.3%, 35.4%, and 44% for the 2003 Beers, STOPP, and 2012 Beers Criteria and the agreement between the 2012 Beers and STOPP Criteria was 0.35. That study did not compare the criteria on adverse outcomes.

Because of the lack of evidence comparing the Beers Criteria with the STOPP Criteria in a United States (US) population, a comparison of the ability of each criteria to predict relevant clinical outcomes is warranted.¹⁹ Therefore, the current study sought to compare the predictive validity of the 2003 Beers, 2012 Beers, and STOPP Criteria using three outcome measures: 1) ADEs, 2) all cause emergency department (ED) visits, and 3) all cause hospitalizations. Further, the prevalence of PIMs detected with each criteria was investigated as well as measures of agreement between the criteria.

METHODS

Data source

The study sample was selected from a 10% random sample of the proprietary Lifelink Health Plans Claims Database comprised of administrative claims from 80 Managed Care Organizations within the US. The data capture the health claims data of the elderly enrolled in health plans offering employer sponsored coverage and Medicare Advantage plans but do not capture data for persons enrolled in traditional Medicare.

Study subjects and design

We used a retrospective cohort study design. Inclusion into the cohort was based on a person being at least 65 years old and having at least 9 months of continuous medical and pharmacy coverage, including a 6 month pre-index period and a minimum 3 months of follow-up between January 1, 2006 and December 31, 2009. The index date was defined as the first day of the seventh month of continuous eligibility. Individuals were followed until the end of continuous enrollment, the end of the study period, or until an outcome event occurred. Because full medical and pharmacy claims data may not be captured, individuals with the payer identified as “Medicaid” were excluded as this group may have additional insurance or incomplete records.

Potentially inappropriate medication exposure

Exposure definitions were individually created according to the 2003 ¹⁶ and 2012 Beers Criteria ¹⁷ and the STOPP Criteria.¹ A composite “All Criteria” exposure measure was also developed based on exposure to any of the three PIM criteria. Therapeutic duplication, present as an over-arching item in the STOPP Criteria, was excluded as this is not unique to the elderly population and was deliberately excluded from the Beers Criteria.¹⁷ Additionally, dabigatran (2012 Beers Criteria) was not on the market during the time period of this study (2006–2009) and propoxyphene (2003 Beers Criteria) was not included because it is no longer available in the U.S. market. Otherwise, all items from each criteria were included.

Drug-only criteria were mapped using the Medi-Span Generic Product Identifier (GPI, Wolters Kluwer Health, Philadelphia, PA) classification system and the American Hospital Formulary Service Pharmacologic-Therapeutic Classification codes (AHFSCC). These hierarchical coding systems allowed for classification from the drug class, individual medications, formulations (e.g. extended release), or dosing of individual products.

Disease-dependent PIM definitions were based on International Classifications of Disease, 9^th Revision, Clinical Modification (ICD-9-CM) codes in conjunction with the GPI and AHFSCC medication codes. As an initial basis for defining disease concepts, the validated Clinical Classification Software (CCS) codes were used to map ICD-9-CM definitions.²⁸ These codes were compared to other validated coding algorithms used by the Center for Medicare and Medicaid Services (CMS) ²⁹, Agency for Healthcare Research and Quality (AHRQ) ³⁰, and coding algorithms used widely with administrative claims data.^31,32 Identified codes were included if they were present in at least two of these sources.

For disease states not defined using the above sources, literature searches were performed in MEDLINE using “ICD-9” and “administrative claims” with a description of the disease. Additionally, a manual search of an ICD-9-CM dataset and web pages for ICD-9-CM coding were queried using disease specific terms (http://www.cms.gov/medicare-coverage-database/staticpages/icd-9-code-lookup.aspx; http://icd9.chrisendres.com/). A review of all code selections was conducted by two clinical pharmacists with experience in administrative claims research and a geriatric pharmacy specialist. (The full details of PIM disease definitions are provided in Supplement 1.)

A time varying approach was used to assess PIM exposure as a monthly binary variable. For drug-only criteria, a subject was only considered exposed to a PIM for the month a medication was dispensed. For PIM definitions based on co-existing disease states, a patient was considered to have that disease in the month of the first inpatient or non-ancillary outpatient claim with a primary or secondary diagnosis for that disease and for all subsequent months of the study.

Outcome variables

ADEs were based on ICD-9-CM codes previously used for surveillance in hospital claims data ³³. A similar manual search strategy was performed with the terms “drug-induced”, “adverse effect”, “caused by”, “poisoning”, “drug”, and “allergy” appearing in code descriptions. ADEs were classified based on the subgroups in the original publication ³³ with the addition of those identified through the manual search and the removal of ADEs specific to the perinatal period. (ADEs identified in this study, along with the ICD-9-CM codes and rates, are available in Supplement 2.)

All-cause ED visits were defined by procedure and place of service codes and hospitalizations were identified by unique confinement numbers. ADEs, all-cause ED visits, and all-cause hospitalizations were considered separate outcomes; therefore, an individual could experience one or more of the outcomes.

Subject characteristics

Cohort demographics and plan characteristics were determined at the beginning of the post-index period. Age was categorized as 65–74, 75–84, and 85 years and older. Region was classified as South, West, East, and Midwest. Insurance coverage was categorized into five categories based on payer type (Medicare or commercial) and plan type: HMO (health maintenance organization, non-HMO (Preferred Provider Organization, Consumer Directed, Indemnity, Point of Service), or unknown.

Comorbidities were based on the Charlson Comorbidity Index using the ICD-9-CM coding algorithms by Quan et al. ³² and were assessed during the 6 month pre-index period. Use of long term care was determined during the pre-index period and included the use of skilled nursing facilities, nursing homes, or hospice care. Additionally, prescription utilization was evaluated separately as the total number of prescription fills and refills annualized to the number per 12 months as well as the total number of unique drug classes used during the post-index period.

Data analysis

Baseline variables for the total cohort were compared for those exposed to a PIM from at least one of the three criteria using two-sided Student t-tests for continuous variables and the Chi-square tests for categorical variables. Unadjusted and adjusted Cox proportional hazards models were used to estimate the relationship between PIM exposure and outcomes. In order to preserve the temporal relationship between PIM exposure and outcomes, individuals having an outcome during the pre-index period were excluded in the model assessing the influence of PIMs on that particular outcome but were included in models exploring one of the other two outcomes. Three time-varying models and one time invariant approach were estimated to explore the temporal relationship between PIM exposure and outcome. The primary model assessed PIM exposure in month t(i) and looked for an outcome in month t(i+1); providing stronger assurances that the exposure preceded the experience of the outcome event. An alternative time-varying model assessed exposures and outcomes within the same month. An additional third time-varying approach used a once-exposed-always-exposed exposure classification where a subject was considered exposed the first month a PIM was detected and all subsequent months.

Dummy variables were created for each covariate. Reference categories were as follows: Age 65–74; Male gender, East region, and Medicare HMO insurance coverage. The Charlson Comorbidity Index diseases were used as individual binary disease states. Prescription utilization variables were considered to possibly exist along the causal pathway and were excluded from the primary analyses. Sensitivity analyses were conducted which considered each prescription utilization variable as a categorical and continuous variable. Additionally, separate models were estimated which stratified the cohort by these measures. The proportionality assumption was evaluated for each covariate in models by specifying interaction terms between each covariate and log-time – where statistically significant coefficients would indicate a violation of the assumption. Hazard ratios and 95% confidence intervals are reported.

To compare the predictive validity of the criteria, a c-index specifically developed for time-varying models was used.³⁴ The c-index is analogous to the c-statistic often used with logistic regression and ranges between 0.5 and 1 where a value of 0.5 indicates model prediction no better than chance and a value of 1 indicates a model which predicts events perfectly. Concordance or discordance occur when the predictor score for the individual having an event is greater or lesser than individuals not having an event at that time.³⁴

Additionally, Cohen’s kappa was calculated to assess the person-level agreement between all possible pairwise PIM criteria. The sensitivity and specificity of each of the PIM criteria were calculated using each of the outcome measures and a composite outcome measure as gold standards. All analyses were conducted using SAS version 9.3 (SAS Institute, Inc., Cary, NC).

RESULTS

A total of 538,532 individuals were 65 or older during the time period January 1, 2006 and December 31, 2009. Applying eligibility inclusion criteria, 257,206 had at least 9 months of continuous medical insurance enrollment and 175,696 also had 9 months of continuous pharmacy benefit enrollment. Combined, 175,581 had at least 9 months continuous enrollment with both medical and pharmacy benefit during the study period. An additional 1,306 (<1%) individuals were excluded having “Medicaid” identified as the payer type. The final cohort consisted of 174,275 individuals representing 32.4% of the original elderly sample (Supplement 3). The mean follow up time of the cohort was slightly over 2 years (24.9 months, Median 27.0 months, IQR 12–39 months) and the cohort contributed a combined 361,621 person-years. Baseline cohort demographics by PIM exposure are presented in Table 1.

Table 1.

Baseline Characteristics for Cohort and Those Exposed to the 2012 Beers, 2003 Beers, and STOPP Criteria

Characteristics	Total Cohort N=174,275 No. (%)	2012 Beers N=59,426 No. (%)	2003 Beers N=56,144 No. (%)	STOPP N=48,121 No. (%)
Age (years)*
65–74	128,306 (73.6)	37,150 (62.5)	36,603 (65.2)	30,951 (64.3)
75–84	34,637 (19.9)	17,098 (28.8)	14,991 (26.7)	13,098 (27.2)
85 and older	11,332 (6.5)	5,178 (8.7)	4,550 (8.1)	4,072 (8.5)
Female*	94,588 (54.3)	34,779 (58.5)	33,997 (60.6)	27,809 (57.8)
Insurance Type*
Medicare HMO	22,570 (13.0)	11,071 (18.6)	9,907 (17.7)	8,630 (17.9)
Medicare non-HMO	24,992 (14.3)	8,995 (15.1)	8,357 (14.9)	7,248 (15.1)
Commercial HMO	20,432 (11.7)	5,449 (9.2)	5,311 (9.5)	4,755 (9.9)
Commercial non-HMO	96,412 (55.3)	31,116 (52.4)	29,868 (53.2)	25,214 (52.4)
Unspecified	9,869 (5.7)	2,795 (4.7)	2,701 (4.8)	2,274 (4.7)
Region
East	35,987 (20.7)	11,739 (19.8)	11,333 (20.2)	10,233 (21.3)
Midwest	57,514 (33.0)	20,388 (34.3)	19,219 (34.2)	16,688 (34.7)
South	43,528 (25.0)	14,294 (24.1)	13,393 (23.9)	11,508 (23.9)
West	37,246 (21.4)	13,005 (21.9)	12,199 (21.7)	9,692 (20.1)
Charlson Co-morbidity
Index (Pre-index)
Mean (SD)*	1.3 (1.7)	1.6 (1.9)	1.6 (1.8)	1.8 (2.0)
0–1	117,690 (67.5)	35,013 (58.9)	33,803 (60.2)	27,111 (56.3)
2–3	39,736 (22.8)	16,513 (27.8)	15,265 (27.2)	13,744 (28.6)
3+	16,849 (9.7)	7,900 (13.3)	7,076 (12.6)	7,266 (15.1)
Prescription utilization
Total prescription fills per 12 months
Mean (SD)*	12.1 (36.5)	20.5 (17.7)	20.0 (16.3)	19.4 (16.3)
Unique drug classes
Mean (SD)*	4.4 (5.3)	8.8 (5.5)	8.7 (5.5)	8.8 (5.7)
Long term care	3,682 (2.1)	2,287 (3.9)	2,088 (3.7)	2,126 (4.42)
Follow-up time (months)
Mean (SD)*	24.9 (13.2)	29.1 (11.8)	29.4 (11.6)	30.1 (11.2)
Median	27.0	36.0	36.0	36.0
IQR	12–39	18–39	19–39	12–39

^*p<0.01 for comparison between “Any Exposure” and Total Cohort.

Significant differences were not observed between criteria

Abbreviations: HMO (health maintenance organization); SD (standard deviation); IQR (inter-quartile range)

Over the entire post-index period, 72,493 (41.6%) of the cohort were exposed to at least one of the criteria and 19.7% were exposed to all three. Exposure to at least one PIM from the 2012 Beers Criteria was 34.1%, the 2003 Beers Criteria 32.2%, and the STOPP Criteria 27.6%. Person-level agreement between each of the PIM criteria, measured by Cohen’s kappa, was “good” between the 2012 and 2003 Beers Criteria (κ = 0.80, Table 2), and “moderate” between the STOPP Criteria and the 2012 and 2003 Beers Criteria (κ = 0.58 and 0.59).³⁵

Table 2.

Inappropriate prescribing criteria person-level concordance and agreement

Exposure to inappropriate prescribing	No. (% of cohort) N=174,275
Any exposure to criteria	72,493 (41.6)
Exposed to more than one criteria	58,915 (32.7)
Exposed to all criteria	34,283 (19.7)
Concordance Between Criteria	No. (% Agree)
2012 Beers*2003 Beers	50,182 (84.4)
2012 Beers*STOPP	38,006 (64.0)
2003 Beers*2012 Beers	50,182 (89.4)
2003 Beers*STOPP	37,293 (66.4)
STOPP*2012 Beers	38,006 (79.0)
STOPP*2003 Beers	37,293 (77.5)
2012 Beers*All Criteria	59,426 (82.0)
2003 Beers*All Criteria	56,144 (77.4)
STOPP*All Criteria	48,121 (66.4)
Agreement Between Criteria	Cohen’s Kappa
2012 Beers*2003 Beers	0.80
2012 Beers*STOPP	0.58
2003 Beers*STOPP	0.59
2012 Beers*All Criteria	0.84
2003 Beers*All Criteria	0.80
STOPP*All Criteria	0.70

The top 5 individual PIMs for each criteria included many of the same medication groups but differed in prevalence because of different definitions of inappropriateness. A “use with caution” criteria which included SSRIs, SNRIs, antipsychotics, and other medications associated with syndrome of inappropriate anti-diuretic hormone (SIADH) was the most prevalent PIM for the 2012 Beers Criteria (16.2% of cohort). This was followed by benzodiazepines (11.3%), skeletal muscle relaxants (6.6%), non-benzodiazepine hypnotics (5.8%), and NSAIDs (5.4%). The top five 2003 Beers Criteria PIMs included anticholinergics and first generation antihistamines (19.4%), SSRIs (“with caution”, 10.5%), benzodiazepines (11.2%), muscle relaxants and antispasmodics (7.4%), and long-term NSAID use (5.1%). The most prevalent STOPP Criteria PIMs included NSAIDs (16.2%), opioids (4.8%), beta-blockers (4.7%), corticosteroids (3.8%), and first generation antihistamines (3.8%). (Complete PIM exposure prevalence is available in Supplements 4–6.)

A total of 1,911 individuals experienced a post-index ADE (67 ADEs per 10,000 person-years, 1.12% of the total cohort) after excluding 3,558 people who had pre-index adverse events. Additionally, 24,614 individuals were excluded who had a pre-index ED visit and an additional 29,864 had a post-index event (140 ED visits per 1,000 person-years, 17.1% of the total cohort). Post-index hospitalizations occurred for 16,444 persons (67 hospitalizations per 1,000 person-years, 9.4% of the total cohort) with 22,190 individuals excluded with hospitalizations occurring in the pre-index period. The associations of demographic and health-related characteristics with each outcome are shown in Supplement 7.

PIM exposure was strongly associated with all study outcomes in both unadjusted and adjusted models (Table 3). In the primary unadjusted model, PIM exposure was associated with a 2 to 3 fold increased risk across all outcomes for the 2003 Beers, 2012 Beers, and STOPP Criteria. The associations were similar across the three outcome measures. A stronger relationship between PIM exposure with all three of the criteria and each of the three outcomes (HRs: 3.67 – 5.30) was observed in the time varying models that assessed exposure and outcome in the same month. The time dependent once exposed always exposed model found more modest associations between all the PIM criteria (HRs: 1.30 – 1.76), however all remained significant. The hazard ratios for the STOPP Criteria in the primary time varying model trended higher than those for either of the Beers Criteria.

Table 3.

Adjusted and Unadjusted Hazards ratios for the 2012 Beers, 2003 Beers, and STOPP Criteria for time varying and non-time varying models.

	Unadjusted models (exposure only)			Adjusted modelsa
Criteria	Time-varying monthly lag (Primary Model)b
	ADEs	Emergency	Hospitalization	ADEs	Emergency	Hospitalization
2012 Beers	2.51 (2.33–2.70)	2.21 (2.16–2.25)	2.25 (2.20–2.30)	2.17 (2.01–2.34)	2.00 (1.96–2.04)	2.03 (1.98–2.07)
2003 Beers	2.65 (2.46–2.85)	2.29 (2.25–2.34)	2.31 (2.26–2.37)	2.33 (2.16–2.52)	2.14 (2.10–2.19)	2.16 (2.11–2.21)
STOPP	2.89 (2.68–3.12)	2.66 (2.60–2.72)	2.80 (2.74–2.87)	2.43 (2.24–2.63)	2.38 (2.32–2.43)	2.46 (2.40–2.52)
	Time-varying month to monthc
	ADEs	Emergency	Hospitalization	ADEs	Emergency	Hospitalization
2012 Beers	4.33 (4.11–4.56)	4.38 (4.31–4.44)	4.27 (4.20–4.34)	3.67 (3.48–3.87)	3.93 (3.87–3.99)	3.75 (3.68–3.81)
2003 Beers	5.01 (4.75–5.28)	4.89 (4.81–4.97)	4.76 (4.68–4.84)	4.30 (4.08–4.54)	4.51 (4.44–4.58)	4.32 (4.25–4.40)
STOPP	5.21 (4.91–5.52)	5.18 (5.09–5.28)	5.30 (5.20–5.41)	4.18 (3.92–4.44)	4.52 (4.43–4.60)	4.47 (4.38–4.56)
	Time-dependent once exposed, always exposedd
	ADEs	Emergency	Hospitalization	ADEs	Emergency	Hospitalization
2012 Beers	1.71 (1.57–1.87)	1.45 (1.42–1.48)	1.46 (1.42–1.49)	1.43 (1.31–1.56)	1.32 (1.29–1.35)	1.30 (1.26–1.33)
2003 Beers	1.66 (1.53–1.81)	1.39 (1.36–1.42)	1.38 (1.35–1.42)	1.45 (1.33–1.58)	1.32 (1.29–1.35)	1.30 (1.26–1.33)
STOPP	1.76 (1.62–1.91)	1.50 (1.46–1.53)	1.54 (1.51–1.58)	1.47 (1.35–1.60)	1.37 (1.34–1.40)	1.38 (1.34–1.42)
	Ever exposuree
	ADEs	Emergency	Hospitalization	ADEs	Emergency	Hospitalization
2012 Beers	3.06 (2.77–3.37)	2.34 (2.28–2.39)	2.58 (2.51–2.65)	2.60 (2.35–2.88)	2.08 (2.03–2.13)	2.27 (2.21–2.34)
2003 Beers	2.83 (2.57–3.12)	2.18 (2.13–2.23)	2.33 (2.27–2.39)	2.49 (2.25–2.74)	2.01 (1.97–2.06)	2.15 (2.09–2.21)
STOPP	3.11 (2.83–3.42)	2.44 (2.38–2.49)	2.71 (2.64–2.78)	2.64 (2.39–2.91)	2.18 (2.13–2.23)	2.38 (2.32–2.45)

^aCovariates included: Age, gender, insurance status, region, long term care, and Charlson comoribidities

^bOutcome events associated with time-varying exposure in the preceding month (i.e. March outcome associated with February exposure

^cOutcome events associated with time-varying exposure in the same month

^dOnce exposed to a criteria, always exposed whether or not exposure status changes

^eExposed at any point during the post-index follow up period

Sample size for final model excluding pre-index events: ADEs (170,717); ED visits (147,661); Hospitalizations (152,085)

For the primary unadjusted model, the c-indices were similar for each of the criteria for each of the outcomes and indicated modest levels of discrimination with c-indices between 0.58 and 0.61 (Table 4). When the models included the pre-index covariates, the levels of discrimination increased to 0.65 to 0.70 and were similar across the criteria for each of the outcomes. The model that assessed PIM exposure and outcome in the same month had the highest measures of discrimination than the other models. Inclusion of prescription utilization measures as covariates increased the discrimination of the models less than 1% and stratification had no significant effect. The sensitivity and specificity of the 2012 Beers, 2003 Beers, and STOPP Criteria for the separate composite outcomes are shown in Table 5.

Table 4.

C-indices and 95% confidence intervals for 2003 Beers, 2012 Beers, and STOPP Criteria for the time varying and non-time varying models

	Unadjusted Model (exposure only)			Adjusted Modela*
	Time-varying monthly lag (Primary Model)b
Criteria	ADEs	Emergency	Hospitalization	ADE	Emergency	Hospitalization
2012 Beers	0.603 (0.597–0.609)	0.585 (0.583–0.587)	0.590 (0.588–0.592)	0.688 (0.677–0.700)	0.652 (0.649–0.655)	0.673 (0.670–0.677)
2003 Beers	0.605 (0.599–0.611)	0.585 (0.583–0.587)	0.588 (0.586–0.590)	0.695 (0.684–0.706)	0.653 (0.650–0.656)	0.673 (0.670–0.676)
STOPP	0.607 (0.601–0.614)	0.590 (0.588–0.592)	0.599 (0.597–0.601)	0.695 (0.685–0.706)	0.661 (0.658–0.664)	0.683 (0.680–0.686)
	Time-varying month to monthc
	ADEs	Emergency	Hospitalization	ADE	Emergency	Hospitalization
2012 Beers	0.642 (0.639–0.645)	0.635 (0.634–0.636)	0.636 (0.634–0.638)	0.733 (0.723–0.744)	0.709 (0.706–0.712)	0.720 (0.717–0.723)
2003 Beers	0.646 (0.643–0.650)	0.635 (0.634–0.636)	0.637 (0.636–0.638)	0.741 (0.730–0.751)	0.708 (0.705–0.711)	0.721 (0.718–0.724)
STOPP	0.642 (0.638–0.647)	0.626 (0.625–0.628)	0.634 (0.633–0.634)	0.741 (0.730–0.752)	0.707 (0.704–0.710)	0.726 (0.723–0.729)
	Time-varying once exposed, always exposed
	ADEs	Emergency	Hospitalization	ADE	Emergency	Hospitalization
2012 Beers	0.566 (0.557–0.574)	0.548 (0.546–0.551)	0.551 (0.549–0.554)	0.666 (0.654–0.679)	0.628 (0.624–0.631)	0.653 (0.648–0.655)
2003 Beers	0.563 (0.554–0.571)	0.542 (0.540–0.545)	0.544 (0.541–0.546)	0.667 (0.655–0.680)	0.626 (0.622–0.629)	0.651 (0.647–0.654)
STOPP	0.567 (0.559–0.574)	0.548 (0.546–0.551)	0.554 (0.552–0.557)	0.670 (0.658–0.682)	0.630 (0.627–0.634)	0.657 (0.652–0.660)
	Ever exposuree
	ADEs	Emergency	Hospitalization	ADE	Emergency	Hospitalization
2012 Beers	0.566 (0.557–0.574)	0.548 (0.546–0.551)	0.551 (0.549–0.554)	0.666 (0.654–0.679)	0.628 (0.624–0.631)	0.652 (0.648–0.655)
2003 Beers	0.563 (0.554–0.571)	0.542 (0.540–0.545)	0.544 (0.541–0.546)	0.667 (0.655–0.680)	0.626 (0.622–0.629)	0.650 (0.647–0.654)
STOPP	0.636 (0.624–0.647)	0.599 (0.596–0.603)	0.612 (0.608–0.615)	0.713 (0.701–0.725)	0.659 (0.656–0.663)	0.687 (0.683–0.691)

^*Covariate only model c-indices: ADE 0.664 (0.651–0.676); Emergency 0.606 (0.603–0.610); Hospitalization 0.647 (0.644–0.651)

^aCovariates included: Age, gender, insurance status, region, long term care, and Charlson comoribidities

^bOutcome events associated with time-varying exposure in the preceding month (i.e. March outcome associated with February exposure

^cOutcome events associated with time-varying exposure in the same month

^dOnce exposed to a criteria, always exposed whether or not exposure status changes

^eExposed at any point during the post-index follow up period

Sample size for final model excluding pre-index events: ADEs (170,717); ED visits (147,661); Hospitalizations (152,085)

Table 5.

Sensitivity and specificity of PIM criteria predicting study outcomes

Criteria	Sensitivity (%)	Specificity (%)
2012 Beers
ADEs	71.2	41.2
Emergency Visits	61.2	70.7
Hospitalizations	64.3	69.0
Composite Outcome	60.6	73.9
2003 Beers
ADEs	67.7	42.8
Emergency Visits	57.8	72.2
Hospitalizations	60.3	70.4
Composite Outcome	57.3	75.4
STOPP
ADEs	64.7	47.8
Emergency Visits	53.8	78.1
Hospitalizations	57.6	76.3
Composite Outcome	53.4	80.2
All Criteria exposure
ADEs	79.8	30.1
Emergency Visits	71.8	63.2
Hospitalizations	74.8	61.4
Composite Outcome	71.4	67.4

Study supplement to facilitate review only, not intended for publication or for inclusion in the word or table count.

DISCUSSION

In studies using the Beers Criteria, PIM rates of 40–50% are common and have ranged as low as 12% ^7,25,36,37 while rates for the STOPP Criteria have ranged from 13–70%.^{20,21,23,38,39} Our study found 41.6% of the cohort to be exposed to at least one of the criteria. The 2003 and 2012 Beers and STOPP Criteria identified PIMs in 32.2% and 34.1%, and 27.6% of the cohort. These rates are similar to a study in Spain comparing the three criteria in an ambulatory population.²⁷ Differences between the Beers and STOPP Criteria when they consider similar drug classes are due to differences in the definition of inappropriateness according to each criteria.

We found that exposure to a PIM from any criteria was associated with an increased risk of ADEs, ED visits, and hospitalizations. Individuals with exposure to PIM from the STOPP Criteria had slightly higher risks than either of the Beers Criteria. Despite the slightly higher risk associations for the STOPP Criteria compared to the Beers Criteria, there were only marginal differences in discrimination between the criteria. The 2012 Beers Criteria performed better in terms of sensitivity across all outcomes but was less specific while the STOPP Criteria was less sensitive but more specific.

For the Beers Criteria, the slightly lower performance appears to be a result of higher exposures resulting in more false-positives weakening the association with outcomes. The STOPP Criteria detected only 53% of individuals having any outcome while the 2012 Beers Criteria detected an additional 7% of individuals having each outcome. When the combined “All Criteria” exposure was considered, sensitivity increased for ADEs, ED visits, and hospitalizations. Overall, the composite exposure measure had a sensitivity of 71.4% and specificity of 67.4% for the composite outcome. Therefore, future updates of the Beers Criteria should consider evidence-based refinement of the criteria to include more drug classes that are predictive of serious adverse outcomes.^40,41

The AGS has adapted the Beers Criteria into a mobile application and a pocket guide for the practicing clinician who they acknowledge as the target audience.¹⁷ The Beers Criteria were also intended for and have been widely used by researchers, pharmacy benefit managers, and policy-makers – greatly broadening the impact of the Beers Criteria over the last twenty years.^42,43 For example, the criteria have been used by the CMS and the National Committee for Quality Assurance (NCQA) as quality indicators in long-term care and ambulatory settings. There have even been cases of “misuse” of the criteria to deny coverage of medications ⁴⁴ leading to an AGS letter to insurers specifying the appropriate uses of the criteria which do not include formulary decision making.⁴⁵

The STOPP/START Criteria were updated in 2014 further expanding and specializing these criteria to detect inappropriate prescribing in older adults.⁴⁶ Additionally, the 2015 update to the Beers Criteria were available for public comment until May 5^th, 2015 on the AGS’s website and will be made available soon for use. The updates to these criteria were not available at the time this study was conducted. Future work should further investigate the predictive validity of these criteria as they are updated.

One of the notable limitations of this study is the outcome measures selected. We used a narrow set of ICD-9-CM codes specific to drug-induced syndromes to define an adverse drug event, some of which are based on supplementary E-codes. These codes were based on previous work which measured the performance of these codes as an ADE surveillance system. They found that the codes had an overall sensitivity and specificity of 55% and 97% for ADEs causing hospital admission and positive predictive value greater than 70%.³³ Though these codes may have only detected half of all adverse drug events in that study, the codes can be expected to detect true ADEs. Conversely, the all-cause hospitalizations and ED visits are not specific to ADEs and may have higher sensitivity detecting serious ADEs but will be less specific. For example, up to 31% of hospitalizations may be medication related⁴⁷ leaving two thirds that are not. This should be considered when interpreting our findings, particularly when we report the sensitivity and specificity which should not be interpreted in the conventional fashion as measures of diagnostic or screening accuracy for a verified outcome.

Non-prescription medications, such as aspirin or NSAIDs, and prescriptions not processed through claims were not present in the data. For example, inappropriate use of proton pump inhibitors based on the STOPP Criteria has been found to be highly prevalent and its underrepresentation in our data may bias the associations between the STOPP Criteria and the outcomes toward the null. The absence of medications considered by all three criteria sets would have a similar but balanced effect. Similarly, disease-dependent PIM definitions may suffer from missing data due to undercoding.^48,49 Thus, our findings are likely conservative as more individuals are likely to be exposed to PIMs than were observed in this study.

We excluded the therapeutic duplication criteria from the STOPP Criteria PIM definition because this item has been specifically mentioned for exclusion from the Beers Criteria as it is not a problem unique to the elderly.⁴¹ While therapeutic duplication has been reported to have a prevalence of nearly 5%, it has not been associated with ADEs in published studies.^5,38 Our exclusion of this item may have decreased the exposure prevalence and the association of the STOPP Criteria with the outcomes.

The most prevalent PIM from the 2012 Beers Criteria considered “use with caution” medications because of the risk of SIADH. Based on the original wording of the 2012 Beers Criteria, this criterion did not require an individual to have previous episodes of SIADH, compared to the STOPP Criteria which did require a previous diagnosis of hyponatremia or SIADH. Thus, all individuals exposed to these commonly used medications, including selective serotonin and norepinephrine reuptake inhibitors, were considered exposed to the Beers Criteria.

This study was strengthened by considering the temporal relationship of exposure and outcomes with a time-varying approach. This allowed for the observation of the initial period of PIM exposure when adverse events may be more apt to occur.⁵⁰ This method also allows individuals to move to and from exposed and unexposed status taking into account changes, additions, and discontinuations of therapy. Our primary model in which exposure was assessed in one month and outcomes assessed in the following month strongly preserves the temporal relationship where exposure precedes outcomes. Though the month-to-month model provided stronger associations between exposure and outcome, reverse causality may explain the stronger association. On the other hand, this approach would not capture associations to PIM and ADEs that have latency periods beyond one month which would bias the findings towards the null.

The administrative claims capture the healthcare utilization of members enrolled in commercial coverage and Medicare Advantage plans and would be expected to be generalizable to that population. Individuals covered under traditional Medicare are not included. This population may differ from the general Medicare population by demographic characteristics such as income status, education, and health behaviors which could not be compared in the current study.

CONCLUSIONS

This was the first study to compare the predictive validity of the updated Beers Criteria to the STOPP Criteria in a population of older adults as well as the first application of the full Beers Criteria including drug-disease items in the US. Our study showed low agreement and no significant differences between the two iterations of the Beers Criteria and the STOPP Criteria in the level of discrimination for ADEs, ED visits, and hospitalizations, though each was moderately prognostic of these outcomes. Future evidence-guided updates of these widely used tools should identify medications and medication classes that may increase the predictive ability of the criteria. These criteria can be used in a complementary fashion to enhance sensitivity of detecting ADEs to decrease adverse drug events in older patients.