Skip to main content
HHS Author Manuscripts logoLink to HHS Author Manuscripts
. Author manuscript; available in PMC: 2019 Sep 1.
Published in final edited form as: J Manag Care Spec Pharm. 2018 Sep;24(9):875–885. doi: 10.18553/jmcp.2018.24.9.875

Prescription Opioid Quality Measures Applied among Pennsylvania Medicaid Enrollees

Gerald Cochran 1,, Wei-Hsuan Lo-Ciganic 2, Walid F Gellad 3, Adam J Gordon 4, Evan Cole 5, Carroline Lobo 6, Winfred Frazier 7, Ping Zheng 8, Chung-Chou H Chang 9, David Kelley 10, Julie M Donohue 11
PMCID: PMC6418411  NIHMSID: NIHMS1009975  PMID: 30156454

Abstract

Background:

The Pharmacy Quality Alliance (PQA) recently developed 3 quality measures for opioid prescribing: high dosages, multiple providers and pharmacies, and concurrent use of opioids and benzodiazepines.

Objective:

Our objective was to examine the prevalence of the PQA measures and identify the patient demographic and health characteristics associated with the measures.

Methods:

We conducted a cross-sectional analysis using Pennsylvania Medicaid data (2013–2015). We limited our analyses to non-cancer patients who were aged 18–64, and not dually Medicare/Medicaid eligible. Per PQA specifications, patients were required to possess ≥2 opioid prescriptions for ≥15 days annual supply each year. Outcome measures included: high dosages, defined as >120 morphine milligram equivalents for ≥90 consecutive days; multiple providers/pharmacies, defined as receiving opioid prescriptions from ≥4 providers and ≥4 pharmacies; and concurrent use of opioids and benzodiazepines, defined as ≥30 cumulative days of overlapping opioids and benzodiazepines among individuals having ≥2 opioids and ≥2 benzodiazepine fills. Patient characteristics assessed included demographics; other medication use; and physical, mental, and behavioral health comorbidities. We present descriptive and multivariable statistical analyses of the data to describe trends in quality measure prevalence and associations with enrollee health characteristics.

Results:

Numbers of enrollees meeting inclusion criteria ranged from 73,082 in 2013 to 85,710 in 2015. From 2013–2015, high dosage prevalence increased from 5.1 to 5.5%; multiple providers/pharmacies decreased from 7.1 to 5.0%, and concurrent use of opioids and benzodiazepines decreased from 29.1 to 28.4% (all p<0.05). A substantial portion of patients with >1 PQA measure from 2013–2015 were Medicaid eligible because of disability (41.8–81.9%). Enrollees with opioid use disorder were more likely to have high dosages (adjusted odds ratio [AOR]=2.01, 95% CI=1.83–2.21), and enrollees with anxiety and mood disorders were more likely to have multiple providers/pharmacies (anxiety: AOR=1.54, 95% CI=1.43–1.65; mood: AOR=1.15, 95% CI=1.06–1.25) and concurrent use of opioids and benzodiazepines (anxiety: AOR=3.50, 95% CI=3.38–3.63; mood: AOR=1.42, 95% CI=1.36–1.48).

Conclusions:

Given high levels of eligibility based on disability and the prevalence of mood, anxiety, and opioid use disorders among those identified by the quality measures; providers may require additional supports to care for the population identified by these measures.

Introduction

To address high rates of problematic opioid consumption and overdose mortality,1 US health systems are increasingly employing a variety of measures in administrative data to monitor patient risk from opioid medication exposure. The 3 most commonly used conceptual definitions of risk for opioid overdose relate to (1) high opioid dosage, measured in daily morphine equivalents;29 (2) indicators of “shopping,” measured by patients visiting multiple providers and/or pharmacists for opioids;1013 and (3) concurrent use of opioid medications with drugs that can heighten negative effects of opioids (e.g., benzodiazepines).1416

While there is broad consensus on the conceptual definition of these risk factors, they have been measured in widely varying forms. For instance, milligram morphine equivalents (MME) ≥100 per day have been observed within the literature to heighten overdose death risk.29 However, variations of this indicator have also been utilized, including doses as low as 9017 or as high as 12018 and 2007,19 morphine equivalents. In terms of shopping, definitions have included possessing narcotic prescriptions from 2,12 4,11 or 510 or more prescribers within 6 11 to 12 months of overdose and/or filling opioids at 312 or 411 or more pharmacies in a 3,12 6,11 or 1812 month period. Overlapping medications have been measured to include 2 or more pharmacy claims of opioids,13 overlapping long- and short-acting opioids,2,4 and overlapping opioids and sedatives (e.g. benzodiazepines).20

The Pharmacy Quality Alliance (PQA) is a national multi-stakeholder, consensus based- organization that has developed and disseminated a series of measures for monitoring medication utilization for many acute and chronic conditions with a focus on safety, adherence, and appropriateness.21 PQA measures have been adopted for use by the Centers on Medicare and Medicaid Services22 and numerous private organizations whose work focuses on system-level medication monitoring and management.23 The PQA has recently established 4 measures of the quality of opioid prescribing that correspond closely to the commonly utilized concepts of risk outlined above: (1) high opioid dosages, (2) multiple providers and multiple pharmacies,24 (3) high dosages/multi-providers, and (4) the concurrent use of opioids and benzodiazepines.25

These quality measures have also begun to receive additional support for their use. For instance, high opioid dosages and multiple providers and multiple pharmacies have been recently endorsed by the National Quality Forum as performance measures to address opioid misuse and abuse.26 Nevertheless, while the PQA measures may bring more uniformity to measurement, some health professionals have raised concerns about the implementation of measures with similar characteristics.27 Specifically, some are concerned about unintended consequences for pain treatment arising from systems’ use of these as thresholds for prescription and medication access restrictions.27 Further, given that findings from empirical studies describing the PQA measures have not previously been published, limited information is available about this patient population’s physical, mental, and behavioral health needs and what supports may be needed for providers to care for their needs.

The purpose of this project was to apply the PQA opioid quality measures using administrative claims and encounter data from the Pennsylvania (PA) Medicaid program from 2013 to 2015 in order to describe their prevalence, how prevalence overall and among subgroups has changed across time, and provide information regarding associations with demographic and health characteristics of patients who met criteria for these measures. Examination of these measures within Medicaid data is particularly important given serious concerns for opioid medication misuse and overdose events within this population2830 and may allow policymakers and payers to better plan resource allocation.

Methods

Data Source and Sample

To develop and examine the prevalence of the PQA measures, we conducted 3 cross-sectional analyses each year from 2013 to 2015. We used pharmacy claims from PA Medicaid to create the PQA measures, along with enrollment files, professional claims, and medical claims to create the cohort (Appendix 1) and measure characteristics of the population. The PA Medicaid program is among the largest in terms of expenditures and enrollment in the US, with about 2.9 million enrollees annually. In addition, PA’s healthcare utilization, access,31 and statewide demographic profile (with the exception of fewer Hispanic enrollees)32 are similar to those seen across the nation. PA also has opioid prescribing rates that are consistently above national averages,1,33 and Medicaid enrollees have generally been observed to have a number of opioid-related health problems and poor outcomes.30,3436 Therefore, PA Medicaid program data is an appropriate and valuable source for examining the PQA opioid measures. We obtained PA Medicaid data directly from the PA Department of Human Services for all fee-for-service (FFS) and managed care enrollees. This project was designated as exempt by the University of Pittsburgh Institutional Review Board.

Patients eligible for inclusion were identified following PQA specifications that comprise a population more likely to be using opioids for chronic rather than for acute pain conditions. Patients were excluded if they were <18 and >64 years, dually eligible for Medicaid and Medicare (given we could not observe the prescription claims for these patients), and had any cancer diagnoses during each calendar year. (International Classification of Disease, 9th Edition [ICD-9]: 140.0–239.9). Eligible patients were also required to have continuous Medicaid enrollment with no more than 1 gap of up to 45 days within a calendar year. The observation period for the measures was across 1 calendar year. Patients must have also had ≥2 documented prescription opioid medication fills on ≥2 separate days wherein the days supplied was ≥15 days during the calendar year period. The opioid medications included buprenorphine, codeine, dihydrocodeine, fentanyl, hydrocodone, hydromorphone, levorphanol, meperidine, methadone, morphine, oxycodone, oxymorphone, pentazocine, tapentadol, tramadol.24,25 Patients filling only injectable opioids, oral opioid cough products, and buprenorphine/naloxone products were not included.

PQA Opioid Measures

We examined 3 binary PQA opioid quality measures. The high opioid dosages measure is defined as those individuals who possessed a daily dosage >120 morphine MMEs for ≥90 consecutive days. We calculated daily MME based on the strength per unit, quantities dispensed and days supplied, and MME conversion factor of each opioid prescription (i.e., strength per unit × [quantities dispensed/ days supplied] × MME conversion factor). The multiple providers and multiple pharmacies measure is defined as individuals who received opioid prescriptions from ≥4 prescribers and who have filled their opioid medications at ≥4 pharmacies. The concurrent use of opioids and benzodiazepines measure (referred to as “opioids + benzodiazepine combination” throughout) is defined as individuals possessing a ≥30 day overlapping supply of opioids and benzodiazepines among individuals having ≥2 opioids and ≥2 benzodiazepine fills.25 We did not include the high dosages/multi-provider measure in our analyses.24 This measure was not included given its small prevalence (<1% of the cohort populations), resulting in prohibitively small cell sizes when comparing characteristics of enrollees meeting criteria for this measure.

Patient Descriptive Characteristics

Demographic and eligibility characteristics from the Medicaid enrollment file were included in the analyses to describe the patient population. We included participant age (18–29, 30–39, 40–49, 50–64), sex, race/ethnicity (white, black, Hispanic, other), urban/rural living location (Rural-Urban Continuum Codes37,38), eligibility category (disabled, newly eligible through Medicaid expansion [implemented in PA in 2015], other non-disabled adults), and dominant plan type (FFS, managed care).

In addition to demographic measures, we constructed several measures of comorbid health conditions during the calendar year. Using ICD-9,39 we included any diagnoses of anxiety or mood disorders. Using ICD-939 and −10,40 we also included the following individual indicators of underlying addiction separately in the descriptive and multivariate analyses: diagnoses for opioid use disorder, fatal and non-fatal heroin/opioid overdose (Appendix 2 for ICD-9/10 codes), use of medication assisted treatment (methadone maintenance [procedure codes H0020/J1230], buprenorphine use [identified by National Drug Codes as forms approved by US FDA for opioid use disorder], and naltrexone [identified by National Drug Codes]). We likewise reported numbers of opioid medications filled (i.e., list of opioids described above) for those with the 3 measures and numbers of benzodiazepine prescriptions filled for those with the opioids + benzodiazepine combination measure in the calendar year. We also included a modified Elixhauser comorbidity index41 (with mood and opioid use disorder removed from the index) based on ICD-9 and 10, depending on the month/year.

Statistical Analyses

We examined the prevalence of the 3 PQA measures each year and described patient characteristics using frequencies and percentages for patients with and without PQA measures. We employed 3 generalized estimating equation models (GEE) to estimate the difference in prevalence for each PQA measure for 2013 compared to 2015 (2015 as the dependent variable and 2013 as the comparison group). These models were adjusted for age, sex, race, and rural/urban living area. Chi-square difference and T-tests were used to assess bivariate differences for enrollee mental and behavioral health as well as for opioid and benzodiazepine filling patterns by PQA measure using data for the most recent year (2015). Lastly, we developed 3 multivariable logistic regression models to examine the cross-sectional associations between the health comorbidities and medication use described above and each of the 3 PQA measures using data from the most recent year (2015). All analyses were conducted in SAS 9.4.42

Results

Demographics

Table 1 displays the unadjusted overall prevalence and cohort demographics by quality measure. Approximately 5% of the cohort was identified as having high dosages of opioid medications (5.1% [n=3,753] in 2013 and 5.5% [n=4,708] in 2015. The prevalence of patients with multiple providers decreased across time, from 7.1% in 2013 (n=5,215) to 5% in 2015 (n=4,311). The prevalence of the opioids + benzodiazepine combination use was 29.1% (n=21,244) in 2013, 30% (n=21,153) in 2014, and 28.4% (n=24,346) in 2015. All of these changes were statistically significant when we compared 2015 to 2013 in the multivariate model, controlling for age, gender, race, and living area (Table 2).

Table 1.

Population characteristics by Pharmacy Quality Alliance opioid quality measures 2013–2015

High dosage Multiple providers Opioid + benzodiazepine
2013 2014 2015 2013 2014 2015 2013 2014 2015
Total patients 73,082 70,622 85,710 73,082 70,622 85,710 73,082 70,622 85,710
Overall cohort prevalence 3,753 (5.1) 3,823 (5.4) 4,708 (5.5) 5,215 (7.1) 3,692 (5.2) 4,311 (5.0) 21,244 (29.1) 21,153 (30.0) 24,346 (28.4)
Age group
 18–29 230 (6.1) 225 (5.9) 259 (5.5) 1,099 (21.1) 761 (20.6) 774 (18.0) 1,716 (8.1) 1,524 (7.2) 1,590 (6.5)
 30–39 717 (19.1) 715 (18.7) 923 (19.6) 1,534 (29.4) 1,144 (31.0) 1,383 (32.1) 4,265 (20.1) 4,285 (20.3) 4,997 (20.5)
 40–49 1,144 (30.5) 1,151 (30.1) 1,402 (29.8) 1,379 (26.4) 920 (24.9) 1,116 (25.9) 6,234 (29.3) 6,016 (28.4) 6,908 (28.4)
 50–64 1,662 (44.3) 1,732 (45.3) 2,124 (45.1) 1,203 (23.1) 867 (23.5) 1,038 (24.1) 9,029 (42.5) 9,328 (44.1) 10,851 (44.6)
Sex
 Female 2,059 (54.9) 2,065 (54.0) 2,477 (52.6) 3,521 (67.5) 2,539 (68.8) 2,838 (65.8) 14,785 (69.6) 14,738 (69.7) 16,840 (69.2)
 Male 1,694 (45.1) 1,758 (46.0) 2,231 (47.4) 1,694 (32.5) 1,153 (31.2) 1,473 (34.2) 6,459 (30.4) 6,415 (30.3) 7,506 (30.8)
Race/ethnicity
 White 2,938 (78.3) 2,875 (75.2) 3,442 (73.1) 3,032 (58.1) 2,222 (60.2) 2,631 (61.0) 14,770 (69.5) 14,484 (68.5) 16,897 (69.4)
 Black 628 (16.7) 754 (19.7) 1,019 (21.6) 1,737 (33.3) 1,152 (31.2) 1,323 (30.7) 3,961 (18.6) 4,122 (19.5) 4,607 (18.9)
 Hispanic 110 (2.9) 132 (3.5) 155 (3.3) 347 (6.7) 248 (6.7) 278 (6.4) 2,160 (10.2) 2,177 (10.3) 2,377 (9.8)
 Other 77 (2.1) 62 (1.6) 92 (2.0) 99 (1.9) 70 (1.9) 79 (1.8) 353 (1.7) 370 (1.7) 465 (1.9)
Living area
 Urban 3,085 (82.2) 3,170 (82.9) 3,894 (82.7) 4,646 (89.1) 3,225 (87.4) 3,775 (87.6) 17,690 (83.3) 17,656 (83.5) 20,283 (83.3)
 Rural 668 (17.8) 653 (17.1) 814 (17.3) 569 (10.9) 467 (12.6) 536 (12.4) 3,554 (16.7) 3,497 (16.5) 4,063 (16.7)
Type of eligibility
 Disabled 3,052 (81.3) 3,130 (81.9) 2,907 (61.7) 3,317 (63.6) 2,349 (63.6) 1,804 (41.8) 16,617 (78.2) 16,708 (79.0) 14,246 (58.5)
 Newly eligible NA a NA 1,101 (23.4) NA NA 1,479 (34.3) NA NA 6,205 (25.5)
 Non-disabled adults 701 (18.7) 693 (18.1) 700 (14.9) 1,898 (36.4) 1,342 (36.3) 1,028 (23.8) 4,627 (21.8) 4,438 (21.0) 3,895 (16.0)
Managed care 3,648 (97.2) 3,787 (99.1) 4,676 (99.3) 5,126 (98.3) 3,633 (98.4) 4,252 (98.6) 20,655 (97.2) 20,546 (97.1) 23,878 (98.1)
a

NA=Not applicable given Medicaid expansion took place in 2015 in Pennsylvania

Table 2:

Difference in prevalence for Pharmacy Quality Alliance measures between 2013a (N=73,082) and 2015 (N=85,710) b,c

N (%)
Estimated Adjusted Difference c (%) 95% CI d p e
2013 unadjusted prevalence 2015 unadjusted prevalence
High dosage 3,753 (5.1) 4,708 (5.5) 0.2 0.03 – 0.4 .02
Multiple providers 5,215 (7.1) 4,311 (5.0) −1.4 −1.7 – −1.2 <.001
Opioid + benzodiazepine 21,244 (29.1) 24,346 (28.4) −0.5 −0.8 – −0.1 .01
a

Reference group=2013.

b

30,103 patients were in both the 2013 and 2015 cohorts.

c

2015 was the year of Medicaid expansion, which resulted in an increase in enrollment.

d

GEE models were adjusted for age, sex, race, living area.

e

95% CI= 95% confidence interval.

fp=probability value comparing 203 vs. 2015.

Enrollees’ characteristics with each of the PQA measures were largely stable over the 3-year period. In 2015, a majority were female (range: 52.6% [high dosages] to 69.2% [opioids + benzodiazepine]), white (range: 61.0% [multiple providers] to 73.1% [high dosages]), and resided in urban areas (range: 82.7% [high dosages] to 87.6 [multiple providers]). A large share of enrollees with the measures was eligible due to disability (range: 41.8 [multiple providers] to 61.7 [high dosages]).

Health Comorbidities

We also examined cross-sectional differences for mental and behavioral health status of patients with and without the quality measures in the 2015 cohort (Table 3). The prevalence of anxiety disorders was significantly higher among those with multiple providers relative to their counterparts (53.3% vs. 34.4%, p<0.001). Anxiety disorders were also more than twice as prevalent among those with the opioid + benzodiazepine combination as those without (58.6% vs. 25.6%, p<0.001). Similarly, the prevalence of mood disorders was significantly higher among those with multiple providers and the opioid + benzodiazepine combination relative to those without the measures (59.8% vs. 44.2%, p<0.001 and 60.7% vs. 37.8%, p<0.001, respectively). Larger portions of enrollees with each quality measure compared to their counterparts also had opioid use disorder (high dosage: 21.8% vs. 11.7%; multiple providers: 24.4% vs. 11.6%; opioid + benzodiazepine: 15.6% vs. 10.9%; p<0.001 for each measure) and heroin/opioid overdose (high dosage: 2.0% vs. 1.1%; multiple providers: 2.7% vs. 1.1%; opioid + benzodiazepine: 1.8% vs. 0.9%; p<0.001 for each measure).

Table 3:

Bivariate description of behavioral health indicators and number of opioid fills among enrollees with the Pharmacy Quality Alliance measures, 2015

Characteristics High dosage Multiple providers Opioid + benzodiazepine

Yes, N (%) No, N (%) p Yes, N (%) No, N (%) p Yes, N (%) No, N (%) p
Total enrollees 4,708 81,002 4,311 76,319 24,346 61,364

Anxiety disorder 1,692 (35.9) 28,270 (34.9) .15 2,298 (53.3) 26,256 (34.4) <.001 14,266 (58.6) 15,696 (25.6) <.001
Mood disorder 1,919 (40.8) 36,077 (44.5) <.001 2,580 (59.8) 33,714 (44.2) <.001 14,785 (60.7) 23,211 (37.8) <.001
Opioid use disorder 1,028 (21.8) 9,442 (11.7) <.001 1,053 (24.4) 8,816 (11.6) <.001 3,789 (15.6) 6,681 (10.9) <.001
Heroin/opioid overdose 96 (2.0) 862 (1.1) <.001 115 (2.7) 806 (1.1) <.001 431 (1.8) 527 (0.9) <.001
Medication assisted treatment 140 (3.0) 3141 (3.9) .002 250 (5.8) 2,822 (3.7) <.001 1,064 (4.4) 2,217 (3.6) <.001
Antidepressant drug use 2,558 (54.3) 44,007 (54.3) .99 2,752 (63.8) 41,484 (54.4) <.001 17,263 (70.9) 29,302 (47.8) <.001
Number opioid fills a 22.0 (9.6) 9.0 (6.4) <.001 16.0 (8.2) 9.4 (7.0) <.001 13.2 (7.8) 8.3 (6.5) <.001
Number benzodiazepine fillsa - - - - - - 10.6 (5.0) 0.9 (2.6) <.001
Elixhauser Index a 3.6 (2.8) 3.5 (2.7) .04 4.7 (3.2) 3.5 (2.7) <.001 4.2 (2.8) 3.3 (2.7) <.001
a

Mean (Standard Deviation).

Medication Filling Patterns

We also examined medication use among patients with and without the PQA measures in 2015 (Table 3). Compared to those without the quality measures, larger portions of patients with multiple providers and an opioid + benzodiazepine combination were identified as receiving medication assisted treatment for opioid use disorder (multiple prescribers: 5.8% vs. 3.7%; opioid + benzodiazepine: 4.4% vs. 3.6%; p<0.001) and taking antidepressants (multiple prescribers: 63.8% vs. 54.4%; opioid + benzodiazepine: 70.9% vs. 47.8%; p<0.001). Patients with each of the measures also had a higher number of fills for opioid medications (range in differences of mean number of fills= 4.9–13.0, p<0.001) compared to those that did not have the quality measures.

Associations with Opioid Risks

Results from multivariable models of cross-sectional associations between demographic and health comorbidity measures and the opioid quality measures in 2015 are displayed in Table 4. In terms of increased likelihood for the measures, enrollees with opioid use disorder were more likely to have high dosages (adjusted odds ratio [AOR]=2.01, 95% CI=1.83–2.21), as were enrollees with heroin/opioid overdose (AOR=1.43, 95% CI=1.10–1.85). Enrollees with a higher number of opioid fills also had higher odds of high dosages (AOR=1.18, 95% CI=1.18–1.19). Enrollees with anxiety disorder were more likely to fill opioid prescriptions from multiple providers (AOR=1.54, 95% CI=1.43–1.65). Enrollees with opioid use disorder also had higher odds of multiple providers (AOR=1.43, 95% CI=1.31–1.56) as did enrollees residing in an urban area (AOR=1.38, 95% CI=1.25–1.52). Use of opioid + benzodiazepine combination was associated with diagnosis of anxiety disorder (AOR 3.50, 95% CI=3.38–3.63), use of antidepressants (AOR=1.53, 95% CI=1.47–1.59), and diagnoses of mood disorder (AOR=1.42, 95% CI=1.36–1.48).

Table 4:

Adjusted logistic regression estimates for three Pharmacy Quality Alliance indicators, 2015

High dosage (N=85,710) Multiple/pharmacies providers (N=80,630) Opioid + benzodiazepine (N=85,710)

Predictor AOR (95% CI) a pb AOR (95% CI) p AOR (95% CI) p
Age, years 1.01 (1.01 – 1.02 ) <.001 0.96 (0.95 – 0.96 ) <.001 1.03 (1.03 – 1.03 ) <.001
Female 0.63 (0.59 – 0.67 ) <.001 0.95 (0.89 – 1.02 ) .16 1.26 (1.22 – 1.31 ) <.001
Race (ref=White)
 Black 0.84 (0.77 – 0.91 ) <.001 1.53 (1.41 – 1.65 ) <.001 0.70 (0.67 – 0.73 ) <.001
 Hispanic 0.46 (0.38 – 0.55 ) <.001 0.85 (0.75 – 0.98 ) .02 1.22 (1.15 – 1.30 ) <.001
 Other 0.87 (0.68 – 1.11 ) .25 0.98 (0.77 – 1.25 ) .89 0.84 (0.75 – 0.95 ) .004
Urban 1.08 (0.99 – 1.19 ) .09 1.38 (1.25 – 1.52 ) <.001 1.15 (1.10 – 1.21 ) <.001
Eligibility (ref=Non-disable adults)
 Disabled 1.18 (1.06 – 1.32 ) .003 0.66 (0.60 – 0.73 ) <.001 1.28 (1.21 – 1.35 ) <.001
 Expansion 0.91 (0.82 – 1.02 ) .12 1.06 (0.97 – 1.16 ) .21 0.94 (0.89 – 0.99 ) .02
Heroin/opioid overdose 1.43 (1.10 – 1.85 ) .01 1.05 (0.84 – 1.31 ) .66 1.25 (1.08 – 1.45 ) .003
Opioid use disorder 2.01 (1.83 – 2.21 ) <.001 1.43 (1.31 – 1.57 ) <.001 1.04 (0.98 – 1.10 ) .18
Anxiety 0.99 (0.92 – 1.07 ) .77 1.54 (1.43 – 1.65 ) <.001 3.50 (3.38 – 3.63 ) <.001
Mood disorder 0.86 (0.79 – 0.93 ) <.001 1.15 (1.06 – 1.25 ) <.001 1.42 (1.36 – 1.48 ) <.001
Medication assisted treatment 0.83 (0.68 – 1.02 ) .08 1.09 (0.93 – 1.27 ) .28 1.22 (1.11 – 1.34 ) <.001
Antidepressant drug use 0.94 (0.87 – 1.01 ) .10 1.00 (0.93 – 1.08 ) .99 1.53 (1.47 – 1.59 ) <.001
Number of opioid fills 1.18 (1.18 – 1.19 ) <.001 1.10 (1.09 – 1.10 ) <.001 1.09 (1.09 – 1.10 ) <.001
Elixhauser index 0.91 (0.90 – 0.93 ) <.001 1.15 (1.13 – 1.16 ) <.001 0.97 (0.96 – 0.98 ) <.001
a

95% CI= 95% confidence interval.

b

p=probability value.

In terms of lower likelihood for having the measures, with the exception of slightly higher odds of having opioid + benzodiazepine combination among Hispanic enrollees (AOR=1.22, 95% CI=1.15 – 1.30), Hispanic and black enrollees were significantly less likely than white enrollees to have the PQA measures across all categories (p<0.05). Enrollees newly eligible for Medicaid in 2015 were less likely to have the opioid + benzodiazepine combination (AOR=0.94, 95% CI=0.89 – 0.99). Enrollees with a greater number of comorbidities, measured by the Elixhauser index, were less likely to have high dosages (AOR=0.91, 95% CI=0.90– 0.93) and opioid + benzodiazepine combination (AOR=0.97, 95% CI=0.96–0.98) but more likely to have multiple providers (AOR=1.15, 95% CI 1.13–1.16)..

Discussion

This study applied 3 opioid quality measures recently developed by the Pharmacy Quality Alliance to PA Medicaid program data from 2013 to 2015. These measures are based on previous research that has linked patient and prescriber behavior with increased risk for problematic patient-level outcomes, including overdose. Limited research is available regarding the prevalence and the characteristics of patients who will be potentially identified by these metrics. Our analyses show findings in 3 key areas for these measures in terms of comorbid health conditions among identified patients, trends showing improvements across time, and their consistency with previous research of patients with problematic opioid medication use.

First, study results showed there were high rates of mental and behavioral health conditions among those with the PQA measures. Approximately 60% of those with the opioid + benzodiazepine combination and multiple providers/pharmacies had mood disorders. Among those with the opioid + benzodiazepine combination, nearly 16% had opioid use disorders compared to almost 11% among those not meeting criteria for concurrent use—both rates of opioid use disorder being higher than general adult Medicaid population in Pennsylvania, which was 3.6% in 2007 and 4.5% in 2012.43 In the multivariable models, both mental and behavioral health disorders and overdose were also highly associated with the PQA measures.

These findings may signal need for improved communication and coordination between prescribers providing pain and mental health medications. The high rates of combination prescribing and high opioid dosages should be carefully examined and monitored by health systems in order to ensure patients are not exposed to unnecessary risks.44 Given the number of concomitant health conditions among identified patients—these measures may be utilized by payers to better target needed supports to those who care for these patients. Moreover, because of the limitations of drug monitoring programs to impact patient health beyond lowering prescribing/filling behaviors,4547 strategies to engage and direct patients to integrated care will be paramount given the apparent needs of these identified populations.48,49 Furthermore, these data also suggest the importance of risk adjustment when comparing these quality measures across populations. For instance, as payers compare prevalence of the PQA measures across plans, they will need to account for differences in prevalence of mental health conditions in order to not penalize plans that serve a disproportionate share of patients with these conditions.

Second, in terms of changes across study years, there were changes in the prevalence of enrollees accessing opioids from multiple providers/pharmacies and having an opioid + benzodiazepine combination from 2013 to 2015, which changes could appear to be relatively minor; however, they represent approximately 1,000 to 3,000 lives—a clinically significant amount of individuals overall. Increasingly stringent laws, formulary management, and public awareness in the state could have helped stimulate these shifts.50 There was a divergent trend in the prevalence of high dosages, which increased to a small degree during the observation period. However, these findings are congruent with recent research that has shown that dispensing of opioids in most US states51 and negative outcomes related to the opioid epidemic1 have also had documented increases during comparable years.

Lastly, results from these analyses are consistent with previous research among patients with problematic opioid medication use and help point to the face and criterion validity of the PQA metrics and their value as measures of prescribing quality across US health plans. The largest proportions of patients positive for these measures were white, lived in urban areas of the state, and were female. Previous research has observed that problematic opioid medication use, prescribing,52 and misuse53 are more prevalent among whites compared to other races/ethnicities, and higher rates of overdose also are more likely to occur among white individuals.29,54 Urban residents likewise have been noted to have higher rates of prescription opioid misuse compared to rural residents.55 In addition, the largest portions of the study sample were eligible for the Medicaid program because of disability, which has been noted to be among the characteristics of individuals who experience overdose.56 Previous research has also documented mood, anxiety, and opioid use disorders are associated with misuse30 and overdose.29 Finally, we found that the newly eligible enrollees had a lower likelihood of having an opioid + benzodiazepine combination, and no significant relationship between enrollment in Medicaid due to the expansion and the other PQA measures. This finding adds to the literature on the role of Medicaid expansion in addressing the opioid crisis.57

Limitations

Although utilization, access,31 and demographics 32 of the PA Medicaid program are similar to other states in the US, the results herein are nonetheless from a single state among a population that have differing needs from the general population, which limit their generalizability. In addition, while these measures are based on evidence and show initial validity for monitoring problematic opioid consumption and behavior, their current construction should continue to be validated. We also recognize our study cohorts and analytical approach are largely descriptive, and causation cannot be inferred nor can we fully control for unmeasured confounders. Our analyses were not able to take into account policies, social, and economic factors that may have influenced the results. The purpose of the current project was not to, however, assess causal inference. Rather, the purpose of this project was to be descriptive in order to increase understanding around the prevalence and characteristics of enrollees positive for these recently developed PQA measures of quality. Future work should seek to employ the PQA measures within research designs, such as difference-in-differences analyses, which have greater ability to infer causal relationships.

Conclusion

As problematic opioid use and overdose continues to take a serious toll on states within the US, health care payers and systems have the important burden of continually monitoring patient risk. The PQA has set forth 3 research-based quality measures that have the potential to be implemented in pharmacy claims data for health surveillance. Through concerted and coordinated surveillance, health systems and payers stand to make a major contribution to confronting the opioid epidemic though monitoring patient risk across systems.

Supplementary Material

Appendix1
Appendix2

Summary Bullets.

What is already known about this subject

  • Continuous monitoring of opioid use using quality metrics is paramount for health care systems and payers.

  • Current opioid use measures employed in the field have been measured inconsistently.

  • The recently developed Pharmacy Quality Alliance measures, two of which that have been endorsed by the National Quality Forum in 2017, have the potential to better standardize measurement of opioid prescribing across health systems.

What this study adds

  • This study measures the prevalence of the Pharmacy Quality Alliance measures in a Medicaid enrolled population using recent data.

  • This study describes the prevalence of physical, mental, and behavioral health characteristics in a Medicaid population with high opioid dosages, use of multiple providers and pharmacies, and concurrent use of opioids and benzodiazepines.

Acknowledgments

Funding

This project was supported by a grant from the Centers for Disease Control and Prevention (U01CE002496). This work was also supported by an intergovernmental agreement between the Pennsylvania Department of Human Services and the University of Pittsburgh. Dr. Lo-Ciganic was supported by the University of Arizona Health Sciences Career Development Award. The conclusions, findings, and opinions expressed by authors contributing to this article do not necessarily reflect the official position of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the Commonwealth of Pennsylvania.

Dr. Cochran reports grants from CDC, other from PA Dept of Human Services, during the conduct of the study.

Dr. Lo-Ciganic has nothing to disclose.

Dr. Gellad reports grants from CDC, during the conduct of the study.

Dr. Gordon reports grants from CDC, grants from PA Medicaid, during the conduct of the study.

Dr. Frazier has nothing to disclose.

Dr. Cole has nothing to disclose.

Dr. Lobo has nothing to disclose.

Dr. Zheng has nothing to disclose.

Dr. Chang has nothing to disclose.

Dr. Kelley has nothing to disclose.

Dr. Donohue reports support from Pennsylvania Department of Human Services, during the conduct of the study.

Footnotes

Previous presentation

A portion of these results were presented at the annual meeting of the Association for Medical Education in Substance Abuse in November 2017.

Contributor Information

Gerald Cochran, University of Pittsburgh, School of Social Work, University of Pittsburgh, School of Medicine, Department of Psychiatry, University of Pittsburgh, Center for Pharmaceutical Policy and Prescribing, 4200 Forbes Ave., 2117 CL, Pittsburgh, PA 15260, Phone: (412) 642-2325, Fax: (412) 624-6323, gcochran@pitt.edu.

Wei-Hsuan Lo-Ciganic, University of Arizona College of Pharmacy, lociganic@pharmacy.arizona.edu, (520) 626-9535.

Walid F. Gellad, VA Pittsburgh Healthcare System, University of Pittsburgh, School of Medicine, Division of General Internal Medicine, University of Pittsburgh, Center for Pharmaceutical Policy and Prescribing, walid.gellad@pitt.edu, (412) 360-2267.

Adam J. Gordon, University of Utah, School of Medicine, Division of General Internal Medicine, VA Salt Lake City Healthcare System, Adam.Gordon@hsc.utah.edu, (801) 581-2121.

Evan Cole, University of Pittsburgh, Graduate School of Public Health, evancole@pitt.edu, (412) 383-2087.

Carroline Lobo, University of Pittsburgh, Graduate School of Public Health, cpl13@pitt.edu, (412) 624-3001.

Winfred Frazier, University of Pittsburgh, School of Medicine, Division of General Internal Medicine, winfred.frazier@gmail.com, (409) 772-2166.

Ping Zheng, University of Pittsburgh, Graduate School of Public Health, pzheng2013@gmail.com, (412) 624-3001.

Chung-Chou H. Chang, University of Pittsburgh, School of Medicine, Division of General Internal Medicine, changjh@upmc.edu, (412) 692-4868.

David Kelley, Chief Medical Officer, Medical Assistance Programs, Pennsylvania Department of Human Services, c-dakelley@pa.gov, (717) 787-1870.

Julie M. Donohue, University of Pittsburgh, Graduate School of Public Health, University of Pittsburgh, Center for Pharmaceutical Policy and Prescribing, jdonohue@pitt.edu, (412) 624-4562.

References

  • 1.Rudd RA, Aleshire N, Zibbell JE, Gladden RM. Increases in Drug and Opioid Overdose Deaths--United States, 2000–2014. MMWR Morb Mortal Wkly Rep 2016;64(50–51):1378–1382. [DOI] [PubMed] [Google Scholar]
  • 2.Logan J, Liu Y, Paulozzi L, Zhang K, Jones C. Opioid prescribing in emergency departments: the prevalence of potentially inappropriate prescribing and misuse. Med Care 2013;51(8):646–653. [DOI] [PubMed] [Google Scholar]
  • 3.Mack KA, Zhang K, Paulozzi L, Jones C. Prescription practices involving opioid analgesics among Americans with Medicaid, 2010. J Health Care Poor Underserved 2015;26(1):182–198. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Liu Y, Logan JE, Paulozzi LJ, Zhang K, Jones CM. Potential misuse and inappropriate prescription practices involving opioid analgesics. Am J Manag Care 2013;19(8):648–665. [PubMed] [Google Scholar]
  • 5.Bohnert ASB, Valenstein M, Bair MJ, et al. Association between opioid prescribing patterns and opioid overdose-related deaths. JAMA 2011;305(13):1315–1321. [DOI] [PubMed] [Google Scholar]
  • 6.Paulozzi LJ, Strickler GK, Kreiner PW, Koris CM. Controlled Substance Prescribing Patterns--Prescription Behavior Surveillance System, Eight States, 2013. MMWR Surveill Summ 2015;64(9):1–14. [DOI] [PubMed] [Google Scholar]
  • 7.Dasgupta N, Funk MJ, Proescholdbell S, Hirsch A, Ribisl KM, Marshall S. Cohort Study of the Impact of High-dose Opioid Analgesics on Overdose Mortality. Pain Med 2015. [DOI] [PubMed]
  • 8.Gwira Baumblatt JA, Wiedeman C, Dunn JR, Schaffner W, Paulozzi LJ, Jones TF. High-risk use by patients prescribed opioids for pain and its role in overdose deaths. JAMA Intern Med 2014;174(5):796–801. [DOI] [PubMed] [Google Scholar]
  • 9.Dunn KM, Saunders KW, Rutter CM, et al. Opioid prescriptions for chronic pain and overdose: a cohort study. Ann Intern Med 2010;152(2):85–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Hall AJ, Logan JE, Toblin RL, et al. Patterns of abuse among unintentional pharmaceutical overdose fatalities. JAMA 2008;300(22):2613–2620. [DOI] [PubMed] [Google Scholar]
  • 11.Peirce GL, Smith MJ, Abate MA, Halverson J. Doctor and pharmacy shopping for controlled substances. Med Care 2012;50(6):494–500. [DOI] [PubMed] [Google Scholar]
  • 12.Cepeda MS, Fife D, Chow W, Mastrogiovanni G, Henderson SC. Assessing opioid shopping behaviour: a large cohort study from a medication dispensing database in the US. Drug saf 2012;35(4):325–334. [DOI] [PubMed] [Google Scholar]
  • 13.Yang Z, Wilsey B, Bohm M, et al. Defining risk of prescription opioid overdose: pharmacy shopping and overlapping prescriptions among long-term opioid users in medicaid. J Pain 2015;16(5):445–453. [DOI] [PubMed] [Google Scholar]
  • 14.Chou R, Deyo R, Devine B, et al. The Effectiveness and Risks of Long-Term Opioid Treatment of Chronic Pain. Evidence Report/Technology Assessment No. 218. (Prepared by the Pacific Northwest Evidence-based Practice Center under Contract No. 290-2012-00014-I.) AHRQ Publication No. 14-E005-EF Rockville, MD: Agency for Healthcare Research and Quality;2014. [Google Scholar]
  • 15.Brands B, Blake J, Marsh DC, Sproule B, Jeyapalan R, Li S. The impact of benzodiazepine use on methadone maintenance treatment outcomes. J Addict Dis 2008;27(3):37–48. [DOI] [PubMed] [Google Scholar]
  • 16.Kuerbis A, Sacco P, Blazer DG, Moore AA. Substance abuse among older adults. Clinics in geriatric medicine 2014;30(3):629–654. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Becker WC, Fenton BT, Brandt CA, et al. Multiple Sources of Prescription Payment and Risky Opioid Therapy Among Veterans. Med Care 2017. [DOI] [PubMed]
  • 18.Braden JB, Russo J, Fan MY, et al. Emergency department visits among recipients of chronic opioid therapy. Arch Intern Med 2010;170(16):1425–1432. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Oliva EM, Bowe T, Tavakoli S, et al. Development and applications of the Veterans Health Administration’s Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide. Psychol Serv 2017;14(1):34–49. [DOI] [PubMed] [Google Scholar]
  • 20.Bernardy NC, Lund BC, Alexander B, Friedman MJ. Increased polysedative use in veterans with posttraumatic stress disorder. Pain Med 2014;15(7):1083–1090. [DOI] [PubMed] [Google Scholar]
  • 21.Pharmacy Quality Aliance. Pharmacy Quality Aliance Performance Measures 2017; http://pqaalliance.org/measures/default.asp. Accessed April 19, 2017.
  • 22.Pharmacy Quality Aliance. Pharmacy Quality Aliance Measures Used By CMS in the Star Ratings 2017; http://pqaalliance.org/measures/cms.asp. Accessed April 19, 2017.
  • 23.Pharmacy Quality Aliance. Licensing Pharmacy Quality Aliance Measures 2017; http://pqaalliance.org/measures/licensing.asp. Accessed April 19, 2017.
  • 24.Pharmacy Quality Aliance. Use of Opioids from Multiple Providers or at High Dosage in Persons Without Cancer Springfield, VA: Pharmacy Quality Aliance;2016. [Google Scholar]
  • 25.Pharmacy Quality Aliance. Concurrent Use of Opioids and Benzodiazepines Springfield, VA: Pharmacy Quality Alliance;n.d. [Google Scholar]
  • 26.Pharmacy Quality Aliance Receives NQF Endorsement of Three Performance Measures to Address Opioid Misuse/Abuse [press release] Alexandria, VA: Pharmacy Quality Aliance;2017. [Google Scholar]
  • 27.Kertesz SG. An Opioid Quality Metric Based on Dose Alone? 80 Professionals Respond to NCQA n.d.; https://medium.com/@StefanKertesz/an-opioid-quality-metric-based-on-dose-alone-80-professionals-respond-to-ncqa-6f9fbaa2338. Accessed April 10, 2017.
  • 28.Sharp M, Melnick T. Poisoning Deaths Involving Opioid Analgesics — New York State, 2003–2012. MMWR Morb Mortal Wkly Rep 2015;64(14):377–380. [PMC free article] [PubMed] [Google Scholar]
  • 29.Cochran G, Gordon AJ, Lo-Ciganic WH, et al. An Examination of Claims-based Predictors of Overdose from a Large Medicaid Program. Med Care 2016. [DOI] [PMC free article] [PubMed]
  • 30.Sullivan MD, Edlund MJ, Fan M-Y, Devries A, Brennan Braden J, Martin BC. Risks for possible and probable opioid misuse among recipients of chronic opioid therapy in commercial and medicaid insurance plans: The TROUP Study. Pain 2010;150(2):332–339. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Centers for Disease Control. Health, United States 2012: With Special Feature on Emergency Care. Hyattsville, MD: Centers for Disease Control;2013. [Google Scholar]
  • 32.Pating DR, Miller MM, Goplerud E, Martin J, Ziedonis DM. New systems of care for substance use disorders: treatment, finance, and technology under health care reform. Psychiatr Clin North Am 2012;35(2):327–356. [DOI] [PubMed] [Google Scholar]
  • 33.Centers for Disease Control. Prevention Status Report 2013: Prescription Drug Overdose, Pennsylvania Atlanta, GA: Centers for Disease Control;2013. [Google Scholar]
  • 34.Centers for Disease Control. Overdose deaths involving prescription opioids among Medicaid enrollees - Washington, 2004–2007. MMWR Morb Mortal Wkly Rep 2009;58(42):1171–1175. [PubMed] [Google Scholar]
  • 35.Edlund MJ, Martin BC, Fan M-Y, Braden JB, Devries A, Sullivan MD. An analysis of heavy utilizers of opioids for chronic noncancer pain in the TROUP study. J Pain Symptom Manage 2010;40(2):279–289. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Faul M, Bohm M, Alexander C. Methadone Prescribing and Overdose and the Association with Medicaid Preferred Drug List Policies - United States, 2007–2014. MMWR Morb Mortal Wkly Rep 2017;66(12):320–323. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.United States Department of Agriculture. Rural-Urban Continuum Codes 2013; http://www.ers.usda.gov/data-products/rural-urban-continuum-codes/documentation.aspx#.VC61zGddUrA. Accessed October 3, 2014.
  • 38.Baldwin LM, Andrilla CH, Porter MP, Rosenblatt RA, Patel S, Doescher MP. Treatment of early-stage prostate cancer among rural and urban patients. Cancer 2013;119(16):3067–3075. [DOI] [PubMed] [Google Scholar]
  • 39.World Health Organization; International Statistical Classification of Diseases and Related Health Problems, 9th Edition (ICD-9) Geneva: World Health Organization; 2011. [Google Scholar]
  • 40.World Health Organization; International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10). Geneva: World Health Organization; 2016. [Google Scholar]
  • 41.University of Manitoba. (2016). Concept: Elixhauser Comorbidity Index http://libguides.gwumc.edu/c.php?g=27773&p=170272. Accessed January 20, 2018.
  • 42.SAS 9.4 [computer program] Cary, NC;2013. [Google Scholar]
  • 43.Gordon AJ, Lo-Ciganic WH, Cochran G, et al. Patterns and Quality of Buprenorphine Opioid Agonist Treatment in a Large Medicaid Program. J Addict Med 2015;9(6):470–477. [DOI] [PubMed] [Google Scholar]
  • 44.DuPont RL. “Should Patients With Substance Use Disorders Be Prescribed Benzodiazepines?” No. J Addict Med 2017;11(2):84–86. [DOI] [PubMed] [Google Scholar]
  • 45.Bao Y, Pan Y, Taylor A, et al. Prescription Drug Monitoring Programs Are Associated With Sustained Reductions In Opioid Prescribing By Physicians. Health Aff 2016;35(6):1045–1051. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Paulozzi LJ, Kilbourne EM, Desai HA. Prescription Drug Monitoring Programs and Death Rates from Drug Overdose. Pain Med 2011;12(5):747–754. [DOI] [PubMed] [Google Scholar]
  • 47.Ali MM, Dowd WN, Classen T, Mutter R, Novak SP. Prescription drug monitoring programs, nonmedical use of prescription drugs, and heroin use: Evidence from the National Survey of Drug Use and Health. Addict Behav 2017;69:65–77. [DOI] [PubMed] [Google Scholar]
  • 48.Cochran G, Gordon AJ, Field C, et al. Developing a framework of care for opioid medication misuse in community pharmacy. Res Social Adm Pharm 2016;12(2):293–301. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Substance Abuse and Mental Health Services Administration. Behavioral Health Homes for People with Mental Health and Substance Use Conditions Washington DC: SAMHSA-HRSA Center for Integrated Health Solutions 2012. [Google Scholar]
  • 50.Katz NP, Birnbaum H, Brennan MJ, et al. Prescription opioid abuse: challenges and opportunities for payers. Am J Manag Care 2013;19(4):295–302. [PMC free article] [PubMed] [Google Scholar]
  • 51.Mack KA, Jones CM, McClure RJ. Physician dispensing of oxycodone and other commonly used opioids, 2000–2015, United States. Pain Med 2017. [DOI] [PMC free article] [PubMed]
  • 52.Pletcher MJ, Kertesz SG, Kohn MA, Gonzales R. Trends in opioid prescribing by race/ethnicity for patients seeking care in US emergency departments. JAMA 2008;299(1):70–78. [DOI] [PubMed] [Google Scholar]
  • 53.Wang KH, Becker WC, Fiellin DA. Prevalence and correlates for nonmedical use of prescription opioids among urban and rural residents. Drug Alcohol Depend 2013;127(1–3):156–162. [DOI] [PubMed] [Google Scholar]
  • 54.Unick GJ, Rosenblum D, Mars S, Ciccarone D. Intertwined epidemics: national demographic trends in hospitalizations for heroin- and opioid-related overdoses, 1993–2009. Plos One 2013;8(2):e54496–e54496. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Rigg KK, Monnat SM. Urban vs. rural differences in prescription opioid misuse among adults in the United States: Informing region specific drug policies and interventions. Int J Drug Policy 2015;26(5):484–491. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Johnson EM, Lanier WA, Merrill RM, et al. Unintentional prescription opioid-related overdose deaths: description of decedents by next of kin or best contact, Utah, 2008–2009. J Gen Intern Med 2013;28(4):522–529. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57.McKenna RM. Treatment use, sources of payment, and financial barriers to treatment among individuals with opioid use disorder following the national implementation of the ACA. Drug Alcohol Depend 2017;179:87–92. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Appendix1
Appendix2

RESOURCES