Appendix Table 2.
ROB Assessment in Studies That Reported on the Association Between PDMPs and Nonfatal and Fatal Drug Overdoses
| Criteria for ROB Assessment in Studies That Reported on PDMP Effects |
Study, Year (Reference) | |||
|---|---|---|---|---|
| Maughan, 2015 (31) | Bachhuber, 2016 (30) | Brown, 2017 (23) | Pauly, 2018 (28) | |
|
| ||||
| Bias due to confounding | Serious – GEE model adjusted for calendar quarter, metropolitan area, interaction between calendar quarter and metropolitan area, area unemployment rate. Inadequate adjustment for co-implemented policies | Serious – GEE model adjusted for calendar quarter, metropolitan area, interaction between calendar quarter and metropolitan area, area unemployment rate. Inadequate adjustment for co-implemented policies | Critical – Failure to account for competing interventions that might have affected the rate of nonfatal and fatal opioid-related overdose events is likely to differentially affect the pre-intervention or post-intervention rate of events. Focus on slopes, instead of intercept, increases risk for time-varying confounding. | Serious – GEE model adjusted for time, geographic region, rate of diagnosed substance use disorder, percentage of population male, percentage aged 25–35 years, and insured population counts. Inadequate adjustment for time-invariant factors and co-implemented policies. |
|
| ||||
| Bias in selection of participants into the study | Low – Selection of metropolitan areas based on data availability | Low – Selection of metropolitan areas based on data availability | Low – Single state examined pre-/post-intervention | Moderate – Data from Truven Health MarketScan administrative claims data. Sample is representative of the privately insured and employed U.S. population |
|
| ||||
| Bias in classification of interventions | Low – Intervention was clearly defined. | Low – Intervention was clearly defined. | Low – Intervention was clearly defined. | Low – Data from NAMSDL and PDAPS |
|
| ||||
| Bias due to deviations from intended interventions | Low – No deviation from the intended intervention. | Low – No deviation from the intended intervention. | Low – No deviation from the intended intervention. | Low – No deviation from the intended intervention. |
|
| ||||
| Bias due to missing data | Low – No missing data were reported. | Low – No missing data were reported. | Low – No missing data were reported. | Low – No missing data were reported. |
|
| ||||
| Bias in measurement of outcomes | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy |
|
| ||||
| Bias in selection of the reported results | Low – Expected analyses were reported. | Low – Expected analyses were reported. | Moderate – Post-implementation change in intercept was not reported. | Low – Expected analyses were reported. |
|
| ||||
| Overall bias | Serious – Inadequate adjustment of time-invariant and time-varying factor | Serious – Inadequate adjustment of time-invariant and time-varying factor | Critical – Inadequate adjustment for key competing interventions. No alternative specifications were explored, e.g., date of legislation vs. implementation. | Serious – Inadequate adjustment of time-invariant and time-varying factors; sample unlikely to represent U.S. population. |
| Paulozzi, 2011 (34) | Kim, 2013 (19) | Li, 2014 (35) | Delcher, 2015 (29) and Delcher, 2016 (36) | |
|
| ||||
| Bias due to confounding | Moderate – State fixed-effects adjust for time-invariant differences and adjustment for time, state, and spatial autocorrelation; however, inadequate adjustment for co-implemented policies | Moderate – State fixed-effects adjust for time-invariant differences, adjustment for demographic, socioeconomic status, health/health care, and gun control laws; however, limited adjustment for time-varying factors | Critical – GEE model adjusted for calendar quarter, demographic characteristics (percentage male, percentage aged 35–54 years, percentage white), geographic region, unemployment rate. Inadequate adjustment for time-invariant state differences and co-implemented policies | Serious – Misspecification of ARIMA model poses serious risk; however, numerous sensitivity analyses were performed; minimal adjustment for co-implemented policies except pill mill laws. |
|
| ||||
| Bias in selection of participants into the study | Low – Selection of 50 U.S. states and D.C. | Low – Selection of 50 U.S. states and D.C. | Low – Selection of 50 U.S. states and D.C. | Moderate – Study limited to state of Florida, unlikely to represent U.S. effect |
|
| ||||
| Bias in classification of interventions | Low – Intervention determined from IJIS Institute. | Low – Intervention determined from IJIS Institute and NAMSDL | Low – Intervention from DEA | Low – Implementation date ascertained from state |
|
| ||||
| Bias due to deviations from intended interventions | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention |
|
| ||||
| Bias due to missing data | Low – No missing data were reported. | Low – No missing data were reported. | Low – No missing data were reported. | Low – Minimal missing data |
|
| ||||
| Bias in measurement of outcomes | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy |
|
| ||||
| Bias in selection of the reported results | Low – Expected analyses were reported. | Low – Expected analyses were reported. | Low – Expected analyses were reported. | Low – Expected analyses were reported. |
|
| ||||
| Overall bias | Moderate – Inadequate adjustment of time-varying factor, but assessment of spatial autocorrelation | Moderate – Inadequate adjustment of time-varying factor | Serious – Inadequate adjustment of time-invariant and time-varying factors | Serious – Misspecification of ARIMA model will produce bias estimates; sensitivity analyses reduce some ROB. |
| Radakrishnan, 2015 (21) | Kilby, 2015 (32) | Patrick, 2016 (27) | Birk, 2017 (20) | |
|
| ||||
| Bias due to confounding | Low – State and year fixed-effects adjust for time-invariant differences; robust adjustment for sociodemographics and adjustment for co-implemented policies | Moderate – State and fixed-effects adjust for time-invariant differences, adjustment for unemployment rate and population over 60 years; limited adjustment for time-varying factors | Moderate – State fixed-effects adjust for time-invariant differences, adjusted for unemployment, education attainment rate; extensive sensitivity analyses | Low – State fixed-effects adjust for time-invariant differences; adjustment for pill mill laws, Good Samaritan laws, naloxone distribution, and medical marijuana legalization |
|
| ||||
| Bias in selection of participants into the study | Low – Selection of 50 U.S. states and D.C. | Moderate – 12 states that adopted PDMPs before 2003 were excluded. | Low – 15 states that did not implement a law during study period were excluded. | Low – Selection of 50 U.S. states and D.C. |
|
| ||||
| Bias in classification of interventions | Low – Intervention determined from NAMSDL | Low – Intervention determined from NAMSDL | Low – Intervention from Law Atlas and NAMSDL | Low – Intervention determined from The Network of Public Health Laws and the PDMP Center for Excellence |
|
| ||||
| Bias due to deviations from intended interventions | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention |
|
| ||||
| Bias due to missing data | Low – No missing data were reported. | Moderate – 12 states that adopted PDMP before 2003 | Moderate – Excluded Florida and West Virginia due to influence of co-implemented laws; some suppressed data, multiple imputation | Low – No missing data were reported. |
|
| ||||
| Bias in measurement of outcomes | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy |
|
| ||||
| Bias in selection of the reported results | Low – Expected analyses were reported. | Low – Expected analyses were reported. | Low – Expected analyses were reported. | Low – Expected analyses were reported. |
|
| ||||
| Overall bias | Low – Adequate adjustment for time-invariant factors (fixed effects) and robust control for co-implemented policies | Moderate – Inadequate adjustment of time-varying factor; exclusion of 12 early-adopter states | Moderate – Inadequate adjustment of time-varying factors; exclusion of states with time-varying confounding instead of adjustment | Low – Robust adjustment of confounding and sensitivity analyses |
| Nam, 2017 (33) | Meinhofer, 2017 (22) | Dowell, 2016 (25) | Pardo, 2017 (26) | |
|
| ||||
| Bias due to confounding | Moderate – State and year fixed-effects adjust for time-invariant differences and secular trends; adjustment for sociodemographics and health care access; however, inadequate adjustment for co-implemented policies | Moderate – State and year fixed-effects adjust for time-invariant differences and secular trends; however, adjustment for co-implemented policies | Moderate – State and year fixed-effects adjust for time-invariant differences and secular trends, adjustment for opioid prescribing rate, pending death rates; however, inadequate adjustment for co-implemented policies | Low – State and year fixed-effects adjust for time-invariant differences and secular trends; adjustment for pill mill laws, Good Samaritan laws, naloxone distribution, and medical marijuana legalization |
|
| ||||
| Bias in selection of participants into the study | Moderate – 15 states that adopted PDMPs before 2000 were excluded. | Low – Selection of 50 U.S. states and D.C. | Low – 12 states were excluded for low outcome counts and complex policy situations. | Low – Selection of 50 U.S. states and D.C. |
|
| ||||
| Bias in classification of interventions | Moderate – No information on how date of intervention was determined | Low – Intervention determined from NAMSDL, TTAC, and ONCHIT | Low – Intervention from WestLaw, TTAC, and NAMSDL | Critical – The continuous score that was the primary independent variable of interest was determined on the basis of factors previously found to reduce the outcome. |
|
| ||||
| Bias due to deviations from intended interventions | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention | Low – No deviation from the intended intervention |
|
| ||||
| Bias due to missing data | Moderate – Some data suppressed due to small counts; 15 states that adopted PDMPs before 2000 were excluded. | Moderate – States with zero counts were imputed 1 dead. | Moderate – 12 states with low number of deaths were excluded. | Low – No missing data were reported. |
|
| ||||
| Bias in measurement of outcomes | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy | Low – Measurement of outcome independent of policy |
|
| ||||
| Bias in selection of the reported results | Low – Expected analyses were reported. | Low – Expected analyses were reported. | Low – Expected analyses were reported. | Serious – The use of a subjective exposure increases the likelihood that many analyses were conducted to determine the best model; however, no information on such analyses was reported. |
|
| ||||
| Overall bias | Moderate – Inadequate adjustment for time-varying factors, exclusion of states that adopted PDMP before 2000 | Moderate – Inadequate adjustment of time-varying factor; unexpected handling of missing data | Serious – Inadequate adjustment of time-varying factor; unexpected handling of missing data | Serious – Subjective measure of intervention; inadequate information on multiple tests; robust adjustment of confounding |
| Phillips, 2017 (24) | – | – | – | |
|
| ||||
| Bias due to confounding | Critical – Random state and year variable fail to account for time invariant differences among states or secular trends; adjustment for education, unemployment, percentage of population on disability, and medical marijuana laws; however, no adjustment for other time-varying policies | – | – | – |
|
| ||||
| Bias in selection of participants into the study | Low – Selection of 50 U.S. states and D.C. | – | – | – |
|
| ||||
| Bias in classification of interventions | Low – Intervention determined from NAMSDL | – | – | – |
|
| ||||
| Bias due to deviations from intended interventions | Low – No deviation from the intended intervention | – | – | – |
|
| ||||
| Bias due to missing data | Low – No missing data were reported. | – | – | – |
|
| ||||
| Bias in measurement of outcomes | Low – Measurement of outcome independent of policy | – | – | – |
|
| ||||
| Bias in selection of the reported results | Low – Expected analyses were reported. | – | – | – |
|
| ||||
| Overall bias | Critical – Inadequate adjustment of time-invariant and time-varying factor | – | – | – |
ARIMA = autoregressive integrated moving average; DEA = U.S. Drug Enforcement Administration; GEE = generalized estimating equation; IJIS = Integrated Justice Information Systems; NAMSDL = National Alliance for Model State Drug Laws; ONCHIT = Office of the National Coordinator for Health Information Technology; PDAPS = Prescription Drug Abuse Policy System; PDMP = prescription drug monitoring program; ROB = risk of bias; TTAC = Brandeis University's PDMP Training and Technical Assistance Center.