Principles and Metrics for Evaluating Oregon’s Innovative Drug Decriminalization Measure

February 1, 2022 marks one year since the enactment of a historic measure in Oregon to decriminalize possession of small amounts of drugs and substantially invest in healthcare, substance use disorder (SUD) treatment, harm reduction, and social services for people who use drugs (PWUD). A departure from decades of punitive approaches, Measure 110 (M110; also known as the Drug Addiction Treatment and Recovery Act of 2020) [1] is being carefully followed by policymakers and advocates across the USA and around the world.

The success or failure of M110 has the potential to shape drug policy in the USA for decades to come, but “success” or “failure” is entirely dependent on what outcomes are being measured, how the data are gathered, and whether the findings are understood within the broader context of what is happening on the ground in Oregon. Evaluations of novel drug policies like M110 are critical for informing future policy, yet often lack engagement with the people who are the most directly impacted, despite their having valuable insight into what should be measured and how. As a working group of researchers and service providers, we came together to interview people who use drugs in Oregon to get their input into principles for how to evaluate M110 as well as the most important outcomes to measure. In doing so, we learned critical information about the approach needed for a robust evaluation, generated a broad array of relevant metrics, and garnered important information about the context of M110 implementation that will help researchers and policymakers interpret findings.

M110 decriminalized personal possession of small amounts of drugs while expanding access to addiction recovery and harm reduction services through new investments of US $302 million over 2 years. The maximum punishment for people possessing small amounts of drugs is a citation and US $100 fine, which is waived if they choose to participate in a social services screening through which they can be linked to services such as SUD treatment, harm reduction, and housing assistance. These are unprecedented and sweeping changes to drug policy.

While the impact of this novel drug policy measure could not possibly be meaningfully evaluated less than a year after its enactment, in October of 2021, media outlets such as Oregon Public Broadcasting began reporting on the low numbers of citations being issued and calling into question the success of the measure [2]. This kind of rush to judgment is premature, rests on faulty assumptions about what kinds of measures matter, and reinforces the outdated and non-evidence-based belief that police involvement and coercion are what will make M110 successful. In focusing on law enforcement involvement, ignoring the context, and failing to take into account the much broader array of forces at work, these early reports evoke many of the failures of drug policy research in general.

Too often, drug policy research focuses on narrow measures [3], such as cost, criminal legal involvement, or abstinence; fails to take into account the complexities of policy implementation and potential confounders [4]; and neglects to involve those directly impacted by the policies in determining the measures of success in the first place [5]. These methodological problems are not unique to drug policy research. Public health researchers often fail to consult with people who are directly impacted by policy changes when they craft their evaluations, and this failure can lead to evaluations that are too narrowly focused, do not answer questions of interest to the community, or fail to take into account significant confounding variables and complexities that can affect outcomes.

To avoid these problems, we interviewed PWUD in Oregon to ascertain important principles and metrics that researchers should use when evaluating M110. Interview candidates were identified by reviewing a list of people who had submitted testimony about M110’s implementation, talking to leaders of organizations that work with PWUD, and following up on leads from initial interviewees. We selected individuals from across Oregon to ensure that both urban and rural perspectives were represented. Participants were paid US $50 each for their time and expertise, and the qualitative interviews, which took place via video conferencing and phone (due to COVID-19), lasted between 45 and 60 minutes. A total of eleven interviews were conducted, after which we reached data saturation. Forty-five percent of the sample was ciswomen; 55% were cismen. Interview recordings and notes were reviewed and analyzed by two members of the working group from which a draft set of principles and metrics were derived. The draft was edited by the working group and then sent back to all the original participants for their review to make sure that the document accurately reflected what they had told us. All participants communicated that the document reflected their input. A full copy of the principles and metrics is available online [6].

The value of engaging with people most likely to be directly impacted by M110, albeit in a limited way given our lack of funding, time constraints, and COVID-19-limited interactions, was readily apparent. Overall, participants wanted PWUD to be more deeply involved throughout the research process and called for comprehensive, rigorous, and nuanced evaluations that include an array of methods and outcome measures. Interviewees had strong opinions about principles that should guide evaluations of M110 based on their experience of and knowledge about prior research projects studying drug use in Oregon. First and foremost, they noted that people who use drugs, their families, and communities that are directly impacted by M110 are the most important resource for evaluations and need to be consulted about their experiences throughout every stage of the process — from research design conceptualization through data collection and analysis and ultimately dissemination of findings. Second, they called for researchers who were experienced, understood their issues, used updated nonjudgmental terminology, and were knowledgeable about the issues surrounding local and national drug policy. For example, one participant noted: “The racial justice component is huge: researchers should know and understand the history of racism in drug policing and the past and presence stance of racism in policing.”

Interviewees also had important insights about evaluation methods, including the desire to see both qualitative and quantitative strategies to ensure appropriate context and optimize explanatory power. They called for evaluations that include primary data collection as well secondary analyses of extant data. This was premised in part on their knowledge of the deficiencies in the available data sources, particularly criminal legal and SUD treatment data. Citing the limitations of relying solely on quantitative data, one participant commented:

Quant data can be used in a vacuum, which can be misleading to the public if context isn’t provided or something is omitted. Like the percentage of people who went to treatment: although important, this doesn’t give us the full picture, because we don’t get the context of the treatment, its quality, what the other options were, etc.

Participants also urged researchers to account for confounding variables, particularly calling out the rise of fentanyl in the drug supply and its impact on overdose rates; potential net widening by law enforcement (e.g., increases in arrests for non-drug related “vagrancy laws”); the impact of COVID-19 on rates of service usage, health and mental outcomes, and drug use; and other concomitant changes in the policy environment.

In a sharp repudiation of narrowly focused research studies, such as those looking exclusively on cost, abstinence from drug use, or crime, interviewees generated more than seventy-five metrics spanning seven domains (criminal legal, law enforcement interactions and culture, social service environment and collateral consequences, healthcare, stigma, and cost and cost savings) by which they thought the success and failure of M110 should be evaluated [6]. The sheer number and array of metrics speaks to the complexity involved in evaluating a policy shift as far-reaching as M110 and the importance of interpreting and contextualizing findings within that complexity.

It is worth noting a few key recurring concerns. First, interviewees were skeptical that Black, Indigenous, and people of color would experience reductions in law enforcement interactions at the same rate as whites and noted that pregnant and parenting individuals who use drugs might continue to be among those most stigmatized. They urged that evaluations of M110 incorporate analyses that assess disparities across these important demographic groups for all key outcomes. Second, they stressed the importance of assessing the outcomes in the context of how policies are actually implemented, noting that, too often, policy evaluations focus on policies as written, ignoring how they actually unfold on the ground, which can profoundly impact their effectiveness. For instance, one participant commented that evaluations should “be transparent about dollars spent versus what the community wanted it spent on” referring to the gap between policymakers’ commitments and “real life” implementation.

Our hope in creating this document [6] was to center the voices of people directly impacted by M110 in evaluations of the policy in order to avoid some of the pitfalls of prior drug policy evaluations. We believe this approach of quickly and efficiently involving people directly impacted by policy change can be more broadly applied to public health research in general. While there has been some attention to community-based and community-driven participatory research [7, 8], these efforts can be challenging and are seldom sufficiently funded to meet the goals. We were able to quickly gather valuable perspectives from people who use drugs in Oregon in ways that will enhance the ability of researchers to conduct thoughtful, comprehensive evaluations of this historic measure, taking into account metrics that matter to impacted communities, while interpreting their findings within the context of conditions on the ground. Consistent with findings of a review of Patient-Centered Outcomes Research Institute studies [9], this kind of approach will improve the rigor of evaluations and their relevance and usefulness as jurisdictions across the country seek to replicate drug decriminalization policies. Importantly, two key takeaways from this work were that people directly impacted (1) need to be meaningfully involved throughout the research process and (2) have invaluable insight and ideas about research questions, research methodologies, and context. This means that researchers who plan to evaluate M110 should engage with people who use drugs throughout their projects while using the developed principles and metrics.

Conducting responsible and ethically grounded research from afar may require coordination to balance benefits and burdens. Out-of-state researchers often have access to large national datasets that could be repurposed to evaluate M110. While we welcome bringing additional data sources to bear on the evaluation, we also caution that analyses conducted in a vacuum may miss key metrics that matter on the ground. In tandem, we also have concerns about the burden to be placed on PWUD and advocates in Oregon who may be tapped to assist remote researchers in evaluating M110 from out-of-state. We hope that researchers will be respectful of these concerns when proposing evaluations.

In 2021, there were over 100,000 overdose deaths in the USA — a record number and 28.5% increase over 2020 [10]. In her recent Health Affairs blog, Dr. Nora Volkow, director of the National Institute on Drug Abuse, argued that SUD treatment should be more realistic and pragmatic [11]. She states, “The magnitude of this [overdose] crisis demands out-of-the-box thinking and willingness to jettison old, unhelpful, and unsupported assumptions about what treatment and recovery need to look like.” M110 and the metrics and methods outlined by the community and this working group are the drug policy and evaluation research answers to this entreaty. We hope this document will be a guide for those seeking to evaluate M110 [6], agencies funding evaluations of the measure, and journal editors and peer reviewers evaluating manuscripts about M110. In addition, we hope that this work serves as one model for how to center the voices of those directly impacted in research efforts more broadly prior to evaluating a new policy.