RANKING EVIDENCE IN SUBSTANCE USE AND ADDICTION

Abstract

Evidence-based medicine has consistently prized the epistemological value of randomized-controlled trials (RCTs) owing to their methodological advantages over alternative designs such as observational studies. However, there are limitations to RCTs that hinder their ability to study chronic and dynamic conditions such as substance use and addiction. For these conditions, observational studies may provide superior evidence based on methodological and practical strengths. Assuming epistemic superiority of RCTs has led to an inappropriate devaluation of other study designs and the findings they support, including support for harm reduction services, especially needle exchange programs and supervised injection facilities. The value offered by observational studies should be reflected in evidence-based medicine by allowing more flexibility in evidence hierarchies that presume methodological superiority of RCTs. Despite the popularity of evidence ranking systems and hierarchies, nothing should replace critical appraisal of study methodology and examining the suitability of applying a given study design to a specific research question.

Long-regarded the gold standard of medical evidence, randomized-controlled trials (RCTs) have been given the paramount role in evaluating the safety and efficacy of new interventions to improve human health and wellbeing. Several ranking systems, including evidence hierarchies and the GRADE framework, consistently bestow superiority to the RCT and place limitations on the value that can be assigned to observational studies (Guyatt, Oxman, Vist, et al., 2008; Rawlins, 2008; Schunemann, Fretheim, & Oxman, 2006). However, the limitations of RCTs are seldom acknowledged, nor is the fact that observational designs are often better suited to characterize certain health conditions, in particular, chronic diseases. Assuming epistemic superiority of RCTs has led to an inappropriate devaluation of other study designs and the findings they support. This trend is particularly important for fields that are less ethically or scientifically well-suited to RCTs, such as substance use and addiction.

Critique of current evidence ranking

The methodological strength from which RCTs draw their scientific credibility is the random allocation of the intervention among trial participants. Other primary strengths include, but are not limited to, blinding, the use of a control group and applying an intervention. Randomization is presumed to remove any between-group differences in prognostic factors associated with the development of the study outcome (Sackett, 1996; Sackett, Strauss, Richardson, Rosenberg, & Haynes, 2000). Observational studies are able to control for known confounders, yet RCTs are the only design to address the distribution of unknown confounders through randomization. However, randomization is not a guarantee that known or unknown confounders will be balanced between groups at the outset of a given study. The distribution of covariates between the intervention and control group after randomization is typically assessed by the trialists, and if uneven distributions are observed the randomization is deemed to have failed and is repeated until balanced groups are produced. Most importantly, it is also possible that imbalances in unknown confounders exist post-randomization, which cannot be detected or corrected by study investigators. Therefore, the extent to which RCTs can remove baseline imbalances between study arms is limited, to some degree, by the existing knowledge of the disease under study (i.e., the number of known confounders that can be assessed by the study investigators post-randomization). Imbalances in confounders are also an issue for observational studies. While randomization is a more effective strategy to balance these factors than approaches used in observational studies (e.g., propensity scores), randomization does not guarantee that prognostic balance is achieved (Austin, 2011; Han, Enas, & McEntegart, 2009; Worrall, 2010).

The ability to establish external validity and extrapolate a trial’s findings beyond the study protocol is a second issue—one particularly germane to RCTs in substance use and addiction. With the objective of restricting any change in study outcome to the treatment intervention, RCTs typically seek to recruit a select and specific group of participants undergoing a highly structured intervention for a brief period with limited follow-up (Rawlins, 2008; Worrall, 2010). Patients living with comorbidities or receiving multiple medications tend to be excluded in order to minimize the heterogeneity among study participants and minimize the risk of randomizing an intervention to vulnerable patients. Among people who use illicit drugs and are living with substance use disorders, minimizing heterogeneity in the study sample is challenging due to the high prevalence of comorbid health conditions and engagement with diverse health and social services. For example, current estimates among people who inject drugs report that upwards of 50% have a lifetime diagnosis of depression, 50% are living with hepatitis C virus infection, 13% are living with HIV and the medications/treatments prescribed to these patients are highly personalized (Conner, Pinquart, & Duberstein, 2008; Kessler et al., 1994; Lengauer, Pfeifer, & Kaiser, 2014; United Nations Office on Drugs and Crime, 2017). Notwithstanding the ethical challenges of placing restrictions on inclusion and exclusion criteria, identifying a homogeneous group of participants based on these complex comorbidities and co-interventions is often practically untenable. Administering a structured intervention to this population is further complicated by high rates of marginalization (e.g., homelessness, social instability, stigmatization) and criminalization (Marshall et al., 2016; Stone et al., 2018). In consequence, the average treatment effect calculated from an RCT is only as valuable as the sample from which it was estimated (Dahabreh, 2018; Deaton & Cartwright, 2018b; Ioannidis, 2018). The superior internal validity of an RCT does not translate to invariance in the treatment effect across contexts if the sample is not representative of all the patients to which the study could be applied (Deaton & Cartwright, 2018a, 2018b). Nevertheless, the results of RCTs are often extrapolated generously despite the limitations imposed by selection criteria and artificial environments that do not reflect real-world applications of the target intervention (Deaton & Cartwright, 2018a, 2018b). As an alternative, some authors advocate for strategies such as propensity scores, instrumental variables and matching to strengthen the methodology of observational studies (Concato & Horwitz, 2018; Deaton & Cartwright, 2018b). Empirical comparisons between RCTs and well-designed observational studies have found similar summary measures of effect size with neither design producing a consistently greater effect (Anglemyer, Horvath, & Bero, 2014; Benson & Hartz, 2000; Concato, Shah, & Horwitz, 2000; Sterne et al., 2002). Moreover, many treatments that continue to be used in clinical and non-clinical settings have been evaluated through observational methods have been found to be both safe and effective (Concato & Horwitz, 2018; Deaton & Cartwright, 2018b; Tsimberidou, Braiteh, Stewart, & Kurzrock, 2009). In these cases, conducting an RCT on the basis that the existing evidence is observational is unnecessary based on the long-term record of safety and effectiveness and RCTs may expose patients to unnecessary risk if patients are denied treatment shown to be beneficial (Deaton & Cartwright, 2018b; Vandenbroucke, 2008).

Some authors have also questioned the value of RCTs and evidence hierarchies for identifying unintended or adverse effects of new treatment interventions (Osimani, 2013; Vandenbroucke, 2008). As the unintended or possibly adverse effects of novel treatments are typically unknown, the treatment allocation is masked with respect to the unintended effects even if the investigator is aware of who is receiving the treatment (Vandenbroucke, 2004). These authors argue that this achieves the same separation between intervention allocation and prognosis that is accomplished through blinding (Osimani, 2013). As a result, observational studies examining adverse events will not be as vulnerable to confounding and selection bias as observational studies evaluating intervention efficacy (Vandenbroucke, 2004, 2006, 2008; Vandenbroucke & Psaty, 2008). This view is supported by empirical evidence that shows no systematic difference in risk estimation for adverse events between RCTs and observational studies (Benson & Hartz, 2000; Concato et al., 2000; Papanikolaou, Christidi, & Ioannidis, 2006). A second critique is that the strict inclusion criteria of RCTs may exclude vulnerable participants with complex comorbidities at an increased risk of experiencing adverse events, which applies to many people who use drugs (Conner et al., 2008; Rawlins, 2008). In these circumstances case reports or case series may be the most sensitive or only tool available to identify side effects (Glasziou, Chalmers, Rawlins, & McCulloch, 2007; Stricker & Psaty, 2004). Based on these arguments, it has been suggested that the evidence hierarchy should be inverted when the objective is to identify unknown adverse events (Vandenbroucke, 2008).

Although RCTs excel at investigating the safety and efficacy of novel pharmacological formulations, issues arise in testing harm reduction-based interventions to benefit people who inject drugs. The risk environments in which needle exchange programs (NEP) and supervised injection facilities (SIF) are implemented are highly variable between settings. For example, the criminalization of illicit drug use in North America has adjusted as policies legalizing and regulating non-medical cannabis continue to expand in the context of an opioid overdose crisis sparked by the contamination of the illicit drug supply. In these circumstances, long-term observational studies in real world or natural settings may be better suited to evaluate the evolving impacts of changes in substance use policy and criminalization than RCTs. While RCTs are not feasible or ethical in this situation, observational studies are advantageous in that they can often include more follow-up time than RCTs, which is needed to better evaluate the long-term effects of structural factors such as policy change and criminalization on the chronic and dynamic nature of substance use and addiction (Kelly, Greene, & Bergman, 2018; Worrall, 2010). The utility of real world data has sparked calls for the integration of alternative data sources such as electronic health records and patient registries to study patient groups who may have been excluded from RCTs (e.g., elderly people living with comorbidities) (Nabhan, Klink, & Prasad, 2019). Other alternatives that have been applied to evaluating substance use include pragmatic trials that blend the traits of observational studies with traditional RCTs (Coulton et al., 2017; Henderson et al., 2017). Alternative trial designs including stepped wedge and crossover trials have been effectively applied to study diseases requiring extended follow-up evaluation (Chotard et al., 1992; Hemming, Haines, Chilton, Girling, & Lilford, 2015). These designs have been used in diverse areas, including HIV and social policy, although they are challenging when the intervention is believed to be effective and receipt should not be denied or delayed, or there are complex contextual and patient factors that moderate the impact of the intervention on the study outcome (Bonell et al., 2011; Brown & Lilford, 2006). Unfortunately, these challenges are common among interventions for people who use drugs. Blinding and providing a placebo are often not possible for these interventions and extrapolating the effects of the intervention beyond the study protocol must be done with caution given that substance use and addiction are chronic and relapsing conditions (Kelly et al., 2018; Kelly, Greene, Bergman, White, & Hoeppner, 2019).

Perhaps the most significant unacknowledged weakness of RCTs is their typically limited periods for the intervention and follow-up. Due to their substantial expense, funding for post-study follow-up is typically limited to six to 24 months, which is often insufficient for chronic diseases (Bluhm, 2010; Vlieland, 2002) such as substance use disorders, characterized as chronically relapsing conditions with recurring transitions between substance use, abstinence, treatment engagement and incarceration (Dennis & Scott, 2007; Kelly et al., 2019). For example, the median time from first to final instance of substance use was estimated at 27 years among people admitted to publicly-funded treatment who were able to abstain for at least one year, and the average time between the first episode of substance use treatment and last instance of substance use was nine years (Dennis, Scott, Funk, & Foss, 2005). A recent analyses from a national study of adults in recovery in the United States (N=39,809) found that an average of two to five attempts were required to resolve an alcohol or drug problem and the number of attempts varied by demographic characteristics, type of treatment and clinical history (Kelly et al., 2019). African American ethnicity, previous use of treatment and mutual-help groups and psychiatric comorbidity were associated with a greater number of recovery attempts (Kelly et al., 2019). Moreover, many patient-important outcomes including quality of life, happiness and self-esteem do not increase monotonically during recovery. These measures were found to decrease significantly in the first year following resolution of an alcohol or drug problem and then increase over subsequent years (Kelly et al., 2018). As a result, the methodological strengths of RCTs are, to some extent, mitigated by the limitations of administering brief structured interventions to a heterogeneous population living with chronic and dynamic disorders. For conditions such as substance use and addiction, RCTs are often limited to providing a snapshot of the effectiveness of interventions in the short term and may fail to identify long-term effects among heterogeneous groups of participants living with chronic and dynamic disorders. In these circumstances, observational studies are often better suited to study interventions designed for a more representative sample of individuals who are often excluded from studies requiring randomization, and who require long-term follow to evaluate enduring changes in study outcomes (Bluhm, 2010; Worrall, 2010). Rather than viewing observational designs as a next best option in situations where RCTs are perceived as practically or ethically unfeasible, it should be recognized that the methodological strengths of observational studies may provide the best available evidence to evaluate the course of chronic conditions and the effect of interventions to address them (Bluhm, 2010; Worrall, 2010).

In contrast to the approach of evidence-based medicine that strives to identify the most accurate estimate of an average treatment effect, other researchers have proposed a framework that conceptualizes health interventions as “evidence-making” (Rhodes & Lancaster, 2019). Rather than assuming a singular and universal effect of an intervention, the evidence-making framework posits that the variability of practice and patients produces multiple realities of the effects of an intervention and treats evidence, interventions and their effects as emergent, contingent and multiple (Rhodes & Lancaster, 2019). By recognizing that the diversity of practices creates multiple realities of an intervention, evidence-making frameworks foster dialogue across diverse forms of knowledge and knowledge actors to recognize how the politics of intervention knowledge and the realities they create influence their effects across settings (Rhodes & Lancaster, 2019). While it is beyond the scope of this paper to provide a detailed analysis of this framework, similar approaches have been raised by authors who advocate for a process of cumulative scientific understanding that is often challenged by the assumptions of evidence-based medicine and evidence hierarchies (Deaton & Cartwright, 2018b). The production of new scientific evidence should build upon and be integrated with existing knowledge to enhance collective scientific knowledge. Unfortunately, it is not uncommon for the findings from observational studies to be dismissed if more recent evidence from a randomized study contradicts the original results (Deaton & Cartwright, 2018a, 2018b). Some experts argue that new findings must be able to explain or be integrated with previous results as knowledge advances, even if previous results are believed to be invalid (Deaton & Cartwright, 2018b). Failure to integrate results from randomized and non-randomized studies undermines the responsibility of science to advance the cumulative understanding of health interventions and acknowledge the legitimate variability of intervention effects created by the diversity of practice and patients (Concato et al., 2000; Rhodes & Lancaster, 2019).

The inappropriate devaluation of observational studies has slowed the implementation and scale-up of several harm reduction interventions for PWUD. The observational evidence supporting SIF and NEP includes many large-scale prospective cohort studies with several years of follow-up from multiple countries (Aspinall et al., 2014; Potier, Laprevote, Dubois-Arber, Cottencin, & Rolland, 2014; Wood, Tyndall, Montaner, & Kerr, 2006). The advantages of being able to prospectively evaluate the health impacts of these interventions among heterogeneous samples of people who use drugs in real-world settings are unique methodological strengths that provide evidence into the long-term trajectories of this population. Existing evaluations of NEP and SIF have provided strong evidence for reducing HIV risk behaviours, overdose mortality and increasing engagement with addiction treatment services without adverse effects on broader public health and safety (Aspinall et al., 2014; Marshall, Milloy, Wood, Montaner, & Kerr, 2011; Potier et al., 2014). In spite of this evidence, these services remain limited in many countries contending with the challenges of substance use harms (Degenhardt et al., 2014).

Although ideological opposition to harm reduction-based interventions like NEP and SIF remains the primary barrier limiting their availability in many settings worldwide (Nadelmann & LaSalle, 2017), their establishment and expansion has also been hindered by an ironic epistemic predicament regarding their scientific evaluation. As these services provide self-evident health and safety benefits to profoundly marginalized and vulnerable individuals, regulators and trialists have determined it would be unethical to evaluate them through RCTs as randomization would restrict the control group from accessing potentially life-saving interventions (Bastos & Strathdee, 2000; Bluthenthal & Kral, 2010; Lurie, 1998). Yet, when observational evidence accumulates demonstrating the benefit of these interventions, the lack of RCTs is critiqued as a methodological weakness (Bluthenthal & Kral, 2010). In effect, the anticipated benefit of these services limits the credibility of subsequent empirical evidence supporting these services (Bluthenthal & Kral, 2010). Throughout the implementation of NEP and SIF, the use of observational methodologies to evaluate these services has often been referenced as a precaution against their expansion or a methodological limitation (Palmateer et al., 2010; Wood et al., 2006).

In addition, the effectiveness of SIFs has been challenged based on uncertainty regarding the effect size of these services, despite the large evidence base that is nearly unanimous in support of SIF, with no SIF overdose deaths reported to date and no reported adverse effects (Caulkins, Pardo, & Kilmer, 2019; May, Holloway, & Bennett, 2019). Criticism about effect heterogeneity relates to the EBM objective of estimating a singular treatment effect, whereas it is entirely possible, and we would argue likely, that the effect size of these interventions may vary across patient populations and contexts (Deaton & Cartwright, 2018b; Rhodes & Lancaster, 2019). Rather than this heterogeneity being perceived as a weakness of study methodology, we would argue that this variability is consistent with the evidence-making intervention framework whereby the diversity of practice and patients creates multiple realities of an intervention’s effects (Rhodes & Lancaster, 2019). In fact, this variability should be expected based on the specific crisis situations in which these services are often implemented (Caulkins et al., 2019). SIFs have been established in diverse countries experiencing different types of illicit substance use, patient populations, service models, local contexts, adjunct services and drug policy (Caulkins et al., 2019). These situations are not well-suited to the evidence-based medicine perspective that attempts to minimize patient and intervention heterogeneity to estimate a precise and singular treatment effect (Deaton & Cartwright, 2018b; Rhodes & Lancaster, 2019). Instead, the existence of multiple contingent effects of an intervention that are adaptive is in keeping with the ontology of the evidence-making intervention framework (Rhodes & Lancaster, 2019). While we acknowledge that randomizing participants to SIF participation (e.g., through community-randomized trials or wait-list studies) would be theoretically valuable, we contend that this is has become unethical and unnecessary given the evidence of benefit and absence of harm associated with these services (Caulkins et al., 2019; Kennedy, Hayashi, Milloy, Wood, & Kerr, 2019). As previously mentioned, RCTs do not have the same benefits for assessing adverse outcomes and the risks associated with denying or delaying SIF use among vulnerable populations of PWUD through RCTs is difficult to justify (May et al., 2019; Osimani, 2013; Vandenbroucke, 2006). Although RCTs are held as the gold-standard for identifying causal associations, observational and qualitative studies are able to provide valuable evidence about the causal mechanisms underlying these associations and contribute to advancing the cumulative understanding of an intervention without the need for experimental closure (Deaton & Cartwright, 2018b; Rhodes & Lancaster, 2019). The study design that provides the best evidence may therefore vary depending on the theory or research question being evaluated; this should be reflected in evidence ranking systems such as the GRADE framework and the Maryland Scientific Methods Scale that place limits on the value that can be assigned to non-randomized studies (Alonso-Coello et al., 2016; Worrall, 2010).

There are examples where observational studies have been sufficient to demonstrate effectiveness and continue to be used clinically without the need for verification through RCTs. An analysis of oncology drugs approved by the United States Food and Drug Administration found that 31 of 68 drugs were approved without an RCT and 30 of these drugs remain fully approved (Concato & Horwitz, 2018; Tsimberidou et al., 2009). Studies using objective endpoints were the most common among those approved and these drugs demonstrated a long-term record of efficacy and safety based on observational evidence (Tsimberidou et al., 2009). There are also cases where the belief that RCTs are needed to confirm observational evidence has caused harm. The controversy surrounding the RCTs of extracorporeal membrane oxygenation (ECMO) therapy in newborns has been well described (Bluhm, 2010; Truog, 1992). Both ECMO and the standard treatment were evolving rapidly and observational studies had already supported the benefit of ECMO (Bartlett, 1984; Wung, James, Kilchevsky, & James, 1985). Yet, variability in the success rate of ECMO across centres prompted many researchers to call for RCTs before ECMO became an accepted therapy (Bluhm, 2010). In the first trial, the one infant who received conventional therapy did not survive while all 11 who received ECMO survived (Bartlett et al., 1985). In phase one of the second trial, four of the ten newborns receiving conventional therapy did not survive while all nine who received ECMO survived (O’Rourke et al., 1989). In phase two, 20 patients received ECMO and 19 survived (O’Rourke et al., 1989). These examples have led some experts to suggest that RCTs should not be used in cases where interventions are rapidly evolving and are potentially life saving (Truog, 1992). In these cases, the interventions or their implementation may be out-dated by the time the RCT has concluded and observational studies of clinical practice (e.g., outcomes research) may provide better evidence of intervention effectiveness while avoiding some of the ethical challenges of withholding or delaying novel treatments in RCTs (Truog, 1992; Worrall, 2010). This is particularly relevant to the situation of SIFs, which are often implemented in situations of crisis that are rapidly evolving as new challenges arise, such as the multiple waves of the opioid overdose crisis. Under these circumstances, variability in the effect should not detract from the fact the evidence is, even in the opinion of critics, “almost unanimous in its support” and is potentially life saving (Caulkins et al., 2019; Kennedy et al., 2019; May et al., 2019). Performing RCTs of SIFs to ascertain a singular measure of effect should be viewed as unnecessary and unethical given the existing evidence and that observational studies may provide better information to evaluate these interventions based on the evolving context of their implementation and the potential for them to be life saving.

Future directions for addiction science and evidence ranking

An alternative observational design that has gained considerable attention, particularly for studying chronic diseases, is Mendelian randomization (Lawlor, Harbord, Sterne, Timpson, & Davey Smith, 2008; Smith, 2006). These studies integrate genetic variants into observational epidemiology to enhance the causal inferences that can be drawn about modifiable risk factors and health outcomes (Smith & Ebrahim, 2003; Youngman, Keavney, & Palmer, 2000). Rather than randomize participants to receive the exposure being studied, Mendelian randomization studies take advantage of the random assignment of an individual’s genotype from their parents to conduct a ‘natural’ RCT (Davey Smith & Ebrahim, 2005; Hingorani & Humphries, 2005). If germline genetic variants for an environmental exposure (e.g., substance use) have been identified, these variants can be used as a proxy measure of the exposure in observational studies and be treated as randomly distributed (Lawlor et al., 2008; Pasman et al., 2018). Mendelian randomization studies can also be performed in representative population samples without the need for exclusion criteria or randomizing participants as is necessary in traditional RCTs (Lawlor et al., 2008; Smith & Ebrahim, 2003). In addition, the associations between these gene variants and health outcomes are not susceptible to reverse causality as germline genotypes are not affected by disease progression, and, if the gene variant is not pleiotropic, the risk of confounding is mitigated (Smith & Ebrahim, 2004). Finally, genetic variants that predict an environmental exposure typically do so throughout the life span, a fact which minimizes regression dilution bias (Smith & Ebrahim, 2004). As the cost of genome sequencing continues to decrease and more large-scale consortia are forming, integrating genetic data into observational designs is becoming increasingly feasible for studies of substance use and addiction (Pasman et al., 2018; Phillips, Deverka, Hooker, & Douglas, 2018; Vaucher et al., 2018). For example, Mendelian randomization studies have been applied to investigate the longstanding debate surrounding the association between schizophrenia and cannabis use (Gage et al., 2017; Pasman et al., 2018; Vaucher et al., 2018). The limitations of Mendelian randomization studies include population stratification, identifying a reliable genetic variant for the exposure of interest that does not have pleiotropic effects on the outcome and is not in linkage disequilibrium with other gene variants associated with the outcome (Cardon & Palmer, 2003; Lawlor et al., 2008; Smith & Ebrahim, 2004). Additional challenges include developmental compensation, contextual influences on the exposures predicted by gene variants and non-linear associations (Gibson & Wagner, 2000). Despite these limitations, Mendelian randomization studies retain the ability of traditional observational studies to study the development of chronic conditions over time with the advantage of studying gene variants for exposures that are randomly distributed (Lawlor et al., 2008; Smith & Ebrahim, 2003). The ability to combine the strengths of observational studies while adding a component of randomization should challenge the assumed superiority of RCTs among policymakers and healthcare professionals. The evidence from these studies may also prevent the need to conduct RCTs to address certain research questions. For example, a Mendelian randomization study evaluating the effects of selenium supplementation on prostate cancer risk provided evidence of no effect, which was consistent with the largest ever prostate cancer prevention trial (Yarmolinsky et al., 2018). This trial was stopped before completion based on a lack of efficacy and adverse events, at a cost of $114 million (Yarmolinsky et al., 2018). As the efficacy and adverse events were predicted by this Mendelian randomization study, these designs may be an efficient and affordable means to study interventions traditionally assessed RCTs as big data resources for genetic information continue to expand (Yarmolinsky et al., 2018).

A second area for improvement in substance use and addiction research is the further integration of patient-important outcomes. The underrepresentation of patient-reported outcomes has been highlighted in several study designs yet is more pronounced in RCTs (Pardo-Hernandez & Alonso-Coello, 2017; Saldanha et al., 2017). Previous field-specific systematic reviews of RCTs have found that less than 25% include a patient-important outcome as a primary outcome (Gaudry et al., 2017; Rahimi, Malhotra, Banning, & Jenkinson, 2010). The lack of patient-important outcomes is highly germane to substance use research and a systematic review is currently ongoing to identify patient-important outcomes to assess the effectiveness of opioid use disorder in the context of the overdose epidemic (Dennis et al., 2015; Sanger et al., 2018). Increased recognition of the value of patient-important data is reflected in the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system, which recommends their inclusion at two different stages (the outset of study design and when ranking the importance of study outcomes) (Guyatt, Oxman, Kunz, et al., 2008). Other frameworks designed to help stakeholders translate scientific evidence to healthcare decisions prioritize the inclusion of patient-important outcomes and considers the degree to which patients value each outcome (Alonso-Coello et al., 2016). The Core Outcome Measures in Effectiveness Trials (COMET) initiative extends this view by promoting the selection of a standardized set of outcomes for each health condition to compare effectiveness across studies (“The COMET Initiative,” 2010; Gorst et al., 2016; Williamson et al., 2012). The lack of patient-important outcomes in addiction literature, and the fact that these measures vary significantly in different stages of addiction, calls for further integration of these outcomes in future research (Kelly et al., 2018; Sanger et al., 2018). Given the value of patient-important outcomes—and that they are prioritized in stakeholder frameworks including GRADE and COMET—further evaluation of these outcomes will be important to promote the uptake of observational evidence on substance use and addiction among policymakers and healthcare officials (Gorst et al., 2016; Guyatt, Oxman, Kunz, et al., 2008; Pardo-Hernandez & Alonso-Coello, 2017; Williamson et al., 2012).

There are, without question, important limitations to studies with non-randomized designs. However, the value offered by observational studies should be reflected in evidence-based medicine by allowing more flexibility in evidence hierarchies that presume methodological superiority of RCTs. Observational designs may provide the best evidence to evaluate interventions to address chronic conditions such as substance use and addiction. Unfortunately, assuming epistemic superiority of RCTs has unnecessarily slowed the uptake of substance use research and harm reduction services for people who use drugs. Despite the popularity of evidence ranking systems and hierarchies, nothing should replace critical appraisal of study methodology and examining the suitability of applying a given study design to a specific clinical question.