Natural Experiments in a Complex and Dynamic Environment: The Need for Measured Assessment of the Evidence

While there are several areas of agreement with the points made by Anderson and Rees, we do have a few key points of departure. First, we disagree with their conclusion that much has been learned from “clearly defined natural experiments” of medical marijuana liberalization policies, not because this literature lacks methodological rigor but because it has not adequately accounted for state-level policy heterogeneity and implementation uncertainty. We addressed the first point above, and take up the second point here.

While the passage of state medical marijuana laws may have indicated a clear shift in the norms or perceived harmfulness of marijuana among voters, the initial laws that passed in most states did not contain clearly specified legal sources of supply. Without an authorized distribution system and clarity on where potential home cultivators could obtain starter plants or seeds, patients had no obvious way outside the black market to access the medicine the law said they could use. Consequently, quasi-legal medical marijuana collectives and retail outlets sprouted up in many early adopting states. Importantly, as these developments were incremental and often subject to variable enforcement activity, legislative amendments, and court rulings, the static binary policy indicators used in most studies on the effects of medical marijuana laws likely obscure as much as they reveal. Even in Colorado, where supply channels were fairly explicit in the initial law and cooperatives emerged quickly, it took several years and numerous court cases before the legal legitimacy of dispensaries was resolved in terms of legal protection under state law (Office of the State Auditor, 2013).

In addition to the uncertainty within states, there also remained tremendous uncertainty regarding how the federal government would respond. The impact of the uncertainty on individual behavior is evident in Colorado, where the state witnessed an enormous rise in the number of registered patients after a 2009 memo written by Deputy Attorney General David Ogden that provided formal guidelines to federal prosecutors not to pursue legitimate medical patients who were in compliance with state laws (Office of the State Auditor, 2013). On January 31, 2009, prior to the memo, the Colorado Department of Public Health and Environment reported that 5,051 patients had active registration cards. By January 31, 2010, just one year later and a few months after the memo, the state reported 55,469 active registered patients, a tenfold increase.

The fact that it took time for these state and federal legal issues to get resolved, along with the wait-and-see approach by patients for the policy environment to become more settled, suggests that one should be cautious inferring policy implications from the initial effective policy dates. While the effective dates may in fact coincide with a shift in norms toward marijuana, marijuana use and other public health outcomes could continue to change in response to a number of different factors, including end-user monetary costs (price), product availability, legal risks, and pharmacological substitution or replacement effects (Lucas et al., 2013).

Second, we place far less confidence in the conclusiveness of findings regarding substitutability between marijuana and alcohol. Contrary to the interpretation presented by Anderson and Rees, we feel the empirical evidence regarding this question remains inconclusive and cannot yet support definitive policy statements. Even with the clearly defined natural experiment of turning 21, rigorous quasi-experiments examining the effects of the minimum legal drinking age (MLDA) on alcohol and marijuana use reveal substantively different treatment effects. The results of a series of regression discontinuity (RD) studies analyzing the effect of the MLDA on marijuana use vary considerably depending upon the data source used, the outcome analyzed (past 30-day prevalence vs. days used in past 30), whether the sample is conditioned by the recency of marijuana use, and the estimator used and related technical specifications (e.g., parametric vs. nonparametric estimators, functional form) (Crost & Guerrero, 2012; Crost & Rees, 2013; Yörük & Yörük, 2011, 2013). For example, the base models reported in these studies suggest that the experience of turning 21 results in either a significant two percentage point decrease in the prevalence of marijuana use (Crost & Guerrero, 2012), no significant association (Crost & Rees, 2013; Yörük & Yörük, 2013), or a significant seven percentage point increase in marijuana use (Yörük & Yörük, 2011). Although the latter study has been criticized for restricting the estimation sample to past-year marijuana users (Crost & Rees, 2013), we remain agnostic that excluding “occasional” marijuana users is the fatal design flaw Crost and Rees make it out to be.

In short, there are legitimate differences in the available research identifying the sign of the association between MLDAs and marijuana use. Moreover, the differences exist not just in the conduct of natural experiments but also in well-constructed and highly controlled lab experiments, as we mentioned previously. Heterogeneity in the types of users examined in each of these studies seems to be key, and the lack of attention paid to this causes much to be lost in discussions that frame the substitution-complement debate in terms of a simple dichotomy rather than along a continuum reflecting the strength of association for a given user group. The fact that Anderson and Rees in their Point argument find no significant impact of dispensary policies on thirty-day prevalence rates among school age youth in local YRBS samples just reinforces this point regarding heterogeneous responses. Their finding is entirely consistent with our own examination of dispensary effects using state-level YRBS (Pacula et al., 2013). However, it does not negate the legitimacy of our findings of a differential effect among those in need of treatment, as indicated by treatment admissions for alcohol and marijuana in TEDS (Pacula et al., 2013). If anything, it further reinforces the point that policy makers need to consider heterogeneous effects if the goal is to understand social costs.

Indeed, the relevance of heterogeneity can be seen in even simple descriptive data, such as emergency department (ED) episodes among youth less than 21 years of age. Online data from the Drug Abuse Warning Network (DAWN) shows that total ED visits involving alcohol among youth were modestly declining from a rate per 100,000 population of 239 in 2004 to 215 in 2010 (DAWN, 2013). This is consistent with a general decline in youth ED mentions involving only alcohol during the same period, from 176 to 140 per 100,000. However, not only did the rate of marijuana-involved ED visits rise among youth, the rate of visits involving alcohol in combination with marijuana nearly doubled from 29.2 in 2004 to 43.4 in 2010 (DAWN, 2013). Thus, speaking in terms of whether individuals “typically substitute marijuana in place of alcohol” may be useful, but perhaps not as important from a social cost perspective as understanding if the proportion of those who use them together is growing or shrinking.

Finally, we believe it is important to reiterate the inherent limitations of trying to draw conclusions from any of the existing literature regarding predictions for a legal marijuana environment. All of the evaluations of state liberalization policies have occurred under concurrent federal prohibition, which imposes stiff criminal sanctions on the cultivation and distribution of marijuana. This likely deters competition in the supply of marijuana in many places. Legalization policies are likely to generate changes in behavior across several margins that cannot be fully reflected by the current research, including declines in price and increases in commercialization and promotion that come with legalization (Kilmer et al., 2010; Caulkins et al., 2012). Predicting the net societal impact of marijuana legalization policies is therefore a difficult exercise. Numerous factors and critical assumptions come into play, including the characteristics of the user population, consumption practices, potency, structure of the legal regulatory system, level of enforcement, and so forth. Even the relationship between marijuana use and other substances, like alcohol, could be dramatically different under a legal regime, as income effects and legal risks could shift market demand in unpredictable ways.

All of this is not to say that the shift from punitive prohibition to legalizing marijuana at the state (or federal) level will not produce a net social benefit. Our read of the available evidence, however, causes us to be more cautious in making definitive claims of either net benefits or net costs given the current state of the science. We need a better understanding of the long term effects of marijuana use on health and mental health, the role of potency and other cannabinoids in influencing health and safety outcomes, the impact of regulated and unregulated markets on prices and youth consumption, and the impact of legalization on the consumption of related goods (alcohol and prescription drugs). Opportunities exist to learn about many of these areas in the coming years, but it appears in this instance the science will lag policy in terms of informing the debate.