Commentary on Harper S, Strumpf EC, Kaufman JS. Do Medical Marijuana Laws Increase Marijuana Use? Replication Study and Extension

Abstract

Replicability is a crucial element of good science, particularly so when the subject matter is sensitive and political. Therefore, we welcome close scrutiny of our brief report in the Annals of Epidemiology in 2011, “Adolescent Marijuana Use from 2002 to 2008: Higher in States with Medical Marijuana Laws, Cause Still Unclear.” We were glad to see that Harper et al. (1) were able to replicate our results showing that states with medical marijuana laws (MML) showed greater rates of marijuana use among their residents during 2002 to 2008 and that states that passed MML after 2002 already had greater use among their residents before they passed the law. However, we have several concerns with the additional analyses run by Harper et al. and further concerns with the way their results were presented. We summarize our concerns in this commentary.

CLARITY ABOUT THE EFFECT ACTUALLY ESTIMATED

The most basic problem with the analysis conducted by Harper et al. is that their estimate of the “causal effect” of MML on marijuana use was entirely determined by changes observed in just 5 (Montana, Vermont, Rhode Island, New Mexico, Michigan) of the 16 states in which MML was passed. Although the other states’ data are used to control for secular trends, only those five states that passed laws during the time frame available in the data 2002 to 2008 contribute directly to the difference-in-differences change estimate. To call the difference-in-differences estimate a “causal effect of passing MML” (introduction) makes the extrapolating assumption that the pre–post changes seen in these five states (during the period 2002–2008) are the same as the changes in the eight states that passed MML before 2002 and implies that the same changes would occur in the three additional states that passed MML after 2008.

In our article, we noted that “[A] longer time window of pre/post data would be needed to provide enough information both before and after passage of MML for each state” to make any conclusion about a direct causal effect of MML. Harper et al. disagree, suggesting that by using a difference-in-difference estimator “even with the existing data it is possible to estimate the causal effect of MMLs under some additional assumptions.” The trouble is that the primary assumption underlying the presentation of the results, although never stated, is that the five states that passed MMLs between 2002 and 2008 and the changes found for them are representative of all 16 states that passed MMLs between 1996 and 2012. Note that we do not assert that there is a fundamental flaw with the difference-in-difference estimator for assessing causal effects from quasi-experiments such as those sometimes created by states passing laws. Rather, we want to emphasize Meyer’s cautionary warnings (2) concerning difference-in-difference estimators, that is, the crucial need to be clear about the specific causal effect estimated, and to avoid overgeneralizing. In the analysis of Harper et al., the causal effect actually estimated is the average effect of MML in Montana, Vermont, Rhode Island, New Mexico, and Michigan on change in marijuana use in those states during the years 2002 to 2008. It is not obvious that generalizations can be or should be made from the results of these five states to the rest, which is why our group chose to conclude that more data were needed before anything should be said about causal effects.

We further note that Harper et al. argue that the difference-in-difference estimators used are valid here by drawing an analogy to their common use in epidemiology as in the case-crossover design. We agree that difference-in-difference estimators are commonly used in case-crossover designs but argue that unlike most case-crossover designs, in this particular article, only 10% of the experimental units (i.e., 5 of the 50 states), actually “crossed-over” during the window of study, implying the effectiveness of “treatment” is being determined based only on a small nonrandom sample of the population of interest. Hence, this analysis of state changes in laws is not analogous to the common implementation of a case-crossover design.

Finally, regarding the actual effect estimated, we note that Harper et al. obtained slightly different estimates in their Table 1 replication of our results for the prevalence in states without MML. A close inspection of their analyses uncovers that they included data for Washington, DC, in their state summaries. Although these data are made available by the National Survey on Drug Use and Health, our group chose to not include this 51st data point to accurately describe our results as pertaining to states.

RESULTS IN THE ABSTRACT RELATIVE TO THE REMAINDER OF THE PAPER

Using the difference-in-difference estimator, Harper et al. find that the effect of MML for adolescents is negative, implying MML cause marijuana use to go down after passage. This result is presented in the abstract in a manner that implies a contrast to our findings showing that marijuana use is greater in states with MML.

There are two problems with this result. First, it is described as a general causal effect of MML but is more accurately described as representing the effect in just five states (as described previously). In our article, we were very careful about statements on causality because we think that a more generalizable design and dataset are needed to address the issue of causality. Furthermore, and more importantly, this particular result is the only one of many presented in the body of Harper et al.’s article that was statistically significant. The rest of the results show no significant “causal effect” of MML. Therefore, including only this one significant effect in the abstract appears selective and not fully representative of the material in the text.

LACK OF ROBUSTNESS OF HARPER ET AL.’S RESULTS

We conducted sensitivity analysis of Harper et al.’s result that are illuminating. Montana and Vermont were two states in which MML was passed in 2004. These states had unusually high marijuana usage in 2002 to 2003, the single year of prepassage data available in the analysis. If either of these states is dropped from the analysis, no significant effect of a decrease for MML remains. We suggest that greater care should be taken when presenting a politically controversial finding. The fact that one state could be driving the results was not addressed by Harper and colleagues.

FAILURE TO FIND A CAUSAL EFFECT DOES NOT INDICATE THAT ONE DOES NOT EXIST

Harper et al. draw a problematic conclusion that not finding a significant “causal effect” constitutes evidence for no causal effect. For example, the abstract concludes: “We find limited evidence of causal effects of medical marijuana laws on measures of reported marijuana use”; in summarizing their many null findings, the discussion concludes: “Once we control for any unmeasured state characteristics that do not change over time, we find very little evidence that passing medical marijuana laws increases reported use, among adolescents or any other age group” and in the final paragraph: “…our analysis suggests this [relationship between MML and marijuana use] is unlikely to be a causal association”. As mentioned previously, the difference-in-difference estimator used by Harper et al. use pre versus post MML data only on five states and these states have an average of 2.8 years of data before passage and 3.2 years after passage. Even if there was no problem with generalizing the interpretation of this estimator (as described previously), it is incorrect, in the face of low statistical power, to conflate not rejecting the null with the null being true.

In summary, we suggest that the article by Harper et al. overreaches in its inference given the methods used to address the research question. Of central concern is that the inferences drawn, especially in the abstract, may mislead the unwary reader on an issue for which there is much opinion and controversy but little empirical data. We welcome further replications of our work, and hope to be able to extend this research using more robust methods in more informative data.