Has cannabis use among youth increased after changes in its legal status? A commentary on use of Monitoring the Future for analyses of changes in state cannabis laws

Abstract

As US states move toward various forms of adult access to cannabis, there has been a great interest in measuring the impact of such changes on adolescent cannabis use. Two recent prominent analyses have used Monitoring the Future (MTF), a nationally representative survey of students, to examine the effects. We compared MTF data for California and for Washington State with other survey data on use by adolescents in those states. In both studies, findings based on MTF were different from those using other larger, state-representative surveys. The discrepancy reflects the high within-state variation in prevalence rates and the small number of schools in MTF state samples. Using the Washington Health Youth Survey we estimate that after recreational cannabis legalization past 30-day cannabis use prevalence in grade 8 decreased by 22.0%, in grade 10 prevalence decreased by 12.7%, and no effect in grade 12. These trends are consistent with those in states without recreational cannabis laws, suggesting legalization did not impact adolescent use prevalence. Long-term trends in MTF are consistent with other data, but year-to-year volatility in state-level series undermine the survey’s suitability for evaluation of state cannabis policy changes. Survey-based analyses at the state level need to be cross-validated with findings from other data sources. When findings are disparate and methodological rigor is equivalent, analyses of data sources specifically designed to describe state-level phenomena are more credible.

Cannabis policy is becoming less restrictive worldwide, and researchers struggle to forecast the consequences of loosened legal restrictions. The number of individuals who consumed cannabis in the past month grew 30% to 32 million over 2010–2016, and over $50 billion is spent on the drug annually amid rapid changes to the forms and potency of products available to both adults legally, and adolescents illicitly (Midgette et al. 2019). Our understanding of the epidemiology and etiology of cannabis use on youth is limited, in part by the recency of both policy changes and increased research attention (Johnson and Guttmannova 2019). The challenges are both conceptual and empirical. At the conceptual level, for example, states may provide cannabis for medical purposes under a great variety of rules regulating production, sales, advertising, store hours, and restrictions on access to products. That is also true for legal recreational cannabis states (Dilley et al. 2017; Klieger et al. 2017). Decriminalization also takes a variety of legal forms and implementation can vary in important ways (Hunt et al. 2018), as can rates of use (Hughes et al. 2016). Studies that treat all medical cannabis and decriminalization regimes as the same and uniform within states, often reflecting the realities of limited data, can be misleading (Pacula et al. 2015).

In this study we describe the data-related challenge of evaluating the impact of drug policies on adolescent drug use. We further demonstrate through comparison of Monitoring the Future (MTF)—which is nationally but not state-representative—with state-representative surveys, a fundamental tenet of statistical reliability in survey research and design: the chance that an analysis produces spurious results is directly related to the appropriateness of its underlying data. We use a rich state-representative youth survey to demonstrate that adolescent past 30-day cannabis use prevalence did not appear to change in response to recreational legalization. This reinforces the need for rigorous evaluation of the reliability of state-level subsamples from a nationally representative surveys. Such sub-samples may yield erratic results if the survey is not designed to be state-representative, so alternative data sources should be used for cross-validation.

Data Limitations in the Study of State and Local Drug Policies

Empirically, research on drug use and its consequences must always rely on imperfect data collected less frequently or from fewer respondents than is ideal. Given that fact, our understanding of any drug policy’s impacts should be more akin to a patchwork quilt comparing available, relevant, and appropriate data than a piece of photorealism form a single perspective. Guttmannova et al. (2019) provide a path forward to analyze the national-level trends in cannabis use among adolescent and emerging adult populations utilizing nationally-representative repeated cross-sectional data sets: the Youth Risk Behavior Surveillance System (YRBS) and the National Survey of Drug Use and Health (NSDUH). We explore options for the evaluation of state-level policies.

Few representative samples of state-level cannabis use exist in the US. The biennial Youth Risk Behavior Surveillance System (YRBS) covers 47 states and 19 large school districts for 9^th through 12^th graders; however, it does not provide state-representative samples in all states where it is deployed. Similar population-representative surveys conducted by individual states are typically biennial due to their cost and administrative burden. The annual NSDUH collects use data for the population twelve years of age and older at the state and sub-state level annually across the nation. The survey requires two years of survey data to be state-representative and three to be to be representative of sub-state regions. Moreover, the age range for which it provides a state representative sample of adequate size, 12–17, is too broad to describe the trajectory of drug use during maturation through adolescence. Given the paucity of relevant and reliable data on cannabis use at the state and local levels where most policy changes are taking place, it is difficult to measure those policies’ impact on youth perceptions and behavior.

To answer key questions about the consequences of changing state and local cannabis policies, researchers have to rely on data not meant for policy evaluation. One option is to use population-representative sources that only indirectly capture the phenomenon of interest. For example, systematically collected state and local-level drug use indicators include criminal justice records such as driving under the influence of cannabis, possession arrests and citations, or public health data such as emergency department visits, treatment and hospital admissions, and poisoning events. These sources, however, may only weakly measure use prevalence and frequency; there are many intervening variables, such as enforcement intensity, that often cannot be independently measured. Alternatively, analysts may look to data that are state representative and capture prevalence and frequency of use information, but are collected infrequently or in only a few locations. State-administered in-school surveys (e.g., the Washington State Healthy Youth Survey and California Health Kids Survey we focus one here) offer large samples of responses for state-specific estimates, though typically in two-year intervals due to cost and administrative burden.

Researchers may also apply rich nationally representative data to infer effects of state-level cannabis policies. Monitoring the Future (MTF) is a nationally-representative survey of students in 8^th, 10^th, and 12^th grade students annually. Nationwide, the survey draws from approximately 140 schools per grade per year, and up to 350 students per school. Schools also rotate out after two years of participation. In total, MTF provides information on roughly 15,000 participants per grade per year. It is thus a prime candidate for use in a variety of research contexts. Alongside NSDUH and YRBS, MTF is the most important series to provide nationally representative data on drug use, risk perception, and other adolescent and young adult behaviors. It has been used since its inception in 1976 for alcohol, tobacco, and cannabis policy research. Recent MTF-based research has contributed much to our understanding of drug use and its correlates in the U.S. Between 2015 and 2018, 80 peer-reviewed journal articles using MTF were published or in press as of this writing.

Most peer-reviewed analyses based on MTF focus on national-level insights, and prior research has taken care to note that the survey’s sampling is not designed to yield state-representative data (e.g., O’Malley and Wagenaar 1991; Wagenaar et al. 2001; Lynne-Landsman et al. 2013). Many analyses utilizing MTF measure the effects of policies by pooling states that enacted similar policies, including examination of state-level alcohol (Carpenter et al., 2007) and tobacco policies (Emery et al., 2005; Tauras et al., 2005; Terry-Mcelrath et al., 2007; Tworek et al., 2010). Notably for cannabis policy research, Hasin et al. (2015) found that past-month use prevalence among high school students is higher in states with medical cannabis laws, but implementation of the laws was not associated with increases in adolescent cannabis use. Amid a downward trend in perception of cannabis harm nationally, perceived risk increased and use decreased among 8^th graders after the passage of MCLs; no effect was found for 10^th or 12^th graders (Keyes et al. 2016).

Two recent studies used MTF to evaluate the impact of individual state-level cannabis policies on adolescent cannabis use. Miech et al. (2015) examined the effect of cannabis decriminalization in 2011 on youth in California. Cerdá et al (2017) estimated the effect of recreational cannabis legalization, on youth in Colorado and Washington. Both studies compared the MTF sub-samples from those states to the remainder of the national sample. Since the MTF sample design was not intended for state-level analyses, findings based on MTF should be considered alongside data from other sources. We provide such a companion analysis for California and Washington. We posit that the state-level policy effects measured may be unreliable if the MTF sample for the state does not capture within-state variation sufficiently over time.

Decriminalization in California

California’s SB 1449 was introduced in February 2010, signed into law in September 2010, and took effect on January 1, 2011. The law reduced the sanction for simple possession from a $100 fine and criminal misdemeanor charge to a $100 fine and infraction. Using 2007 to 2013 MTF data, Miech et al. (2015) report past 30-day prevalence among California 12^th graders increased 25% in a period corresponding to the passage of California SB 1449, while prevalence among both of the state’s younger cohorts and 12^th graders in the rest of the United States were unaffected. They conclude “decriminalization may be a risk factor for future increases in youth marijuana use and acceptance.”

As Miech et al. (2015) note, a misdemeanor conviction can have significant negative effects on its recipient, restricting access to federal student loans and eliminating job prospects in some fields. Possession of an ounce of cannabis had been a misdemeanor in California with no other criminal sanction since the enactment of SB 95 in 1975 (California State Legislature, 1975). Nevertheless, SB 1449’s important change in law was a relatively small component of the stimulus high-school-age potential users received regarding cannabis’ legal status around that time. Gov. Schwarzenegger signed the bill into law in advance of a statewide referendum on Prop 19, in part to undermine the latter’s prospects of success (McGreevy 2010). Prop 19 would have legalized and taxed retail cannabis production and sale. Moreover, in February 2009 the legislature considered AB 390 with the same basic goals as Prop 19. These legalization efforts received considerable local and national media attention, as did the rapid growth in the state’s medical cannabis dispensaries beginning in 1996 and extending through the study period. The number of cannabis dispensaries operating in Los Angeles alone exploded from 4 to 600 between 2005 and mid-2010, 400 of which were shut for operating illegally (Webley, 2010).

The findings from 12^th graders in MTF suggest a large increase in past 30-day use prevalence between 2011 and 2012, which differs from the survey’s younger cohorts in the state. They also differ from alternative estimates from comparable surveys, which systematically suggest prevalence was trending upward prior to decriminalization, and continued afterward. Figure 1 plots reported past-month prevalence from California’s MTF alongside the California Healthy Kids Survey (CA-HKS) for grades 7, 9 and 11 and NSDUH respondents ages 12–17. MTF data for all cohorts in California show more volatility than the other comparable series, including a jump in past 30-day use prevalence among 12th graders from 24% to 29%, which Miech et al attribute to the state’s decriminalization law. Indeed, no other series for any age/grade ever shows a one year change of more than 2% at any time; the MTF shows two one year changes of 5 percentage points and one of 4 percentage points.

Fig 1: Trends in reported past-month cannabis use among California adolescents vary across MTF and CA-HKS.

Note: Estimates for CA-HKS were not produced in 2012.

The jump in past-month use in 2012 reported in California’s MTF lags increases in reported less-frequent use by two years and the implementation of SB 1449 by a year or more, depending on the within-year timing of MTF surveys. There is no clear explanation of this lag that would affect only the grade 12 cohort. To wit, none of the series the authors analyze appear to respond to SB 1449 in 2011 when the policy took effect. Rather, an increase in past 30-day use prevalence among 10th graders, from 13% in 2008 to 18% 2009, is precisely measured and at least as large in magnitude as the measured change in 12^th graders between 2012 and 2013, but does not coincide with any cannabis policy action in of note in California.¹

Both CA-HKS and NSDUH aggregate several years of data, so will tend to smooth any spikes or drops, but the trends they each suggest merit consideration alongside MTF. Both alternative series show an upward trend in use through the pre-SB 1449 period that is in line with those reported by MTF 8^th and 10^th graders. The increase continues in CA-HKS through 2013. NSDUH’s measure of use reports flat-to-downward trajectory in the post-decriminalization period, with 2011 as the point of inflection, but its aggregation across age groups loses the granularity needed to compare school grade-specific responses to policy changes.

Considering simple long-term trends in prevalence, Table 1 demonstrates that over the study period employed in Miech et al and comparable series from CA-HKS the rate of change in prevalence increases as the grade cohort increases. We use the standard errors and sample size reported by Miech et al. (2015) to estimate a 95% confidence interval on the trends based on a simulation of approximately 12,000 respondents, so approximately 1,715 per grade per year. The trend for Grade 8 from MTF suggests prevalence increases by 0.51 percentage points each year over the period, 95% confidence interval (CI) [0.12 – 0.89]. This estimated trend falls between the trends in grades 7 and 9 in CA-HKS. The trend in Grade 10 of a 0.75 percentage point increase 95% CI [0.27 – 1.16] in prevalence per year in MTF falls between CA-HKS grades 9 and 11. The highest rate of change is observed among Grade 12 MTF respondent data, which suggest prevalence rate increases by 1.29 percentage points each year, 95% CI [0.39 – 2.26].

Table 1:

Reported cannabis use in the past 30 days in California by grade and data source

MTF Year	2007	2008	2009	2010	2011	2012	2013		95% Confidence Interval on the Trend
CA-HKS Period	2005–2007	2006–2008	2007–2009	2008–2010	2009–2011	Missing	2011–2013	Trend (2007–2013)
NSDUH Period	2006–2007	2007–2008	2008–2009	2009–2010	2010–2011	2011–2012	2012–2013
CA-HKS Grade 7	4	5	6	6	6		7	0.44
MTF Grade 8	6	5	7	9	10	9	7	0.51	0.12 – 0.89
CA-HKS Grade 9	12	12	13	15	15		15	0.61
MTF Grade 10	14	13	18	20	19	17	18	0.75	0.27 – 1.16
CA-HKS Grade 11	16	18	19	20	21		24	1.26
MTF Grade 12	21	23	20	14	24	29	28	1.29	0.39 – 2.26
NSDUH Ages 12–17	6.81	6.88	7.70	8.60	9.43	8.83	7.80	0.31

Note: Reported trend and confidence intervals are based on simulated data using prevalence proportions and standard errors reported by Miech et al. (2015)

The point estimates from both surveys suggest the rate of prevalence growth in California increases with cohort age but the uncertainty around these estimates suggests these trends are statistically indistinguishable. We question the reliability of the provided standard errors as they are estimated. The data required for more rigorous validation are unfortunately not publicly available. This is an area for future study.

The piecewise linear regression method Miech and colleagues employ on MTF data cannot rule out other plausible alternative hypotheses, including the simple idea that the year-to-year volatility observed in each cohort in California’s MTF is an artifact not of cannabis policy changes, but of sample construction. If our hypothesis is correct, a smoothed trend line through the MTF California data would look similar to the other series. While this alone is not confirmatory, it is exactly what we find. This suggests MTF is at minimum useful indicator of the general trajectory of state-level use prevalence over of several years of data, but does not suggest it is appropriate for the accurate measurement of state-level policy impacts.

MTF reports that approximately 12,000 students from 46 different schools contributed 12th grade data over the period 2007–2013, with each school staying in the sample for at most two years. Thus, each year there are approximately 12 different schools contributing data on 12^th graders. In absolute terms, this seem to be a large number, but represents a sample of less than 0.5% of students in a large and diverse state (California Department of Education 2019a). The total MTF sample of approximately 40 to 50 high schools per grade over seven years and three grades is from among more than 3,600 public, private, and charter schools including high school grades in California (California Department of Education 2019b). Depending on the sample size and school participation rates, the MTF sample for an individual grade (each school can contribute only one grade) may not be very stable at the state-level due to systematic non-reporting and the mechanics of school replacement across sub-state areas.

There is little publicly available information about the replacement protocol for schools that don’t participate, but MTF aims to include two or more schools from each of the 28 largest metropolitan areas nationally (Bachman, et al 2006). Miech et al. report that, “replacements are chosen carefully to be as similar as possible to the original school being replaced,” and that, “most regional variation in substance use is within schools and not across them.” However, in California, there is substantial spatial variation in reported use, and variation over time within areas.

The NSDUH sub-state data covering 2008–2010 estimate Region 1R (northern California) had a reported rate that was 80% higher than Region 19R (Imperial County) on the state’s southern border. Local variation is also non-trivial. Point estimates for prevalence ranged between 6.9% and 9.5% in the eight Los Angeles County sub-regions alone. In the subsequent survey covering 2010–2012, estimated prevalence rates fell in eight of the state’s 21 region groups with reported prevalence, though the confidence intervals for the differences were all overlapping.

Since sub-state MTF data are not publicly available, we plot past-month prevalence for NSDUH 12–17 year old respondents in Los Angeles County sub-regions with effective sample sizes of at least 200 respondents per period to highlight the variation in estimates that might occur if, for example, a high school in an otherwise similar but low-prevalence district is replaced with a school in a higher prevalence area, or vice versa. This may affect both within-year samples and comparisons across time (Figure 2). If, for example, a school in LA SPA 3 that was sampled before SB 1449 is replaced by a school in LA SPA 8 after, the estimated difference in prevalence would be a 37% increase rather than the 8% increase actually estimated for the region in NSDUH. This may partially explain the swings in California’s point estimates from MTF.

Fig 2. Reported past-month cannabis use among 12–17 year olds in NSDUH varies widely within Los Angeles County.

Sources: Substance Abuse and Mental Health Services Administration 2014a, b

Since Miech et al was published, major changes in California cannabis policy have continued. In 2015, the state passed the Medical Cannabis Regulation and Safety Act, which was intended to add structure to previously lax medical cannabis regulations, leading to a new set of draft regulations in spring 2017. Before the new regulations were implementation, voters passed Ballot Initiative 64 in 2016, which legalized recreational cannabis production, sale, and possession by adults. The Medicinal and Adult-Use Cannabis Regulation and Safety Act (MAUCRSA) was then passed to create a general framework for the regulation of adult use of commercial and medicinal cannabis in the state (State of California 2019). In June 2018, emergency regulations were passed to speed the enactment of MAUCRSA, but the law was not implemented uniformly. Over 70% of the 540 cities and counties in the state prohibit cannabis retail locations, and development of regulations to shape the new markets has otherwise been slow.

As of mid-2019 only 631 dispensaries operate statewide (McGreevy 2019). While this is double the number a year earlier and roughly equal to the number of cannabis retailers in Oregon (Oregon Liquor Control Commission 2019), it is a signal that the market in California is still relatively nascent. California has five times the number of reported past-month cannabis users as Oregon (Substance Abuse and Mental Health Services Administration 2019). Due to the recency and slow roll-out of these policies subsequent to decriminalization across the state, the impact on prevalence among youths and adults cannot yet been rigorously studied.

Alternative data series describing California youth drug use suggest volatility in MTF data that is difficult to reconcile. The longer-term trends in California’s MTF are consistent with these other data, and with the slow accumulation of policy actions taking place statewide over time. We highlight potential sources of disparity between the MTF and HKS estimates using aggregated youth prevalence indicators from NSDUH, but this mostly suggests ambiguity. The research design used by Miech et al. is more rigorous than simple tabular analyses, but still relies on the strength of underlying data for the purpose of policy impact evaluation. We demonstrate the vulnerability of state-level MTF analyses due to sampling method using more detailed data from Washington in the next section.

Recreational Legalization in Washington

Colorado and Washington passed ballot initiatives legalizing the production, sale, and use of cannabis by adults in 2012, and recreational sales in each state commenced in 2014. Analysis of cannabis use among adolescents in the states based on MTF data suggests the behaviors in response to looser restrictions on cannabis access are not just heterogeneous across states, but idiosyncratic. Cerdá et al. (2017) used a differences-in-differences research design to compare prevalence and risk perception in Colorado and Washington before (i.e., 2010–2012) and after the laws were enacted (2013–2015), to non-recreational cannabis law states over the same period. The analysis used data from 14 schools per period in Colorado, and 24 then 23 in Washington. Ex ante, we might expect similar impacts in the two states. The states had similar pre-legalization prevalence rates, as well as similar regimes for legalizing and monitoring recreational cannabis markets.

The authors found respondents in Colorado showed no response to the establishment of the legal recreational market. Using he Healthy Kids Colorado Survey—also comparable in design to CA-HKS—Brooks-Russell et al. (2019) find no effect on past 30-day or lifetime marijuana use, but the latter do find a statistically significant decline in frequent use among past-month users. In Washington MTF, Cerdá and colleagues found that, “among 8th and 10th graders in Washington, perceived harmfulness of marijuana use decreased and marijuana use increased following legalization of recreational marijuana use.” However, the authors also noted, “sample design may lead to discrepancies between MTF results and those found in other large-scale surveillance efforts.”

Their caveat is well-supported by alternative data. Washington conducts in-school surveys analogous to the CA-HKS and YRBS. The Washington Healthy Youth Survey samples over 200,000 students per year across roughly 1,000 schools, with over 50,000 respondents per year for each of grade 8 and grade 10 and nearly 40,000 respondents per year for grade 12. The state reports statistics from a representative sample of approximately 25,000 8^th, 10^th, and 12^th grade respondents every other year. The representative sample reports a small decrease in reported past-month use among 8^th graders and essentially flat reporting for 10th and 12th graders between 2012 and 2014 (askhys.net, 2017).

It is difficult to reconcile the MTF findings from the tabular results to the state-representative in-state surveys and NSDUH in Washington. NSDUH reports changes in past month prevalence among 12–17 year olds were not different over time in Washington (Substance Abuse and Mental Health Services Administration 2019), but individual-level data are the best available basis for comparison with MTF. Figure 3 plots published point estimates and 95% confidence intervals for all three data sources in Washington. By simple pre-post comparison, in MTF only Washington 10th graders show a statistically significant increase in prevalence rate, but the same group decreased in past-month use. That was also true for 8th graders according to the larger HYS. The NSDUH estimates suggest a positive change for 12–17 year olds between 2012–2013 prior to recreational legalization and 2014–2015 after, but not by a statistically significant margin.

Fig. 3. Trends in reported past-month cannabis use among Washington adolescents vary across MTF, NSDUH, and HYS.

Note: Pre and post periods vary by data source based on data availability. Cerdá et al. (2017) use MTF and define the pre-legalization period as 2010–2012 and post-legalization as 2013–2015; We define in HYS the pre period as 2010–2012 and post as 2014–2016, and in NSDUH the pre period is 2012–2013 and post is 2014–2015.

The HYS and NSDUH prevalence rates are not directly comparable to the adjusted rates computed by Cerdá et al. for several reasons, though none should hamper comparisons of their changes in response to legalization. The authors adjust for observable differences in the samples pre and post-legalization to reduce the risk that differences in sample composition would affect their estimates. By contrast, HYS and NSDUH prevalence rates are weighted state-representative rates. The impact of this difference is ambiguous, and should be negligible if the NSDUH and HYS data for Washington are indeed state-representative. Dilley et al., (2018) demonstrate the differences between HYS and MTF trends persist even when using the Cerdá et al. research design on HYS data.

Cerdá et al. pool several years of MTF data into pre- and post-legalization periods, and we follow suit using HYS. Again, potential bias due to pooling of annual MTF surveys versus bi-annual reporting in HYS is ambiguous but minimal. Additionally, NSDUH is an in-person household survey, rather than a school-based survey, which may also influence reporting, but likely not differentially over time. Finally, we consider calendar year 2013 in the pre-legalization period, unlike Cerdá, since recreational cannabis sales in Washington commenced in July 2014 (Kreamer 2014), though this distinction does not affect our analysis of HYS data.

To test the notion that the 2.0 percentage point (or 32.3%) increase in past 30-day use prevalence in Grade 8 and 4.1 percentage point (25.3%) increase in Grade 10 measured by Cerdá et al. (2017) is potentially confounded by sample construction, we simulate the described MTF sample selection process in Washington using the much larger full HYS data set of approximately 1,200 participating school with at least 15 students in each grade. In Washington, MTF draws from approximately 16 schools per grade per year, and up to 350 students per school. We select 16 schools per grade per survey, and 350 students per school, before and after legalization. Consistent with the MTF sampling restriction capping school participation at two consecutive years, in the simulation we require eight schools from the previous sample be retained, summing to 24 per period, pre and post. We then mimic the sample HYS sample design, sampling 110 schools of the roughly 1,200 public schools in the two surveys prior to legalization and two after. From each sample, we calculate the percent change in past-month use prevalence for each grade pre/post. We repeat this process 10,000 times and report the distributions in Figure 4 for grades 8, 10, and 12. The vertical green lines represent the effect size reported by Cerdá et al. (2017) expressed as a percent change in past-month prevalence from pre- to post-legalization.

Fig 4: Simulated distributions of changes in Washington past-month use prevalence before and after the enactment of Initiative 502.

Note: Green vertical lines represent the percent change in prevalence reported by Cerdá et al. (2017).

We find that grade 8 consumption decreased by statistically significant margins after cannabis legalization. In grade 8 past 30-day prevalence decreased by 22.0%, 95% CI [−25.2% – −18.7%], or 2.1 percentage points, to 7.1%. This is consistent with Dilley et al. (2018). In grade 10 prevalence decreased by 12.7%, 95% CI [−16.3% – −9.4%], or 2.5 percentage points, to 17.2%. For grade 12, we find no effect, and again these findings align with published figures from the smaller HYS random sample.² Our findings for Washington are actually similar to the downward trend in cannabis use prevalence among 8^th graders (−1.3 percentage points, p <.001) and 10^th graders (−0.9 percentage points, p = .07), and null changes among 12^th graders in non-recreational cannabis states reported by Cerdá et al. (2017). This suggests that recreational cannabis legalization had little impact on the state’s youth, which is consistent with findings from Colorado (Brooks-Russell et al. 2019). We believe there is little reason to question the validity of national and near-national level estimates produced from MTF.

The effect sizes estimated by Cerdá et al. (2017) are beyond the tails of the HYS design-based distributions, and are in the tails of the simulated values based on the MTF sample design. From this finding, we believe these are plausible but unlikely estimates considering the wide distribution of potential estimates based on the MTF sample design imposed on the more expansive HYS data.

Standard errors are inversely related to sample size, so it is not surprising that the larger HYS simulation yields smaller distributions. Other factors may also influence the discrepancy. The differences could be because the underlying subpopulation of students drawn by MTF behaves differently than the larger pool sampled in HYS. Johnson et al. (2019) find past-month cannabis use prevalence varied significantly across race/ethnicity groups based on the state-representative HYS sample. HYS-conducted bias analyses suggest the survey may underrepresent students in small and non-urban locations, and students in alternative schools (Hawkins et al. 2013). It is unlikely that MTF fairs better. While the MTF sample includes private schools and HYS is exclusively public schools, the mild difference between the two sample frames is unlikely to account for the differences we observe in the two samples. We are not aware of a reason why one survey would evoke more participation from hard-to-reach sub-populations than the other. If MTF truly captures a segment of students HYS misses, our understanding of youth responses to cannabis policy would benefit from extricating the surveys’ complementarity from their apparent redundancy.

Consistent with the hypothesis that the MTF sample changes within Washington may lead to inconsistent findings in state-level analyses, Dilley et al. (2018) find statistically significant changes in the share of white non-Hispanic respondents and low socio-economic status respondents in the MTF sample between the pre-legalization and post-legalization periods. While these explicit differences can be accounted for as covariates in a regression model, the endogenous characteristics they are intended to proxy may not be accounted for.

Discussion

The temptation to use MTF for state level analyses is strong as an increasing number of states are implementing new regimes of legal cannabis sales. MTF has been the staple of analyses of adolescent cannabis use. The data collected offer a rich description of drug use and health behaviors among youth at the national level in the U.S. Restricted-use MTF data include geographic and school characteristics are the basis for rigorous high-quality research, but as is often the case in academic research, access to these sensitive data is limited. This requires careful stakeholders to connect the dots between similar data sets, or as attempted here, to consider why the dots do not connect.

Miech et al. (2015) conducted an innovative and thorough analysis of the MTF data. They concluded that decriminalization in California was the probable driver of an increase in prevalence of cannabis use in grade 12, and thus suggested that decriminalization was a risk factor for cannabis use among youth. However, prevalence rates in grades 8 and 10 did not increase in MTF. Tellingly, other data do not suggest that 2010 was a turning point year. We demonstrate using Washington HYS that changes in sample composition are more of a threat to the analysis than Miech et al. suggest.

Cerdá et al. (2017) provide one of the first analyses attempting to isolate the causal effect of legalization on youth cannabis consumption and perceptions. However, the analysis of Cerdá et al. (2017) illustrates the same problem created by the fact that the nationally representative MTF survey is not state representative. Considerable within-state heterogeneity of prevalence that turns this caveat into a problem, given the small number of schools in the state-level sample. We find that MTF is useful to describe multi-year trends in youth cannabis use, but caution the use of these data to estimate policy impacts. Stakeholders cannot take any single source of data on the effect of policy on youth cannabis prevalence as gospel. Future MTF analyses should include comparisons with NSDUH youth and state level school data when the latter are available. When findings are disparate and methodological rigor is equivalent, analyses of data sources specifically designed to describe state-level phenomena are more credible.