Scientific Quality of Health-Related Articles in Specialty Cannabis and General Newspapers in San Francisco

Abstract

Recreational cannabis is being legalized in states across the USA. The public relies on popular media for health information about cannabis. We assessed the accuracy of reporting on health effects of cannabis use in GreenState, a specialty publication on cannabis published by the San Francisco Chronicle and the main newspaper using the Index of Scientific Quality for Health Related News Reports. Results were compared using t-tests. Seventeen GreenState articles and four San Francisco Chronicle articles were identified for analysis. Health articles in GreenState scored 2.9 (±1.1 [SD]) Global, with the highest scoring category Applicability (4.5 ± 0.4) and the lowest Precision (2.4 ± 1.0) on a scale of 1–5. In contrast, the San Francisco Chronicle articles received a Global rating of 4.6 (±0.2), ranging from Applicability (5.0 ± 0) to Benefits (3.8 ± 0.9). Articles in the San Francisco Chronicle scored significantly higher in all categories but Benefits which was not significantly different for the San Francisco Chronicle compared with GreenState (3.8 vs. 3.6, p = 0.77). The public, clinicians, and policymakers need to be aware of this pattern and treat information in publications like GreenState with an appropriate level of skepticism until the quality of reporting improves to general journalistic standards.

In the United States, cannabis is legal for medical purposes in 30 states and for recreational purposes in nine states and Washington, D.C. Americans view cannabis as beneficial for many health conditions. The perception of “great risk” from weekly cannabis use dropped from about 50% in 2002 to about 34% in 2014 (Azofeifa et al., 2016; Compton, Han, Jones, Blanco, & Hughes, 2016). In 2017, 81% of US adults believed cannabis had at least one medical benefit, most commonly pain management (66%), followed by treatment of epilepsy and multiple sclerosis (48%), and relief from anxiety, stress, and depression (47%), with 29.2% agreeing that smoking marijuana prevents health problems (Keyhani et al., 2018). Eighteen percent believed exposure to secondhand cannabis smoke is somewhat or completely safe for adults, 8% indicated it was somewhat or completely safe for children, and 7% agreed that marijuana use is somewhat or completely safe during pregnancy (Keyhani et al., 2018). Consistent with these changing perceptions, cannabis use among US adults more than doubled between 2001 and 2013 (Hasin et al., 2015). Despite increased legalization and use, conclusive evidence regarding the health effects of cannabis remains limited. In 2017, the National Academies of Science, Engineering and Medicine summarized the evidence on direct human health effects to date (National Academies of Sciences, Engineering, and Medicine, 2017), highlighting the lack of conclusive research on many important topics. (NASEM only found conclusive evidence to support use of cannabis for treatment of chronic pain in adults, to treat chemotherapy-induced nausea and vomiting, and for improving reported symptoms of multiple sclerosis.) Because more Americans are using cannabis and legalization is influenced by popular media, it is important to understand the sources from which people obtain information about the health effects of cannabis.

The media impact the opinions and behaviors of both patients and clinicians (Moynihan et al., 2000; Stryker, 2003). Positive news coverage of cannabis has been tied to lower risk perception and less disapproval (Beaudoin & Hong, 2012). Similarly, lower risk perception was found among adolescents who viewed websites communicating how to use cannabis and its benefits (Belenko et al., 2009). In a 2013 survey of college students, over half reported encountering cannabis information during routine media use (Cheng, Ahn, Lewis, & Martinez, 2017).

The news media often provide incomplete information on the harms of medical treatments and overreport the benefits (Moynihan et al., 2000; Van Trigt et al., 1995). When reporting on scientific research, the news media also tend to understate limitations and exaggerate results (Woloshin & Schwartz, 2002). Consistent with this trend, patients overestimate the benefits and underestimate the harm of medical interventions (Hoffmann & Del Mar, 2015).

News coverage on cannabis increased in several states after legalization (McGinty et al., 2016), and cannabis-focused specialty publications have emerged in several states (Ibsen, 2018). One such publication, GreenState, is published online and distributed in print by the San Francisco Chronicle (Hearst Corporation, 2018). The San Francisco Chronicle and GreenState were chosen for analysis because of the timeliness of recreational cannabis legalization in California, the size of this cannabis market (Friese, Slater, & Battle, 2017), and the large readership of these publications. GreenState, first published in November 2017, is an online magazine that occasionally appears in print as an insert in the San Francisco Chronicle print version. GreenState magazine is directed specifically toward the newly legalized California cannabis market and is poised to serve as an important source of information for a growing number of Californians. We compared Green State with the San Francisco Chronicle because they share the same publisher but have no overlap in writers or editors. These publications target different readerships: GreenState is a publication targeted at people interested in cannabis while the San Francisco Chronicle is a general audience newspaper.

While cannabis-related content in general market media has been described, there has not yet been formal study of the content of these cannabis-focused specialty publications. This lack of knowledge about cannabis-focused publications, combined with increasing acceptance of alternative medicine by the medical field (Frass et al., 2012) and the recent legalization of recreational cannabis, led to our research question: Are the health benefits and risks of cannabis use accurately presented in specialty cannabis-focused media and how does this compare to other general audience media?

Methods

Article Selection

GreenState (www.greenstate.com) articles were selected for analysis if they were determined to be health related. A “health related” article is one which discussed the medical benefits, toxicity, or a health outcome pertaining to cannabis use. A search for the term “health” on 9th January 2018 yielded 64 articles published between 7th November 2017 (the date of GreenState’s first article) and 9th January 2018 (GreenState, 2018). Seventeen of the 64 articles were independently categorized as “health related” by five reviewers (the four authors of this report and an attorney who works on cannabis). In addition, we searched the San Francisco Chronicle (www.sfchronicle.com) using the terms “cannabis health,” “marijuana health,” and “pot health” on 16 January 2018. Four health related articles were identified that were published between 7 November 2017 through 9 January 2018. These 21 articles are evaluated in this report.

Article Scoring

Articles were scored using the Index of Scientific Quality (ISQ) for Health Related News Reports (Oxman et al., 1993). The ISQ has been used in previous research to assess the quality of health information in the general market media (Hasin et al., 2015; Lewis, Orrock, & Myers, 2010; Molnar et al., 1999; Krauth & Apollonio, 2015). This index captures important qualities in accurate and effective news reporting, particularly on a substance with medical usage. The ISQ is comprised of eight categories: Applicability, Opinions versus Facts, Validity, Magnitude, Precision, Consistency, Consequences, and Global. The Consequences category was subdivided into Consequences: Benefits and Consequences: Risks and Costs, for a total of nine categories (Table 1).

Table 1.

Index of scientific quality

Metric	Key question	Criterion for “No”	Criterion for “Partially”	Criterion for “Yes”
Applicability	Is it clear to whom the information in the report applies (i.e. to which population the evidence is applicable)?	Potentially misleading	Minor lack of clarity	Minimal ambiguity
Opinions versus Facts	Are facts clearly distinguished from opinions?	Potentially misleading	Statements are attributed to sources, but the underlying evidence is ambiguous	The evidence underlying the main points is clearly cited
Validity	Is the assessment of the credibility (validity) of the evidence clear and well-founded (not misleading)? (In making this assessment take into account only the explicit message, not the implicit assessment.)	Not done or potentially misleading	Study design or type of evidence reported, but not properly assessed	Strength of the research methods adequately assessed
Magnitude	Is the strength or magnitude of the findings (effects, risks, or costs) that are the main focus of the article clearly reported?	Not done or potentially misleading	The magnitude of effects or risks is reported incompletely or ambiguously	Magnitude of main effects or risks clearly reported (including, if relevant, the baseline risk and dose-response relationship
Precision	Is there a clear and well-founded (not misleading) assessment of the precision of any estimates that are reported or of the probability that any of the reported findings might be due to chance?	Not done or potentially misleading	Indirectly or not completely; e.g. sample size reported but not properly assessed	Statistical significance or precision adequately assessed
Consistency	Is the consistency of the evidence (between studies) considered and is the assessment well-founded (not misleading)? (In making this assessment take into account only the explicit message, not the implicit assessment.)	Not done or potentially misleading	More than one study discussed, some ambiguity re how many studies there are or their consistency	Number of studies and consistency (with respect to the direction of their findings) clearly reported
Consequences: Benefits	Are all of the important consequences (benefits) of concern relative to the central topic of the report identified?	Potentially misleading	Potentially important benefits not considered	Most important benefits are clearly identified
Consequences: Risks and Costs	Are all of the important consequences (risks and costs) of concern relative to the central topic of the report identified?	Potentially misleading	Potentially important risks or costs not considered	Most important risks or costs are clearly identified
Global	Based on your answers to the above questions, how would you rate the overall scientific quality of the report?	Critical or extensive shortcomings	Potentially important but not critical shortcomings	Minimal shortcomings

Applicability scores reflect whether it was clear to whom the evidence in the article applied. Opinions versus Facts represents whether statements made in the articles were the opinions of the reporter or scientific facts. Validity assesses how the article analyzed the credibility of the evidence. Magnitude reflects whether an article misrepresented the literature by either over or under emphasizing the effects, risks, or costs of cannabis usage. Precision assesses the precision of reported statistical claims and estimates. Consistency examines whether an article discussed the consistency of the evidence in the context of other studies. Consequences: Benefits and Consequences: Risks and Costs represent whether the article considered relevant benefits or risks associated with findings. Global scores reflect the overall scientific quality of the article.

A panel of three scorers (the first three authors of this report) independently scored each article. Discrepancies were resolved in a face to face meeting, after which each scorer had the option to alter his or her scores. The mean of these three scores was compiled for each category for every article, and the mean of these scores is reported for each of the nine categories for GreenState and the San Francisco Chronicle. Intercoder reliability was assessed using a two-way mixed effects single measure intraclass correlation (Overall ICC 0.78, 95% CI 0.73–0.82; Global ICC 0.83, 95% CI 0.68–0.92).

Comparison to Scientific Literature

The claims reported in the articles were compared with the National Academies of Science, Engineering and Medicine’s The Health Effects of Cannabis and Cannabinoids: The Current State of Evidence and Recommendations for Research (National Academies of Sciences, Engineering, and Medicine, 2017). The NASEM report is a “comprehensive, in-depth review of the most recent evidence regarding human health effects of using cannabis and cannabis-derived products,” prepared at the request of the US Centers for Disease Control and Prevention, Food and Drug Administration, National Institute on Drug Abuse, and other philanthropic and non-governmental organizations. The writing committee considered over 10,700 abstracts published since the previous NASEM report on the health effects of cannabis in 1999, with emphasis on “recently published systematic reviews and high-quality primary research.” (The committee only considered human studies, not research conducted using animals.) As of July 2018, this report had been cited at least 178 times (Google Scholar) since it was published in 2017. The scope, breadth, and timeliness of the report made it an authoritative source to use as a comparator for statements in the articles we reviewed.

Analysis

Mean scores in each category for GreenState and the San Francisco Chronicle were compared using unpaired t-tests. (We also conducted a sensitivity analysis by redoing the analysis with Mann-Whitney rank-sum tests; the results were qualitatively the same.) Intercoder reliability was assessed using a two-way mixed effects single measure intraclass correlation. All analyses were performed using Stata 14.2/IC (College Station, TX).

To complement our quantitative analysis, we incorporated a qualitative description of themes prevalent in GreenState and San Francisco Chronicle articles. The qualitative analysis was two-fold. First, relevant quotations were selected from both publications that represented trends in the ISQ scores. Second, reviewers identified general themes present in articles that influenced but were not explicit components of the ISQ system (e.g. types of evidence and promotional content).

Results

Seventeen GreenState articles and four San Francisco Chronicle articles were identified for analysis.

Quantitative Analysis

Health articles in GreenState (Table S1) scored 2.9 (±1.1 [SD]) Global, with the highest scoring category Applicability (4.5 ± 0.4) and the lowest Precision (2.4 ± 1.0) (Table 2). In contrast, San Francisco Chronicle articles received a Global rating of 4.6 (±0.2), ranging from Applicability (5.0 ± 0) to Benefits (3.8 ± 0.9). Articles in the San Francisco Chronicle (Table S2) scored significantly higher in all categories except Benefits which was not significantly different from GreenState (3.8 vs. 3.6, p = 0.77).

Table 2.

Comparison of mean (standard deviation) scores

Category	GreenState	San Francisco Chronicle	P value*
Applicability	4.5 (0.4)	5.0 (0.0)	0.04
Opinions vs Facts	3.3 (1.2)	4.9 (0.2)	0.01
Validity	3.0 (1.4)	4.8 (0.3)	0.02
Magnitude	2.9 (1.3)	4.7 (0.4)	0.01
Precision	2.4 (1.0)	4.3 (0.4)	0.002
Consistency	2.8 (1.2)	4.8 (0.2)	0.004
Consequences: Benefits	3.6 (0.9)	3.8 (0.9)	0.77
Consequences: Risks and Costs	2.8 (1.4)	4.8 (0.4)	0.009
Global	2.9 (1.1)	4.6 (0.2)	0.006
*Unpaired t-test.

Because four GreenState articles contained promotional content, analysis was repeated after removing these articles. When our analysis was repeated with these articles removed, the Applicability and Validity categories were no longer significantly lower than the San Francisco Chronicle, but all other categories were unchanged (Table S3). Further, we evaluated the differences between promotional and non-promotional articles within GreenState (Table S4). When compared to non-promotional GreenState articles, promotional articles scored significantly lower in Validity, Magnitude, Precision, Consistency, and Consequences: Risks and Costs.

Qualitative Analysis

The supplemental file contains detailed results for each article and Tables S1 and S2 summarize these results. Some illustrative examples are presented below.

Promotional Content

Nearly 25% of the GreenState articles were promotional in nature. Several promotional articles were included in the “Health” section. Further, the promotional articles frequently conflated content consisting of advertising information with health claims, for example, in the GreenState article “Does ‘Spraying’ Cannabis Beat Smoking It”, “Many patients are using them medically…Formulated to treat menstrual cramps, hormone-related stress and other ‘reproductive concerns.’”

Use of Anecdotal Evidence When Medical Evidence Is Lacking GreenState articles often relied on anecdotal evidence to support their claims. For example, in a GreenState article about maternal cannabis use and academic performance, the author claims to know “several healthy babies whose mothers used cannabis throughout their pregnancy.” However, there is evidence that cannabis use during pregnancy has negative health effects (National Academies of Sciences, Engineering, and Medicine, 2017, conclusions 10–1, 10–2, 10–3). In contrast, we did not find similar use of anecdotal evidence in the San Francisco Chronicle.

Articles Featuring Interviews with Health Professionals

Interviews in the San Francisco Chronicle were conducted with well-established research physicians, and published quotes were representative of the state of medical knowledge. In an article on cannabinoid hyperemesis syndrome, interviews take place with the director of a large hospital and a Stanford research physician. The information provided by these interviews agreed with the science published in the NASEM report.

Use of Credible Evidence

San Francisco Chronicle articles consistently based their articles on high quality studies. For example, the article “More pregnant women are using marijuana, study says” cited a Kaiser Permanente study published in the Journal of the American Medical Association that surveyed cannabis use in over 300,000 pregnant women. Several GreenState articles, such as “Challenging assumptions around marijuana addiction” and “Case studies from the field of medical cannabis for gerontology,” made claims without citing peer-reviewed research.

Failure to Consider Risks

The GreenState article “Sublingual relief: From underground to under the tongue” described that cannabis use resulted in fewer side effects and less pain for treatment of rheumatoid arthritis, but did not discuss any of the short-term adverse effects of cannabis. This article also relied on anecdotal evidence from a single individual and promoted products (see themes above).

Discussion

The specialty publication GreenState, which reported on health issues related to cannabis, exhibited significant shortcomings when compared to its parent publisher, the San Francisco Chronicle. While GreenState reported equally well on the benefits of cannabis as the San Francisco Chronicle (p = 0.77), it reported significantly worse on risks (p = 0.009), as well as on all other ISQ categories. The differences between GreenState and the San Francisco Chronicle persisted even after removing the promotional articles from analysis in all but two categories. Within GreenState, the promotional articles scored significantly lower than the non-promotional articles in six out of nine categories. Further, many troubling themes emerged in GreenState, such as conflation of advertisement and reporting, supporting claims with anecdotal evidence, and failing to cite research for claims of the health benefits of cannabis.

GreenState’s limited reporting on the negative consequences of cannabis use aligns with documented trends in news coverage of medical interventions and cannabis (Keyhani et al., 2018; Moynihan et al., 2000). Bias toward reporting on positive aspects of use is consistent with previous findings on reporting of alternative therapies (Milazzo & Ernst, 2006). Additionally, GreenState scored significantly lower in categories assessing accurate representation of study results (e.g. Magnitude, Validity, and Precision), which has been noted in health news reporting (Woloshin & Schwartz, 2002).

The finding that this specialty publication overstated the positive impact of cannabis use has important implications. The media influence public perception of cannabis, which has become increasingly favorable (Keyhani et al., 2018; Moynihan et al., 2000). Recent studies have also shown positive reporting leads to increased cannabis use in adolescents (Roditis, Delucchi, Chang, & Halpern-Felsher, 2016; Stryker, 2003).

The 17 GreenState articles were written by six freelance journalists, who often are involved in aspects of the cannabis business (Table S5). In contrast, the San Francisco Chronicle employs full time health reporters, who cover cannabis among other health issues. No writers were authors for both publications. Understanding the background of the authors is important, as differences in qualifications and expertise could account for some of the difference in quality between the two publications.

Increased cannabis use will likely necessitate more frequent conversations between clinicians and patients on the health effects of cannabis as both an intervention, as well as a recreational substance. It has been demonstrated that the majority of patients overestimate benefits and underestimate harm of medical treatments (Hoffmann & Del Mar, 2015). Since the research on cannabis usage remains generally inconclusive (National Academies of Sciences, Engineering, and Medicine, 2017), it is important for the patient-physician relationship that media represent the actual state of knowledge in their reporting.

Another concern is the accuracy of media in states considering legalization, since media can affect peoples’ perceived societal risk of use (Beaudoin & Hong, 2012). Moving forward, media guidelines on reporting of drug use should be examined and strengthened. Analogous guidelines to those imposed on alcohol or tobacco could be considered (Barry, 2016; Keyhani et al., 2018; Pacula, Kilmer, Wagenaar, Chaloupka, & Caulkins, 2014).

Limitations

There are several limitations of this research. Our sample size was relatively small for both GreenState and the San Francisco Chronicle. We examined 17 articles in GreenState, which at the time of analysis was exhaustive. In the San Francisco Chronicle we selected all articles (4) in the same time window as GreenState, since articles written during this time period would be based on the same published literature. Additionally, we exclusively analyzed San Francisco publications. It will be important for future researchers to conduct similar analyses on additional cannabis-focused publications as more states legalize cannabis use, and especially since media coverage increases in states which have legalized or are considering legalizing cannabis (McGinty et al., 2016). The ISQ requires a high standard of scientific rigor and may not assess articles along the same metrics newspapers use to evaluate their own stories (Wells, Marshall, Crawley, & Dickersin, 2001). However, we applied the same standard to both publications and found significant differences between publications, but there are other issues that may be relevant in analyzing the quality of articles in the general media market.

Conclusion

Trust in information from the Internet and media has been increasing rapidly in the past decade (Ek, Eriksson-Backa, & Niemelä, 2013). Over half of US adults claim to pay attention to health news (Schwartz & Woloshin, 2004). Therefore, it is especially important to fairly represent the health benefits and risks of cannabis to the public. As cannabis specialty publications become more commonplace, the public, clinicians, and policymakers need to be aware to treat information in publications like GreenState with an appropriate level of skepticism until the quality of reporting improves.