National cross-sectional survey of US adults to assess the reliability of current and lifetime cannabis smoking

Abstract

Objective

The objective is to examine the test–retest reliability and internal reliability of six self-report questions assessing both current (past 30 days) and lifetime cannabis smoking in an internet survey in the adult US population.

Design

Cross-sectional national survey.

Participants

Out of 957 US adults who completed a national 2020 survey administered through Ipsos KnowledgePanel, 557 completed a second survey (‘reliability survey’) aimed at assessing the test–retest and internal reliability of questions asking about current and lifetime cannabis smoking. The sample size used in the analysis for the six self-report questions varied and was dependent on respondents answering the questions in both the 2020 survey and the reliability survey.

Primary outcome measure

Test–retest and internal reliability of six self-report questions asking about current and lifetime cannabis smoking.

Results

Among respondents who had smoked cannabis in the past 30 days, 33.8% were aged 18–34, 29.7% were 35–49, 27.7% were 50–64 and 8.8% were 65 or older. Current cannabis smokers were primarily men (59.5%) and white (63.0%). Almost half of current cannabis smokers had a high school diploma or less followed by some college and a bachelor’s degree or higher (45.7%, 30.6%, 23.8%, respectively). The question assessing number of days participants smoked cannabis in the past 30 days demonstrated good test–retest (r=0.87) and excellent internal reliability (α=0.94). The questions assessing the number of years, the most common form of use and the number of times participants smoked cannabis over their lifetime also demonstrated test–retest (r=0.77 (acceptable), r=0.75 (acceptable) and κ=0.65 (substantial), respectively) and excellent-to-good internal reliability (α=0.91, α=0.87 and α=0.88, respectively).

Conclusions

We found simple questions assessing current and lifetime cannabis use to demonstrate both test–retest reliability and internal reliability. These questions can serve as a simple framework for clinicians to evaluate the frequency of cannabis smoking in their patients. Future work should examine if these simple frequency measures of smoking cannabis are associated with adverse health outcomes.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• Test–retest reliability and internal reliability of cannabis use were tested in a national sample of US adults.

• Generalisability may be limited as those who chose to participate may differ from those who chose not to participate.

• We did not test the validity of the cannabis use measures.

Introduction

Cannabis use is legal in some form in 37 states and Washington, DC. Changes in the prevalence of cannabis use have accompanied legalisation. Past year cannabis use increased from 10.1% in 2003 to 18.7% in 2020 and past month use doubled from 6.0% in 2003 to 12.4% in 2020 in the USA.^{1 2}

Given the rising prevalence of cannabis use in the general population, there is a need for standardised questions with demonstrated validity and reliability to quantify cannabis use, particularly cannabis smoking, which is the most common form of use.³ The National Academy of Sciences has called for the development of robust standardised tools to quantify cannabis use.⁴ Such tools are necessary to understand the risks and benefits of cannabis use. Currently, cannabis use is assessed as part of national surveys; however, each survey has a different approach to assessing use. National surveys such as the Behavioral Risk Factor Surveillance System (BRFSS), the National Survey on Drug Use and Health (NSDUH), Monitoring the Future (MTF) and the National Longitudinal Study of Adolescent to Adult Health (Add Health) Wave IV, vary in the formats of questions and answer choices. BRFSS and NSDUH ask ‘during the past 30 days, on how many days did you use marijuana or hashish?’ and participants respond with the number of days or choose ‘none’.^{5 6} MTF asks ‘on how many occasions (if any) have you used marijuana (grass, pot) or hashish (hash, hash oil) during the last 30 days?’ and provides the answer choices: 0 occasions, 1–2 occasions, 6–9 occasions, 20–39 occasions and 40 or more occasions.⁷ The Add Health Wave IV uses the same question as BRFSS, but uses different answer choices: none; 1 day; 2 or 3 days; 1 day a week; 2 days a week; 3–5 days a week; every day or almost every day.⁸ Among these different methods of assessing frequency, the survey question used by BRFSS can be adapted to assess past month frequency of cannabis use by both asking about days of use and number of times cannabis is used on average each day.

In this study, we were interested in identifying a simple tool that could be reliably used to assess the past month frequency of cannabis use and lifetime cannabis use. We examined the test–retest reliability of six self-report questions assessing both current (past 30 days) and lifetime cannabis smoking in an internet survey in the adult US population. We focused on smoking cannabis because it is the most common form of cannabis use in the general population and there are established harms associated with smoking.^{3 9 10}

Methods

Survey administration

Ipsos KnowledgePanel is a probability-based, nationally representative online panel composed of 60 000 civilian, non-institutionalised US adults.¹¹ A national cross-sectional survey was launched through Ipsos KnowledgePanel in October 2017. A follow-up study (referred to now as the ‘2020 survey’) aimed at gathering information on cannabis and tobacco use, and perceptions towards risk of use was launched on 5 August 2020 to the 9003 US adults age 18 years and older who had completed the original survey in October 2017. Questions on the frequency of past month and lifetime cannabis use were added to this follow-up 2020 survey.¹² Data collection for the 2020 survey was completed on 1 September 2020. Respondents to the 2020 cross-sectional survey who reported that they had smoked cannabis within the past year were eligible to complete an additional follow-up cross-sectional survey (referred to as the ‘reliability survey’, see online supplemental additional file 1), which repeated questions regarding current and lifetime cannabis smoking to measure their test–retest reliability. Ipsos fielded the reliability survey two times per week until data collection was completed on 17 September 2020. Ipsos has created an incentive structure to ensure an adequate response rate. For completing the 2020 survey, participants received 5000 points. Participants received another 5000 points for completing the reliability survey. Participants can redeem points for rewards, such as gift cards and virtual prepaid cards, through Ipsos’ rewards catalogue.

Survey questions

Questions assessing past 30-day use

First, we asked participants how long it had been since they last used cannabis (within the past 30 days, more than 30 days ago but within the past 12 months or more than 12 months ago). If they had used cannabis in the past 30 days, we asked in what form they had used it (smoking, vaping, edibles, dabbing concentrate (wax/shatter) and/or topically). If they had smoked cannabis in the past 30 days, they were asked, ‘on how many of the past 30 days did you smoke marijuana in a joint, pipe or bong?’. Then, they were asked how many joints, pipes or bongs they smoked per day.

Questions assessing lifetime use

If a participant responded that they had smoked marijuana either within the past 30 days or more than 30 days ago but within the past 12 months, they were asked the question ‘over the entire period you were smoking marijuana, how many years did you smoke marijuana on a daily or near daily basis?’ to assess lifetime use. This question was adapted from the Addiction Severity Index, which uses the question ‘how many years have you regularly used (substance)?’ to assess drug use.¹³ If participants smoked at least 1 year on a daily or near daily basis, they were asked in which form they had most often smoked cannabis (joints, pipes or bongs). Next, they were asked how many joints, pipes or bongs they smoked per day during the year(s) they smoked on a daily or near daily basis. If participants entered that they smoked 0 years on a daily or near daily basis, they were asked to pick which category (1–50, 51–500, 501–1000 or more than 1000) best described the total number of times they have smoked marijuana over their lifetime.

Statistical analysis

The response rate, determined by using methods outlined by the American Association for Public Opinion Research, was the ratio of respondents to all participants who received the survey.¹⁴ Results were weighted by using weights provided by Ipsos to approximate the US population on the basis of age, sex, race, ethnicity, education level, household income and metropolitan area (see online supplemental additional file 2). All analyses used weighting commands based on variables provided by Ipsos to generate national estimates. Online supplemental figure A1 shows how many respondents were included in the analysis of each question for current and lifetime cannabis use. Respondents were dropped from the analysis if they did not answer the question in both the 2020 survey and the reliability survey. As such, the sample size used to conduct the analyses varied for each question. The sample size used for the analysis for each question is shown in both online supplemental figure A1 and table 1. Sample size was limited by the number of participants available. No a priori sample size calculations were performed. Descriptive statistics were calculated for all survey items. Test–retest reliability of cannabis use questions was measured using Pearson’s correlation coefficient. Previous research has recommended that a Pearson’s correlation coefficient of at least 0.70 is necessary for use of health status questionnaires.¹⁵ Test–retest reliability coefficients are categorised as follows: r=1: perfect; r>0.9: excellent; 0.9>r≥0.8: good, 0.8>r≥0.7: acceptable; 0.7>r≥0.6: questionable; 0.6>r≥0.5: poor; r>0.5: unacceptable; r=0: none.¹⁶ For categorical questions (‘during the years that you smoked on a daily or near daily basis, in which form did you most often smoke marijuana?’ Joints, pipes or bongs; ‘which category best describes the total number of times you’ve smoked marijuana over your lifetime?’ 1–50, 51–500, 501–1000 or more than 1000), Cohen’s Kappa coefficient (agreement) was calculated. Kappa coefficients from 0.61 to 0.80 are considered ‘substantial’.¹⁷ Internal reliability was measured for all questions using Cronbach’s alpha. Alpha values are categorised as follows: α>0.9: excellent; 0.9>α>0.8: good; 0.8>α>0.7: acceptable; 0.7>α>0.6: questionable; 0.6>α>0.5: poor; 0.5>α: unacceptable.¹⁸ Analyses were performed using SAS V.9.4.

Table 1. Reliability of cannabis assessment tools.

Question	Response choices	Complete observations	Pearson’s r or agreement	Cronbach’s alpha
Current cannabis use
On how many of the past 30 days did you smoke marijuana in a joint, pipe or bong?	___ days	329	0.87* (p<0.0001)	0.94*
On those days, how many joints, pipes or bongs did you smoke per day?	___ joints, pipes or bongs per day	328	0.16 (p=0.0038)	0.26
Lifetime cannabis use
Over the entire period you were smoking marijuana, about how many years did you smoke marijuana on a daily or near daily basis?	___ years	528	0.77* (p<0.0001)	0.91*
During the years that you smoked on a daily or near daily basis, in which form did you most often smoke marijuana?	JointsPipesBongs	330	0.75* (95% CI 0.67 to 0.83)	0.87*
During the __ years that you smoked (joints/pipes/bongs) on a daily or near daily basis, how many (joints/pipes/bongs) did you smoke per day?	___ joints___ pipes___ bongs	329	0.67 (p<0.0001)	0.67
Which category best describes the total number of times you’ve smoked marijuana over your lifetime? Please choose the option which best captures your use.	1–5051–500501–1000More than 1000	137	0.65* (95% CI 0.51 to 0.80)	0.88*

^*Indicates qQuestion determined to be reliable.

Patient and public involvement

Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.

Results

Participant characteristics

Of 957 eligible respondents (those who reported smoking cannabis either ‘within the past 30 days’ or ‘more than 30 days ago but within the past 12 months’) from the 2020 survey, 557 (58.2%) completed the reliability survey, of which 435 (78.1%) smoked cannabis in the past 30 days. Table 2 summarises the demographic characteristics of respondents to the 2020 survey and reliability survey, as well as of respondents with current cannabis use. Among respondents who had smoked cannabis in the past 30 days, 33.8% were aged 18–34, 29.7% were 35–49, 27.7% were 50–64 and 8.8% were 65 or older. The majority of current cannabis smokers were men (59.5%) and white (63.0%). Almost half of current cannabis smokers had a high school diploma or less followed by some college and a bachelor’s degree or higher (45.7%, 30.6%, 23.8%, respectively).

Table 2. Baseline characteristics of respondents to the 2020 survey, reliability survey and current cannabis smokers^*.

Characteristic	2020 Survey respondents(n=957)	Reliability survey respondents†(n=557)	Current cannabis smokers‡(n=435)
Age, years
18–34	38.0%	35.5%	33.8%
35–49	26.7%	28.5%	29.7%
50–64	24.5%	26.8%	27.7%
≥65	10.8%	9.2%	8.8%
Gender
Male	51.1%	58.7%	59.5%
Female	48.9%	41.3%	40.5%
Race/ethnicity
White	63.2%	63.8%	63.0%
Black	15.2%	16.3%	15.6%
Hispanic	15.6%	5.3%	6.5%
Other	6.2%	14.6%	14.9%
Education
High school diploma or less	39.3%	45.3%	45.7%
Some college	29.1%	29.1%	30.6%
Bachelor’s degree or higher	31.7%	25.7%	23.8%
Employment status
Working	69.8%	67.9%	66.1%
Not working	30.2%	32.1%	33.9%
Household income
<US$20 000	14.6%	16.6%	16.4%
US$20 000–US$49 999	21.9%	24.9%	26.2%
US$50 000–US$74 999	18.9%	19.9%	21.1%
≥US$75 000	44.6%	38.6%	36.4%

^*Values are weighted percentages, weighted to the 2020 US. population.

^†Eligible follow-up survey respondents were those who reported smoking cannabis either ‘within the past 30 days’ or ‘more than 30 days ago but within the past 12 months’ in the main survey.

^‡Current cannabis smokers were those who reported smoking cannabis ‘within the past 30 days’ in the reliability survey.

Current use

Test–retest reliability

Participants with current cannabis use were reliably able to estimate the number of days they had smoked a joint, pipe or bong in the past 30 days (r=0.87, p<0.0001, good reliability) (table 1). Participants were not able to reliably estimate the number of joints, pipes and bongs per day (r=0.16, p=0.0038, unacceptable reliability) (table 1).

Internal reliability

The question ‘on how many of the past 30 days did you smoke marijuana in a joint, pipe or bong?’ demonstrated excellent internal reliability (α=0.94) (table 1). The question ‘on those days, how many joints, pipes or bongs did you smoke per day?’ was not reliable (α=0.26, unacceptable reliability) (table 1).

Lifetime use

Test–retest reliability

Among all respondents to the reliability survey, individuals were reliably able to estimate the number of years they had smoked cannabis on a daily or near daily basis (r=0.77, p<0.0001, acceptable reliability) (table 1). Participants were reliably able to answer the question ‘during the years that you smoked on a daily or near daily basis, in which form did you most often smoke marijuana?’ (agreement=0.75, 95% CI=0.67 to 0.83, substantial reliability) (table 1). The measure asking how many joints, pipes or bongs the participants smoked per day during the years they smoked on a daily or near daily basis did not meet the threshold of acceptable reliability (r=0.67, p<0.0001, questionable reliability) (table 1). The categorical measure asking ‘Which category best describes the total number of times you’ve smoked marijuana over your lifetime’ was reliable (agreement=0.65, 95% CI 0.51 to 0.80, substantial reliability) (table 1).

Internal reliability

The questions assessing the number of years participants had smoked cannabis on a daily or near-daily basis (α=0.91, excellent reliability) and the form in which they had most often smoked cannabis were found to be internally reliable (α=0.87, good reliability) (table 1). The question asking for participants to estimate the number of joints, pipes or bongs they smoked per day over the entire period they smoked cannabis on a daily or near-daily basis did not meet the threshold of acceptable reliability (α=0.67, questionable reliability) (table 1). The categorical question asking for the total number of times participants had smoked cannabis over their lifetime demonstrated good internal reliability (α=0.88) (table 1).

Discussion

We assessed the test–retest reliability of internet survey items on current (past 30 day) and lifetime cannabis smoking in a national survey of US adults. We found the question ‘how many of the past 30 days did you smoke marijuana in a joint, pipe or bong?’ to demonstrate both test–retest reliability and internal reliability. We also found that participants were able to reliably report how many years they had smoked cannabis on a daily or near daily basis and in which form they most often smoked cannabis during this time. These two questions also demonstrated internal reliability. Our prior research has also shown the lifetime measures were reliable in a group of 100 Veterans.¹⁹ At last, we found that participants who never smoked on a daily or near daily basis were able to reliably estimate the total number of times they had smoked cannabis in their lifetime. This question was also found to be internally reliable. There are standard methods to assess the frequency of alcohol and tobacco use. In contrast, there are no uniformly agreed on standard units of measurement or standardised questions to evaluate the frequency of cannabis use. Cannabis use frequency has been measured as part of tools designed to assess cannabis use disorder, but no tool uses the same time period of assessment. For example, the Cannabis Use Disorder Identification Test Short Form²⁰ asks about cannabis use in the past 6 months and the Diagnostic and Statistical Manual of Mental Disorders²¹ frames questions regarding use in the past year. Furthermore, many of these self-report instruments have not been assessed for test–retest reliability. The only self-report instrument that has been studied for its test–retest reliability that we could identify was the timeline followback (TLFB) calendar. The TLFB calendar has been found to be a reliable way of assessing cannabis use at different intervals.22,24 However, it can take 10–30 min to complete depending on the time period being evaluated and this may be too long depending on the setting (ie, in a clinical appointment).²⁵ Studies show that time is one of the barriers to substance use screening in primary care.²⁶ Thus, having a short screening tool is key to improving cannabis use documentation in the medical record. Additionally, implementing a simple and standard method of quantifying use would support epidemiologic studies focused on understanding the risk and benefits of cannabis use. Quantifying cumulative cannabis smoked is particularly important for cancer and cardiovascular (eg, coronary artery disease) and respiratory diseases (eg, chronic obstructive pulmonary disease) that develop after longstanding smoking and exposure to particulate matter.^{27 28} This study demonstrated the number of days smoked in the past 30 days and years of lifetime smoking can be reliably assessed using only a few questions. Future validation of these simple measures in studies examining the association of cannabis use assessed with these metrics and health outcomes is necessary to determine if these metrics also are useful in identifying health risks associated with cannabis use.

Limitations

Generalisability may be limited since this was an internet survey. However, Ipsos provides tablets and internet for those without access. Additionally, Ipsos KnowledgePanel provides a statistically valid representation of the US population, including harder-to-reach populations, such as African Americans, Latinos and rural communities.¹¹ Those who choose to participate may be different from those who choose not to do so. However, past work has shown that there were few differences in the demographic characteristics of internet panels and other national surveys.²⁹ Generalisability of the study may also be limited due to the small final sample sizes used in the analyses for the current and lifetime cannabis smoking questions. Future work should aim to include larger sample sizes to decrease the effect of selection bias, and thus, increase generalisability.

In this study, we only evaluated the reliability of questions assessing cannabis smoking, as it is the most common form of use.³⁰ However, given the increase in legalisation laws, other forms of use are becoming more common, including vaping and consumption of edibles. More work is needed to develop reliable questions assessing different forms of cannabis use.

One of the measures we included (‘on those days, how many joints, pipes, or bongs did you smoke per day?’) demonstrated unacceptable test–retest and internal reliability. This may be due to day-to-day differences in the amount of cannabis smoked or differences resulting from different 30-day windows for the 2020 survey and reliability survey.

We did not include any items to validate the measures of cannabis use in the survey. However, the question on frequency of cannabis use in the past 30 days has been validated in other settings by our team. Days of cannabis use in the past month has been associated with stroke and myocardial infarction with more frequent use associated with worse outcomes.³¹

Conclusions

As recreational use of cannabis increases, it is crucial for there to be standardised methods to assess cannabis use in both research and clinical settings. The question ‘on how many of the past 30 days did you smoke marijuana in a joint, pipe or bong?’, as well as the questions assessing quantity of lifetime use (over the entire period you were smoking marijuana, about how many years did you smoke marijuana on a daily or near daily basis?) provided reliable estimates for estimating the frequency of cannabis use. Future work should examine the validity of these simple measures in identifying health risks. Additionally, given that legalisation of cannabis for recreational use has become more widespread, regulation in terms of labelling may allow for more accurate measurements of use and future work should include questions that gather information on quantities such as tetrahydrocannabinol and cannabidiol content and serving size.

Data availability statement

Data are available on reasonable request.