Abstract
Background
Validated Internet-based screening tools for cannabis use and abuse are needed. The present study aimed to establish equivalence between the previously validated Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) as a paper-and-pencil (PaP)-administered questionnaire and its online use.
Methods
Two groups of cannabis users took part in this study and the results were analyzed using structural equation modeling. One group consisted of 150 participants and was assessed with the ASSIST PaP questionnaire in a face-to-face interview (the PaP group). They were recruited from three settings: a primary health care outpatient clinic, a general psychiatric facility, and an ambulatory specialized addiction treatment facility. The other group (the Web group) comprised 1382 persons who answered the online version of the same questionnaire. This sample was drawn from people who naturalistically visited a website dedicated to helping people with cannabis addiction.
Results
The internal consistency was good for the online questionnaire (0.74) and high for the already validated PaP questionnaire (0.91). The Web group, however, had higher scores on cannabis use than did the PaP group. The results show support for configural invariance, meaning that the one-factor structure was preserved across groups, although measurement equivalence between these two survey modes was not achieved. However, when the Web group was split into two random subsamples, measurement invariance was demonstrated between them by cross-validation.
Conclusions
Measurement equivalence was not achieved between the two survey modes. Nonetheless, subanalyses of the Web group demonstrated that the cannabis screening questions of the ASSIST can be used for online screening. Differences in ASSIST scores between samples may be due to the sensitive nature of the information surveyed, with possible underreporting in face-to-face interviews, or to the different characteristics of the Web group because of the specialized nature of the website.
Keywords: The alcohol, Smoking and substance involvement screening test, Cannabis, Psychometrics, Screening test, Internet
Background
The Internet has come to be widely used as a tool for drug treatment and research. For instance, the Web is now commonly used in searches for medical and psychological information [1-3]. Furthermore, different forms of online treatments have been studied for various mental health and addiction-related conditions [4-6]. In consequence, an increased number of medical and psychological studies have been performed by using Web-based questionnaires [7], including topics related to substance use [8].
Internet-based surveys have a number of advantages in comparison with paper-and-pencil (PaP) questionnaires [7,9-11], such as easy data collection and a reduction in data entry errors. The wide dissemination of online surveys may also lead to an increase in recruitment and lower desirability bias compared with traditional methods of data collection. The latter is of particular interest for surveys on stigmatizing topics such as drug use [7]. For example, Web-based, self-administered questionnaires lead to higher reporting rates of substance use in adolescents compared with the rates reported from PaP self-administered questionnaires [12]. Furthermore, the quality of the data provided by Internet-based surveys on topics such as smoking [13] and alcohol use [14] have equal or even better reliability in Web-based questionnaires than in PaP or face-to-face approaches. The Internet, probably because of the perception that it offers anonymity, has also been found to be a useful tool in reaching stigmatized populations, such as illicit drug users and those who use party drugs such as ecstasy or cannabis [15]. The feasibility of using online research for substance use and illegal substance use has been demonstrated in several studies [15].
Despite the wide dissemination of Internet-based surveys, participants in such studies may differ from the general population. In particular, they may be more educated or more likely to be employed [16,17]. Nevertheless, if the possible limitations are acknowledged, Internet-based surveys are suitable for research on topics related to substance use and addictive behaviors, in particular to generate hypotheses that can later be tested in more representative samples [15].
Regardless of the number of opportunities linked to online research, however, some concerns remain regarding the reliability (whereby a given procedure produces the same results in tests repeated with the same empirical or equivalent tools) [18] and validity of the data (the degree to which a given procedure operationalizes the intended concept) [18] obtained via online questionnaires [7]. Studies investigating the invariance of Web-based questionnaires and PaP surveys have demonstrated inconsistent findings. Some studies found no major differences between the two delivery modes [19], whereas others found variations related to particular scales [20] and still others reported differences between online and offline versions of the same tests in terms of score distributions achieved and psychometric properties of the tests [20,21]. The latter findings are particularly true for studies related to self-disclosure of sensitive information [22]. Social desirability may affect answers in face-to-face interviews [23]. For instance, reports of alcohol use and sexually risky behaviors were more likely to be fully disclosed in Web surveys than in traditional formats [24]. Also noteworthy is the possible impact of different designs on the outcomes in these studies. Some researchers used within-subjects designs, whereas others used between-subjects designs in which the online questionnaire or the PaP form was randomly administered, or not administered, to the participants [25].
While an increased number of instruments classically used for mental health assessments have been developed and validated for Internet use [26-29], such developments are needed for the study of addictive disorders, especially for frequently used substances such as cannabis [30-33]. In the last few years, international efforts have been made to develop and validate screening assessment tools related to cannabis use [34]. In particular, after the success of the Alcohol Use Disorder Identification Test (AUDIT) [35], efforts were made to offer similar screening tools for other substance use, including cannabis. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) [36,37] was developed by the World Health Organization to screen for problematic or risky substance use (http://www.who.int/substance_abuse/activities/assist/en/index.html). The ASSIST instrument was validated as a face-to-face questionnaire in different populations and in a number of linguistic versions, in which it was found to have sound psychometric properties. Its internal consistency, assessed with Cronbach’s α, ranged from 0.74 to 0.93. Concurrent validity was demonstrated by significant correlations between ASSIST scores and scores from the Addiction Severity Index [38], AUDIT [35], Revised Fagerstrom Tolerance Questionnaire [39], and Mini International Neuropsychiatric Interview (MINI-Plus) [40] for the diagnosis of substance abuse or dependence. Moreover, the ASSIST yielded interesting sensitivity and specificity values for discrimination between substance use and abuse and between substance abuse and dependence [37,41-44]. In its current version (V3.0), the ASSIST comprises eight questions for each of the following substances: tobacco, alcohol, cannabis, cocaine, amphetamine-type stimulants, inhalants, sedatives, hallucinogens, opiates, and other drugs. Question 1 deals with the lifetime use of each substance. If a substance is used, people are invited to answer additional questions related to this specific substance.
PaP and Web-based surveys may not produce similar results; therefore, measurement equivalence of Web-based and PaP surveys cannot be taken for granted and must be empirically demonstrated [45,46]. To our knowledge, the ASSIST, despite the possibility of its use in naturalistic online settings, has not yet been used in Internet studies and has yet to be validated for online studies.
In the current study, we aimed to do the following:
Describe the distribution of self-reported cannabis use in two samples of cannabis users: visitors to a cannabis prevention website who answered an online questionnaire, and patients of three clinics who answered a PaP questionnaire in a face-to-face interview. Both groups answered the ASSIST version 3.0 questionnaire.
Analyze whether the items in the ASSIST instrument operate equivalently across Internet and PaP groups for gathering sensitive information (i.e., is the measurement model group invariant?).
Methods
Participants and procedures
We obtained two groups of cannabis users. The first group was assessed with the ASSIST PaP questionnaire on a face-to-face basis, and the second group answered an online version of the same questionnaire.
The first group
The first group consisted of the adults who took part in a previous study related to the validation of the French version of the ASSIST questionnaire [44]. It included 150 participants, of whom 50 were patients from a primary health care outpatient clinic, 50 were outpatients from a general psychiatric facility, and 50 were patients from an ambulatory specialized addiction treatment facility. The use of tobacco, alcohol, and illicit substances (cannabis, cocaine, opiates, etc.) was assessed. The three settings were concerned with these substances, except for the primary care setting, where cocaine and opiates were not identified as problematic. Opiate dependence was found in 7.3% of the sample, alcohol abuse or dependence in 43.3%, and cannabis use in 85.3%. Of the latter, 6.3% were considered to have cannabis abuse or dependence. The mean age of the participants was 41 ± 11.5 years. They were mainly men (64%), single (76.7%), and not working (68.5%). All participants treated in these settings were eligible if they were older than 18 years, able to speak in French, and able to give their written informed consent.
The ASSIST (PaP) assessment (in French) was made during an interview with a trained psychologist or psychiatrist who completed the ASSIST during the interview. The ethics committee of the Geneva University Hospitals approved the study.
The second group
The second group was drawn from people who naturalistically visited (without specific advertisement or invitation) a French-language website dedicated to online cannabis help (stop-cannabis.ch). The website offers information and online help for cannabis users. In particular, visitors may access a number of automated interventions such as an online motivational interview, a brief intervention (computer-tailored feedback based on the ASSIST scores), and support for cannabis cessation in a text-messaging format. Although the website is primarily dedicated to people asking for information or help related to cannabis use, it is accessible to anyone, and no specific registration is required to use it. It also includes information for the relatives of cannabis users. The website has been visited by 676,000 Internet users since 2008. No special advertising strategies promote the website: It is mostly accessed via keyword search on general search engines such as Google, as well as from links on other websites such as a sister website dedicated to tobacco smokers. Participants could answer the ASSIST questions related to cannabis use online without any registration procedure and without answering additional socio-demographic questions. After answering the questionnaire (in French), participants received, on a new Web page, a personalized, computer-tailored feedback report based on their answers (one graph and 250 words) that commented on their cannabis use and recommended treatment if necessary. We did not collect any demographic information about participants. Answers emerging from the same IP address were excluded. Participants had the option to accept or decline storage of their answers for survey purposes.
Measurements
After a positive screening result on lifetime substance use of cannabis (ASSIST, Question1), both groups of participants answered the following questions:
Questions 2 to 7 of the ASSIST:
-
Q2:
In the past three months how often have you used cannabis?
-
Q3:
During the past three months how often have you had a strong desire or urge to use cannabis?
-
Q4:
During the past three months how often has your use of cannabis led to health, social, legal or financial problems?
-
Q5:
During the past three months how often have you failed to do what was normally expected of you because of your use of cannabis?
-
Q6:
Has a friend or relative or anyone else ever expressed concern about your use of cannabis?
-
Q7:
Have you ever tried to control, cut down or stop using cannabis?
Questions 2 to 5 are rated on a 5-point Likert scale ranging from “never” (in the past 3 months) to “daily or almost daily”, whereas Questions 6 and 7 use a three-category rating (“no never”, “yes in the past 3 months”, “yes but not in the past 3 months”). The total score, calculated as the sum of scores received for Questions 2 through 7 inclusive, is bound to measure one latent factor: the specific substance involvement score as defined by the ASSIST, here the cannabis involvement score.
This score enabled us to identify three groups of cannabis users:
Low risk (score: 0–3): The participant is at low risk of health and other problems from his/her current pattern of use.
Moderate risk (score: 4–26): The participant is at risk of health and other problems from his/her current pattern of substance use.
High risk (score ≥ 27): The participant is at high risk of experiencing severe problems (health, social, financial, legal, relationship) as a result of his/her current pattern of use and is likely to be dependent.
Statistical analyses
SPSS 18.0 (Statistical Package for the Social Sciences, SPSS Inc., Chicago, IL) and AMOS 19.0 (Analysis of Moment Structures; SPSS Inc., Chicago, IL) software programs were used to perform the statistical analyses. The distribution of self-reported cannabis use was compared by type of survey using χ2 tests.
We next assessed internal consistency of the subscale in the Web survey by Cronbach's alpha coefficient. Ideally, α should be above 0.70, but not much higher than 0.90 [47].
For the multigroup invariance hypothesis, we used the structural equation modeling (SEM) procedure described by Jöreskog [48]. Depending on the research question, searching for group equivalence may imply a series of tests performed in the following restrictive order: configural equivalence, measurement equivalence, and structural equivalence. Configural invariance testing focused on the extent to which the number of factors and the pattern of their structure are similar between groups. In testing for measurement and structural invariance, we focused more specifically on the extent to which parameters in the measurement and structural components of the model are equivalent across groups [49-51]. It is worth noting, however, that the determination of an appropriate baseline model is required for each group separately, from which the configural model is derived. Given that one of our research questions concerns measurement equivalence across groups, statistical analyses focus on configural invariance and measurement invariance.
Evaluation of model fit
A model’s goodness of fit is examined through various indices, as described below [52-54]:
The χ2 to degrees of freedom ratio (χ2/df)
The comparative fit index (CFI)
The root mean square error of approximation (RMSEA)
Changes in goodness-of-fit statistics were also examined to detect differences in the models. A significant difference in χ2 values between nested models means that all equality constraints do not hold across the groups. Since the variables are measured on an ordinal scale with relatively few categories and the values between categories are not equidistant, asymptotically distribution-free estimation instead of maximum likelihood estimation was used here as one of the strategies to accommodate non-normally distributed data.
Sample size considerations
Sample size plays an important role in providing unbiased parameter estimates and accurate model fit information. Bentler and Chou [55] recommended a ratio of at least 5:1 of subjects to variables for normal and elliptical distributions, and there seems to be a general consensus among researchers to adopt this ratio. However, for categorical or non-normally distributed variables, as is the case here, larger samples are required than for continuous or normally distributed variables. A ratio of at least 10 subjects per variable for this type of distribution is recommended [55,56]. The sample in the current study fulfills this requirement.
Results
A sample of 1382 persons participated in the Web study, whereas 150 persons filled in the PaP questionnaire. In the Web survey, 1% of the participants were excluded because of missing data, leaving a sample size of 1366 for the analysis. There were no missing data in the PaP survey.
Cannabis use
There were statistically significant differences between the two groups for all ASSIST questions (Table 1). Participants from the Web survey scored higher on all items.
Table 1.
Self-reported cannabis use by group and questionnaire format (expressed in %), using Questions 2 to 7 of the ASSIST V3.0 questionnaire
| Question | Format | χ 2 | p | |
|---|---|---|---|---|
| Outpatients PaP (n = 150) | Web (n = 1366) | |||
| 2. In the past three months how often have you used cannabis? | 830.2 | <0.0005 | ||
| - Never | 75.3 | 2.5 | ||
| - One or two times | 3.3 | 3.9 | ||
| - Monthly | 5.3 | 4.2 | ||
| - Weekly | 6.0 | 15.4 | ||
| - Daily or almost daily | 10.0 | 74.0 | ||
| 3. During the past three months how often have you had a strong desire or urge to use cannabis? | 381. 9 | <0.0005 | ||
| - Never | 78.0 | 12.3 | ||
| - One or two times | 4.0 | 14.0 | ||
| - Monthly | 2.7 | 5.3 | ||
| - Weekly | 6.7 | 18.4 | ||
| - Daily or almost daily | 8.7 | 50.1 | ||
| 4. During the past three months how often has your use of cannabis led to health, social, legal or financial problems? | 110.9 | <0.0005 | ||
| - Never | 88.6 | 45.0 | ||
| - One or two times | 3.4 | 21.2 | ||
| - Monthly | 2.0 | 13.3 | ||
| - Weekly | 2.7 | 11.8 | ||
| - Daily or almost daily | 3.4 | 8.7 | ||
| 5. During the past three months how often have you failed to do what was normally expected of you because of your use of cannabis? | 168.7 | <0.0005 | ||
| - Never | 96.0 | 40.5 | ||
| - One or two times | 0 | 22.3 | ||
| - Monthly | 0.7 | 10.9 | ||
| - Weekly | 2.7 | 15.6 | ||
| - Daily or almost daily | 0.7 | 10.8 | ||
| 6. Has a friend or relative or anyone else ever expressed concern about your use of cannabis? | 147. 8 | <0.0005 | ||
| - No, never | 60.2 | 35.3 | ||
| - Yes, in the past three months | 9.3 | 17.2 | ||
| - Yes, but not in the past three months | 6.0 | 47.5 | ||
| 7. Have you ever tried to control, cut down or stop using cannabis? | 157.7 | <0.0005 | ||
| - No, never | 88.7 | 35.3 | ||
| - Yes, in the past three months | 5.3 | 25.8 | ||
| - Yes, but not in the past three months | 6.0 | 38.9 | ||
| Cannabis involvement score (sum of responses to Q2-Q7) | 4.2 (8.7) | 22.8 (9.9) | −24.3* | <0.0005 |
| Risk level according to ASSIST v3.0 | 874.3 | <0.0005 | ||
| - Low (0–3) | 77.3 | 3.3 | ||
| - Moderate (4–26) | 18.7 | 54.2 | ||
| - High (≥27) | 4.0 | 42.5 | ||
*Obtained by t-test.
Internal consistency
According to the validation study of the French version of the ASSIST, the internal consistency of the cannabis involvement scale yielded a Cronbach’s alpha coefficient of 0.91 [44]. In the current study, these coefficients were 0.91 and 0.74 for the PaP and the Web surveys, respectively.
Testing for configural equivalence
As noted earlier, a prerequisite to testing for instrument equivalence is to establish a well-fitting model for each group separately, followed by a combined baseline model in which the same parameters are estimated again within the framework of a multigroup model. Results showed no misspecification for the PaP model, whereas possible misspecifications regarding two error terms (Items 2 and 3, 3 and 7) were highlighted for the Web model. It was subsequently respecified and reestimated with these two error covariances included. These baseline models were considered optimal in representing the data for the PaP sample (χ2/df = 1.93; CFI = 0.90; RMSEA = 0.08) and for the Web sample (χ2/df = 1.97; CFI = 0.99; RMSEA = 0.03).
Next, these differentially specified baseline models were incorporated into one file for the purposes of testing cross-group equivalence. Assessment of this model revealed a good fit to the data, as indicated by the CFI (0.99) and RMSEA (0.02). The χ2 value (8.06, df = 6) provides the baseline value against which all subsequent tests for invariance are compared (Table 2).
Table 2.
Summary of goodness-of-fit statistics for tests of invariance across format types (PaP and web format)
| Model description | RMSEA | CFI | χ 2 | df | Δχ 2 | Δdf | Statistical significance |
|---|---|---|---|---|---|---|---|
| 1. Configural model: no constraints imposed | 0.02 | 0.99 | 8.06 | 6 | - | - | - |
| 2. Factor loadings: constrained equally across groups | 0.03 | 0.97 | 45.44 | 19 | 37.38 | 13 | p < 0.0005 |
Testing for measurement equivalence
A model with loadings constrained to be equal across groups had a fit that was significantly poorer than that of the unconstrained model (CFI = 0.97 and RMSEA = 0.03). As can be seen from Table 2, the χ2 difference value between the constrained model and the configural model is statistically significant. This finding suggests that the constrained model is significantly worse than the unconstrained model and argues for nonequivalence of factor loadings across the groups. In other words, the hypothesis that the PaP survey and the Web survey have the same factor means is rejected. The factor loadings by type of format are displayed in Table 3.
Table 3.
Factor loadings by format type
| Question | Pen-and-paper (n = 150) | Web (n = 1366) |
|---|---|---|
| Q2: In the past three months how often have you used cannabis? | 0.92 | 0.51 |
| Q3: During the past three months how often have you had a strong desire or urge to use cannabis? | 0.98 | 0.62 |
| Q4: During the past three months how often has your use of cannabis led to health, social, legal or financial problems? | 0.76 | 0.71 |
| Q5: During the past three months how often have you failed to do what was normally expected of you because of your use of cannabis? | 0.24 | 0.67 |
| Q6: Has a friend or relative or anyone else ever expressed concern about your use of cannabis? | 0.46 | 0.44 |
| Q7: Have you ever tried to control, cut down or stop using cannabis? | 0.22 | 0.50 |
Testing for invariance across two random subsamples of the Web sample
From the results reported above, the equivalence of the online and the offline versions of the ASSIST has not been demonstrated. To check whether the ASSIST could be used in online studies, we randomly split the Web sample into two comparable subgroups from the perspective of the ASSIST risk level (Table 4). We then applied the SEM methodology described earlier to the first subsample and cross-validated the results by using the other subsample in a multigroup analysis.
Table 4.
Self-reported cannabis use by random subsample (expressed in %), using the cannabis-related questions of the ASSIST V3.0 questionnaire
| Question | Web total sample (n = 1366) | Web subsample 1 n = 675 | Web subsample 2 n = 691 | p 1 |
|---|---|---|---|---|
| 2. In the past three months how often have you used cannabis? | 1.0 | |||
| - Never | 2.5 | 2.5 | 2.5 | |
| - One or two times | 3.9 | 3.9 | 3.9 | |
| - Monthly | 4.2 | 4.1 | 4.3 | |
| - Weekly | 15.4 | 16.0 | 14.8 | |
| - Daily or almost daily | 74.0 | 73.5 | 74.5 | |
| 3. During the past three months how often have you had a strong desire or urge to use cannabis? | 0.6 | |||
| - Never | 12.3 | 12.6 | 12.2 | |
| - One or two times | 14.0 | 15.3 | 12.6 | |
| - Monthly | 5.3 | 5.3 | 4.9 | |
| - Weekly | 18.4 | 17.9 | 18.8 | |
| - Daily or almost daily | 50.1 | 48.9 | 51.5 | |
| 4. During the past three months how often has your use of cannabis led to health, social, legal or financial problems? | 0.8 | |||
| - Never | 45.0 | 44.4 | 45.7 | |
| - One or two times | 21.2 | 22.4 | 19.8 | |
| - Monthly | 13.3 | 13.2 | 13.5 | |
| - Weekly | 11.8 | 11.3 | 12.3 | |
| - Daily or almost daily | 8.7 | 8.7 | 8.7 | |
| 5. During the past three months how often have you failed to do what was normally expected of you because of your use of cannabis? | 0.8 | |||
| - Never | 40.5 | 41.5 | 39.4 | |
| - One or two times | 22.3 | 21.0 | 23.7 | |
| - Monthly | 10.9 | 10.8 | 11.1 | |
| - Weekly | 15.6 | 15.7 | 15.2 | |
| - Daily or almost daily | 10.8 | 11.0 | 10.6 | |
| 6. Has a friend or relative or anyone else ever expressed concern about your use of cannabis? | 0.3 | |||
| - No, never | 35.3 | 34.2 | 36.0 | |
| - Yes, in the past three months | 17.2 | 16.1 | 18.2 | |
| - Yes, but not in the past three months | 47.5 | 49.6 | 45.7 | |
| 7. Have you ever tried to control, cut down or stop using cannabis? | 0.7 | |||
| - No, never | 35.3 | 36.0 | 35.2 | |
| - Yes, in the past three months | 25.8 | 24.7 | 26.8 | |
| - Yes, but not in the past three months | 38.9 | 39.3 | 38.1 | |
| Cannabis involvement score (sum of responses to Q2-Q7) | 22.8 (9.9) | 22.8 (9.8) | 22.8 (10.0) | 1.0 |
| Risk level according to ASSIST v3.0 | 0.9 | |||
| - Low (0–3) | 3.3 | 3.1 | 3.6 | |
| - Moderate (4–26) | 54.2 | 53.8 | 53.8 | |
| - High (≥27) | 42.5 | 43.1 | 42.5 |
p-Values resulting from comparison of Subsample 1 and Subsample 2.
The baseline results showed good model fit for Subsample 1 (χ2/df = 1.77; CFI = 0.99; RMSEA = 0.03), as well as for Subsample 2 (χ2/df = 2.06; CFI = 0.99; RMSEA = 0.04). The next step consisted in establishing configural invariance for both groups together without constraint: it also yielded satisfactory results. As can be seen in Table 5, configural invariance (Model 1) is supported, indicating that the one-factor structure in each group is preserved. From Model 2 to Model 4, all successive invariance tests (metric invariance, residual variance invariance, strict factorial invariance) held, given the nonsignificant difference in the χ2 values (Table 5).
Table 5.
Summary of goodness-of-fit statistics for invariance tests across the two random subsamples
| Model description | RMSEA | CFI | χ 2 | df | χ 2 /df | Δχ 2 | Δdf | Statistical significance |
|---|---|---|---|---|---|---|---|---|
| 1. Configural invariance: no constraints imposed | 0.03 | 0.99 | 26.80 | 14 | 1.9 | - | - | - |
| 2. Factor loadings: constrained equally across groups | 0.02 | 0.99 | 32.31 | 19 | 1.7 | 5.51 | 5 | p = 0.4 |
| 3. Factor loadings and variances: constrained equally | 0.02 | 0.99 | 32.33 | 20 | 1.6 | 5.53 | 6 | p = 0.5 |
| 4. Factor loadings, variances, and error covariances: constrained equally | 0.02 | 0.99 | 38.76 | 28 | 1.4 | 11.97 | 14 | p = 0.6 |
Discussion
In this study, the internal consistency and the validity of the ASSIST (for the assessment of cannabis use) were compared in two samples: visitors of a cannabis prevention website who answered an online survey in French, and outpatients of three clinics in Switzerland who answered an ASSIST PaP questionnaire. The internal consistency was high for the PaP survey and good for the Web survey (0.91 and 0.74, respectively). A possible explanation for this difference in value is that Cronbach’s alpha is sensitive to deviations from normality. Therefore, if the data do not meet the assumptions of normality and linearity, as in the Web sample, the reliability may be underestimated by the current formulas used for the calculation of this value [57]. Regarding validity, the results show that the study supports configural invariance, meaning that the one-factor structure is preserved across groups.
Measurement invariance was not supported by the study results, however. Cannabis use was more frequent and problematic for the Web group than it was for the clinic-recruited sample. Only 10% of the PaP sample reported daily use of cannabis in the past 3 months, in comparison to 74% of the Web sample (Question 2). As shown in Table 1, very few PaP participants (4%) admitted to having failed in tasks normally expected from them because of their cannabis use (Question 4), whereas 60% of the Web sample endorsed this item. Consequently, the loading of this item in the factor analysis (Table 3) was very different across groups. The same problem applies to Question 7 (tried to stop or cut down).
These results may be explained, in part, by the absence of a randomization procedure (on the PaP and total Web sample), leading to selection bias, one of the limitations of the study [11,58]. The characteristics of the Web sample (from the naturalistic visitors of a website dedicated to helping people facing cannabis addiction) are probably explained by the involvement (via self-selection) of participants who were more concerned by their cannabis use than would be the case in the general population, as suggested by another study on the self-selection bias associated with online studies [59]. The participants from the stop-cannabis.ch website selected themselves to visit the website and chose to assess their cannabis use with the ASSIST, probably in relation to concerns about their cannabis consumption. The participants from the clinic centers, in contrast, may not have been preoccupied by their cannabis use [44].
The contribution of a desirability bias to the differences observed between the PaP and the Web group should also not be ignored. This bias was previously reported in face-to-face interviews [7]. The phenomenon leads to underreporting of stigmatized behaviors (i.e. cannabis use) in face-to-face settings in comparison to that in computerized self-assessments [60,61]. Furthermore, differences in the understanding of some questions cannot be fully ruled out [7,58].
Study limitations and strengths
This study has several limitations, the most important of which is that we did not include randomly equivalent groups by type of survey. This limitation may have undermined external validity. It further highlights selection [59] and desirability biases [45,62], hence restricting the statistical comparison of the PaP and the Web groups to a descriptive level. Another limitation is a lack of other clinical assessments. Comparison with a diagnostic instrument, for instance, would allow us to calculate the sensitivity and the specificity of the Internet-administered ASSIST, two inextricably linked measures that help clinicians in deciding whether to rule a diagnosis in or out. A third limitation is that detailed socio-demographic characteristics of the Internet sample were not available to examine the profile of the respondents.
These limitations having been acknowledged, the study nonetheless has two major strengths. The first is that, owing to the large Web-sample size, it has been possible to analyze two randomly selected groups whose measurement equivalence results support the view that the ASSIST is a valid instrument in Web surveys.
The second strength that must be highlighted is that, although some studies have been performed on the online assessment of substance use [63] or on Internet-based prevention [64] or treatment [65,66] of cannabis use, our study is, to the best of our knowledge, the first to describe cannabis use of naturalistically self-selected users of a specialized website. This is an important addition, demonstrating the acceptability of such websites in naturalistic settings by cannabis users who are concerned with their behavior.
Conclusions
Despite the limitations, it appears that the cannabis screening questions of the ASSIST instrument are useful in a Web-based format. The instrument could be used for screening and then possibly for online prevention and intervention. Further studies using the ASSIST for the assessment of other addictions, in other populations, or under different conditions while using the same statistical designs (e.g. random within-subjects, between-subjects designs) are warranted to generalize these findings.
Acknowledgements
The authors would like to thank Dr. Barbara Broers for her contribution to the collection of data. They also thank the anonymous reviewers who provided useful feedback upon the first submission of this study, and Barbara Every, ELS, of BioMedical Editor, for language editing of the manuscript.
Footnotes
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
Y. Khazaal designed the study. A. Chatton undertook the statistical analysis. Y. Khazaal, A. Chatton, D. Zullino, J-F Etter, and R. Khan contributed to the writing of the manuscript. D. Zullino and J-F Etter contributed to the editing and review of the final manuscript. G. Monney and A. Nallet contributed to data collection. All authors contributed to and have approved the final manuscript.
Contributor Information
Yasser Khazaal, Email: khazaal@hcuge.ch.
Anne Chatton, Email: chatton@hcuge.ch.
Grégoire Monney, Email: monney@hcuge.ch.
Audrey Nallet, Email: nallet@hcuge.ch.
Riaz Khan, Email: khan@hcuge.ch.
Daniele Zullino, Email: zullino@hcuge.ch.
Jean-François Etter, Email: Etter@unige.ch.
References
- 1.Khazaal Y, Chatton A, Cochand S, Hoch A, Khankarli MB, Khan R, et al. Internet use by patients with psychiatric disorders in search for general and medical informations. Psychiatr Q. 2008;79:301–9. doi: 10.1007/s11126-008-9083-1. [DOI] [PubMed] [Google Scholar]
- 2.Khazaal Y, Chatton A, Cochand S, Coquard O, Fernandez S, Khan R, et al. Brief DISCERN, six questions for the evaluation of evidence-based content of health-related websites. Patient Educ Couns. 2009;77:33–7. doi: 10.1016/j.pec.2009.02.016. [DOI] [PubMed] [Google Scholar]
- 3.Morel V, Chatton A, Cochand S, Zullino D, Khazaal Y. Quality of web-based information on bipolar disorder. J Affect Disord. 2008;110(3):265–9. doi: 10.1016/j.jad.2008.01.007. [DOI] [PubMed] [Google Scholar]
- 4.Andersson G, Bergstrom J, Hollandare F, Carlbring P, Kaldo V, Ekselius L. Internet-based self-help for depression: randomised controlled trial. Br J Psychiatry. 2005;187:456–61. doi: 10.1192/bjp.187.5.456. [DOI] [PubMed] [Google Scholar]
- 5.Etter JF. Internet-based smoking cessation programs. Int J Med Inform. 2006;75(1):110–6. doi: 10.1016/j.ijmedinf.2005.06.014. [DOI] [PubMed] [Google Scholar]
- 6.Schaub M, Sullivan R, Haug S, Stark L. Web-based cognitive behavioral self-help intervention to reduce cocaine consumption in problematic cocaine users: randomized controlled trial. J Med Internet Res. 2012;14(6):e166. doi: 10.2196/jmir.2244. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.van Gelder MM, Bretveld RW, Roeleveld N. Web-based questionnaires: the future in epidemiology? Am J Epidemiol. 2010;172(11):1292–8. doi: 10.1093/aje/kwq291. [DOI] [PubMed] [Google Scholar]
- 8.Becker J, Hungerbuehler I, Berg O, Szamrovicz M, Haubensack A, Kormann A, et al. Development of an integrative cessation program for co-smokers of cigarettes and cannabis: demand analysis, program description, and acceptability. Subst Abuse Treat Prev Policy. 2013;8:33. doi: 10.1186/1747-597X-8-33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Whitehead L. Methodological issues in Internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res. 2011;13(4):e109. doi: 10.2196/jmir.1593. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Ramo DE, Prochaska JJ. Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. J Med Internet Res. 2012;14(1):e28. doi: 10.2196/jmir.1878. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Fenner Y, Garland SM, Moore EE, Jayasinghe Y, Fletcher A, Tabrizi SN, et al. Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res. 2012;14(1):e20. doi: 10.2196/jmir.1978. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Wang YC, Lee CM, Lew-Ting CY, Hsiao CK, Chen DR, Chen WJ. Survey of substance use among high school students in Taipei: web-based questionnaire versus paper-and-pencil questionnaire. J Adolesc Health. 2005;37(4):289–95. doi: 10.1016/j.jadohealth.2005.03.017. [DOI] [PubMed] [Google Scholar]
- 13.Brigham J, Lessov-Schlaggar CN, Javitz HS, Krasnow RE, McElroy M, Swan GE. Test-retest reliability of web-based retrospective self-report of tobacco exposure and risk. J Med Internet Res. 2009;11(3):e35. doi: 10.2196/jmir.1248. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Miller ET, Neal DJ, Roberts LJ, Baer JS, Cressler SO, Metrik J, et al. Test-retest reliability of alcohol measures: is there a difference between internet-based assessment and traditional methods? Psychol Addict Behav. 2002;16(1):56–63. doi: 10.1037/0893-164X.16.1.56. [DOI] [PubMed] [Google Scholar]
- 15.Miller PG, Sonderlund AL. Using the internet to research hidden populations of illicit drug users: a review. Addiction. 2010;105(9):1557–67. doi: 10.1111/j.1360-0443.2010.02992.x. [DOI] [PubMed] [Google Scholar]
- 16.Dickerson S, Reinhart AM, Feeley TH, Bidani R, Rich E, Garg VK, et al. Patient Internet use for health information at three urban primary care clinics. J Am Med Inform Assoc. 2004;11(6):499–504. doi: 10.1197/jamia.M1460. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Smith AB, King M, Butow P, Olver I. A comparison of data quality and practicality of online versus postal questionnaires in a sample of testicular cancer survivors. Psychooncology. 2013;22(1):233–7. doi: 10.1002/pon.2052. [DOI] [PubMed] [Google Scholar]
- 18.Piergiorgio C. Social Research: Theory, Methods and Technique. London: SAGE Publications; 2011. [Google Scholar]
- 19.Eaton DK, Brener ND, Kann L, Denniston MM, McManus T, Kyle TM, et al. Comparison of paper-and-pencil versus Web administration of the Youth Risk Behavior Survey (YRBS): risk behavior prevalence estimates. Eval Rev. 2010;34(2):137–53. doi: 10.1177/0193841X10362491. [DOI] [PubMed] [Google Scholar]
- 20.Vallejo MA, Jordan CM, Diaz MI, Comeche MI, Ortega J. Psychological assessment via the internet: a reliability and validity study of online (vs paper-and-pencil) versions of the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R) J Med Internet Res. 2007;9(1):e2. doi: 10.2196/jmir.9.1.e2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Buchanan T. Internet-based questionnaire assessment: appropriate use in clinical contexts. Cogn Behav Ther. 2003;32(3):100–9. doi: 10.1080/16506070310000957. [DOI] [PubMed] [Google Scholar]
- 22.Tourangeau R, Yan T. Sensitive questions in surveys. Psychol Bull. 2007;133(5):859–83. doi: 10.1037/0033-2909.133.5.859. [DOI] [PubMed] [Google Scholar]
- 23.Kelly D, Harper DJ, Landau B. Questionnaire mode effects in interactive information retrieval experiments. Inf Process Manag. 2008;44(1):122–41. doi: 10.1016/j.ipm.2007.02.007. [DOI] [Google Scholar]
- 24.Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, Sonenstein FL. Adolescent sexual behavior, drug use, and violence: increased reporting with computer survey technology. Science. 1998;280(5365):867–73. doi: 10.1126/science.280.5365.867. [DOI] [PubMed] [Google Scholar]
- 25.De Beuckelaer A, Lievens F. Measurement equivalence of paper-and-pencil and Internet organisational surveys: a large scale examination in 16 countries. Appl Psychol. 2009;58(2):336–61. doi: 10.1111/j.1464-0597.2008.00350.x. [DOI] [Google Scholar]
- 26.Spek V, Nyklicek I, Cuijpers P, Pop V. Internet administration of the Edinburgh Depression Scale. J Affect Disord. 2008;106(3):301–5. doi: 10.1016/j.jad.2007.07.003. [DOI] [PubMed] [Google Scholar]
- 27.Donker T, van Straten A, Marks I, Cuijpers P. Quick and easy self-rating of Generalized Anxiety Disorder: validity of the Dutch web-based GAD-7, GAD-2 and GAD-SI. Psychiatry Res. 2011;188(1):58–64. doi: 10.1016/j.psychres.2011.01.016. [DOI] [PubMed] [Google Scholar]
- 28.Donker T, van Straten A, Marks I, Cuijpers P. Brief self-rated screening for depression on the Internet. J Affect Disord. 2010;122(3):253–9. doi: 10.1016/j.jad.2009.07.013. [DOI] [PubMed] [Google Scholar]
- 29.Lindner P, Andersson G, Ost LG, Carlbring P. Validation of the internet-administered Quality of Life Inventory (QOLI) in different psychiatric conditions. Cogn Behav Ther. 2013;42(4):315–27. doi: 10.1080/16506073.2013.806584. [DOI] [PubMed] [Google Scholar]
- 30.Noack R, Hofler M, Lueken U. Cannabis use patterns and their association with DSM-IV cannabis dependence and gender. Eur Addict Res. 2011;17(6):321–8. doi: 10.1159/000333338. [DOI] [PubMed] [Google Scholar]
- 31.Norberg MM, Gates P, Dillon P, Kavanagh DJ, Manocha R, Copeland J. Screening and managing cannabis use: comparing GP's and nurses’ knowledge, beliefs, and behavior. Subst Abuse Treat Prev Policy. 2012;7:31. doi: 10.1186/1747-597X-7-31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Rooke SE, Gates PJ, Norberg MM, Copeland J. Applying technology to the treatment of cannabis use disorder: comparing telephone versus Internet delivery using data from two completed trials. J Subst Abuse Treat. 2014;46(1):78–84. doi: 10.1016/j.jsat.2013.08.007. [DOI] [PubMed] [Google Scholar]
- 33.Fernandez-Artamendi S, Fernandez-Hermida JR, Muniz-Fernandez J, Secades-Villa R, Garcia-Fernandez G. Screening of cannabis-related problems among youth: the CPQ-A-S and CAST questionnaires. Subst Abuse Treat Prev Policy. 2012;7(1):13. doi: 10.1186/1747-597X-7-13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Legleye S, Kraus L, Piontek D, Phan O, Jouanne C. Validation of the Cannabis Abuse Screening Test in a sample of cannabis inpatients. Eur Addict Res. 2012;18(4):193–200. doi: 10.1159/000336553. [DOI] [PubMed] [Google Scholar]
- 35.Bohn MJ, Babor TF, Kranzler HR. The Alcohol Use Disorders Identification Test (AUDIT): validation of a screening instrument for use in medical settings. J Stud Alcohol. 1995;56(4):423–32. doi: 10.15288/jsa.1995.56.423. [DOI] [PubMed] [Google Scholar]
- 36.Group WAW. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): development, reliability and feasibility. Addiction. 2002;97(9):1183–94. doi: 10.1046/j.1360-0443.2002.00185.x. [DOI] [PubMed] [Google Scholar]
- 37.Newcombe DA, Humeniuk RE, Ali R. Validation of the World Health Organization Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): report of results from the Australian site. Drug Alcohol Rev. 2005;24(3):217–26. doi: 10.1080/09595230500170266. [DOI] [PubMed] [Google Scholar]
- 38.Kosten TR, Rounsaville BJ, Kleber HD. Concurrent validity of the addiction severity index. J Nerv Ment Dis. 1983;171(10):606–10. doi: 10.1097/00005053-198310000-00003. [DOI] [PubMed] [Google Scholar]
- 39.Heatherton TF, Kozlowski LT, Frecker RC, Fagerstrom KO. The Fagerstrom Test for Nicotine Dependence: a revision of the Fagerstrom Tolerance Questionnaire. Br J Addict. 1991;86(9):1119–27. doi: 10.1111/j.1360-0443.1991.tb01879.x. [DOI] [PubMed] [Google Scholar]
- 40.Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59:Suppl 20:22–33–quiz 34–57. [PubMed] [Google Scholar]
- 41.Humeniuk R, Ali R, Babor TF, Farrell M, Formigoni ML, Jittiwutikarn J, et al. Validation of the Alcohol, Smoking And Substance Involvement Screening Test (ASSIST) Addiction. 2008;103(6):1039–47. doi: 10.1111/j.1360-0443.2007.02114.x. [DOI] [PubMed] [Google Scholar]
- 42.Hides L, Cotton SM, Berger G, Gleeson J, O'Donnell C, Proffitt T, et al. The reliability and validity of the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) in first-episode psychosis. Addict Behav. 2009;34(10):821–5. doi: 10.1016/j.addbeh.2009.03.001. [DOI] [PubMed] [Google Scholar]
- 43.Khan R, Chatton A, Thorens G, Achab S, Nallet A, Broers B, et al. Validation of the French version of the alcohol, smoking and substance involvement screening test (ASSIST) in the elderly. Subst Abuse Treat Prev Policy. 2012;7(1):14. doi: 10.1186/1747-597X-7-14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Khan R, Chatton A, Nallet A, Broers B, Thorens G, Achab-Arigo S, et al. Validation of the French version of the alcohol, smoking and substance involvement screening test (ASSIST) Eur Addict Res. 2011;17(4):190–7. doi: 10.1159/000326073. [DOI] [PubMed] [Google Scholar]
- 45.Buchanan T, Ali T, Heffernan TM, Ling J, Parrott AC, Rodgers J, et al. Nonequivalence of on-line and paper-and-pencil psychological tests: the case of the prospective memory questionnaire. Behav Res Methods. 2005;37(1):148–54. doi: 10.3758/BF03206409. [DOI] [PubMed] [Google Scholar]
- 46.Cole MS, Bedeian AG, Feild HS. The measurement equivalence of Web-based and paper-and-pencil measures of transformational leadership: a multinational test. Organ Res Methods. 2006;9(3):339–68. doi: 10.1177/1094428106287434. [DOI] [Google Scholar]
- 47.Streiner DL, Norman GR. Health measurement scales. 4. New York: Oxford University Press; 2008. [Google Scholar]
- 48.Joreskog KG. Simultaneous factor analysis in several populations. Psychometrika. 1971;36:409–26. doi: 10.1007/BF02291366. [DOI] [Google Scholar]
- 49.Hoyle RH. Handbook of structural equation modeling. New York: The Guilford Press; 2012. [Google Scholar]
- 50.Laursen BP, Little TD, Card NA. Handbook of developmental research methods. New York: The Guilford Press; 2012. [Google Scholar]
- 51.Byrne BM. Structural equation modeling with AMOS. 2. New York: Routledge; 2009. [Google Scholar]
- 52.Byrne BM. A primer of LISREL: basic applications and programming for confirmatory factor analytic models. New York: Springer; 1989. [Google Scholar]
- 53.Browne M, Cudeck R. Testing structural equation models. Edited by Bollen KA, Long JS. London, Sage Publications; 1993: 111–136.
- 54.Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling. 1999;6:1–55. doi: 10.1080/10705519909540118. [DOI] [Google Scholar]
- 55.Bentler PM, Chou C-P. Practical issues in structural modeling. Sociol Methods Res. 1987;16:78–117. doi: 10.1177/0049124187016001004. [DOI] [Google Scholar]
- 56.Kline R. Principles and practice of structural equation modeling. 3. New York: The Guilford Press; 2011. [Google Scholar]
- 57.Helms JE, Henze KT, Saas TL, Mifsud VA. Treating Cronbach’s alpha reliability coefficients as data in counseling research. Couns Psychol. 2006;34(5):630–60. doi: 10.1177/0011000006288308. [DOI] [Google Scholar]
- 58.Ramo DE, Prochaska JJ. Prevalence and co-use of marijuana among young adult cigarette smokers: an anonymous online national survey. Addict Sci Clin Pract. 2012;7(1):5. doi: 10.1186/1940-0640-7-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Khazaal Y, van Singer M, Chatton A, Achab S, Zullino D, Rothen S, et al. Does self-selection affect samples’ representativeness in online surveys? An investigation in online video game research. J Med Internet Res. 2014;16(7):e164. doi: 10.2196/jmir.2759. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Buchanan T, Smith JL. Research on the Internet: validation of a World-Wide Web mediated personality scale. Behav Res Methods Instrum Comput. 1999;31(4):565–71. doi: 10.3758/BF03200736. [DOI] [PubMed] [Google Scholar]
- 61.Richman WL, Kiesler S, Weisband S, Drasgow F. A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. J Appl Psychol. 1999;84:754–75. doi: 10.1037/0021-9010.84.5.754. [DOI] [Google Scholar]
- 62.Kays K, Gathercoal K, Buhrow W. Does survey format influence self-disclosure on sensitive question items? Comput Human Behav. 2012;28(1):251–6. doi: 10.1016/j.chb.2011.09.007. [DOI] [Google Scholar]
- 63.Duncan DF, White JB, Nicholson T. Using Internet-based surveys to reach hidden populations: case of nonabusive illicit drug users. Am J Health Behav. 2003;27(3):208–18. doi: 10.5993/AJHB.27.3.2. [DOI] [PubMed] [Google Scholar]
- 64.Newton NC, Teesson M, Vogl LE, Andrews G. Internet-based prevention for alcohol and cannabis use: final results of the Climate Schools course. Addiction. 2010;105(4):749–59. doi: 10.1111/j.1360-0443.2009.02853.x. [DOI] [PubMed] [Google Scholar]
- 65.Hoch E, Buhringer G, Pixa A, Dittmer K, Henker J, Seifert A, et al. CANDIS treatment program for cannabis use disorders: findings from a randomized multi-site translational trial. Drug Alcohol Depend. 2014;134:185–93. doi: 10.1016/j.drugalcdep.2013.09.028. [DOI] [PubMed] [Google Scholar]
- 66.Schaub MP, Haug S, Wenger A, Berg O, Sullivan R, Beck T, et al. Can reduce–the effects of chat-counseling and web-based self-help, web-based self-help alone and a waiting list control program on cannabis use in problematic cannabis users: a randomized controlled trial. BMC Psychiatry. 2013;13:305. doi: 10.1186/1471-244X-13-305. [DOI] [PMC free article] [PubMed] [Google Scholar]