Abstract
Background
Craving is a clinically important phenotype for the development and maintenance of nicotine addiction. Virtual reality (VR) paradigms are successful in eliciting cue-induced subjective craving and may even elicit stronger craving than traditional picture-cue methods. However, few studies have leveraged the advances of this technology to improve the assessment of craving.
Objective
This report details the development of a novel, translatable VR paradigm designed to both elicit nicotine craving and assess multiple eye-related characteristics as potential objective correlates of craving.
Methods
A VR paradigm was developed, which includes three Active scenes with nicotine and tobacco product (NTP) cues present, and three Neutral scenes devoid of NTP cues. A pilot sample (N=31) of NTP users underwent the paradigm and completed subjective measures of nicotine craving, sense of presence in the VR paradigm, and VR-related sickness. Eye-gaze fixation time (“attentional bias”) and pupil diameter toward Active versus Neutral cues, as well as spontaneous blink rate during the Active and Neutral scenes, were recorded.
Results
The NTP Cue VR paradigm was found to elicit a moderate sense of presence (mean Igroup Presence Questionnaire score 60.05, SD 9.66) and low VR-related sickness (mean Virtual Reality Sickness Questionnaire score 16.25, SD 13.94). Scene-specific effects on attentional bias and pupil diameter were observed, with two of the three Active scenes eliciting greater NTP versus control cue attentional bias and pupil diameter (Cohen d=0.30-0.92). The spontaneous blink rate metrics did not differ across Active and Neutral scenes.
Conclusions
This report outlines the development of the NTP Cue VR paradigm. Our results support the potential of this paradigm as an effective laboratory-based cue-exposure task and provide early evidence of the utility of attentional bias and pupillometry, as measured during VR, as useful markers for nicotine addiction.
Keywords: nicotine, craving, cue-exposure, virtual reality, attentional bias, pupillometry, development, smoking, addiction, eye-tracking
Introduction
Craving for substances is considered essential for understanding the pathogenesis and maintenance of addiction, as highlighted by the incentive salience model [1,2] and for the inclusion of craving as a criterion for substance use disorder in the Diagnostic and Statistical Manual of Mental Disorders (5th edition; DSM-5) [3] and the International Classification of Diseases (10th edition; ICD-10) [4]. Nicotine craving specifically has been shown to predict lapse to cigarette smoking following cessation [5,6] and is frequently identified by individuals as an important barrier to quitting and maintaining abstinence [7]. Thus, craving represents a clinically important phenotype of nicotine addiction [8] with great potential for intervention.
Accurate assessment of craving is essential for the identification, management, and treatment of nicotine and tobacco product (NTP) use and the use of other substances [9,10]. In human laboratory studies, craving for nicotine and other abused substances is commonly measured using the cue-exposure paradigm. The translational value of the cue-exposure paradigm to the naturalistic environment is predicated on the observation that relapse to drug use is often precipitated by exposure to drug-related cues that provoke craving [11,12]. However, naturalistic cues can be very complex and involve a number of contextual factors that are difficult to replicate in laboratory-based cue-exposure paradigms [13], limiting their ability to invoke a true craving state [9,14]. New technologies such as virtual reality (VR) afford the opportunity to increase the ecological validity of cue-exposure paradigms through the implementation of interactive and immersive presentations of cues within the typical context of use (eg, the presence of others within a setting where the substance is commonly taken), greatly enhancing our ability to invoke craving in the laboratory [9]. Studies using VR cue-exposure have found great support for its effectiveness in inducing subjective, and in some cases objective, craving for tobacco [15-17], as well as alcohol [13,18], cannabis [19], and methamphetamine [20].
Furthermore, despite decades of research, the field of addiction has yet to establish reliable, objective measures of craving. A number of objective correlates of craving have been investigated, including psychophysiological (eg, heart rate variability and skin conductance) and neurological (eg, functional magnetic resonance imaging and blood oxygenation level dependent activation) measures with varying success [14,21]. Attentional bias, or the ability of drug cues to capture the attention of the user, can be conceptualized as a behavioral marker of incentive salience [22] and represents an objectively measurable and clinically important phenomenon for the study of addiction. Attentional bias toward smoking cues has been previously demonstrated among regular tobacco smokers [23-26], and importantly, it has been related to the risk of subsequent relapse following smoking cessation [27].
Multiple theoretical models suggest that cue-induced subjective craving and attentional bias reflect closely linked underlying processes [1,28,29]. Not surprisingly, measures of attentional bias have been shown to correlate with subjective craving [30]. However, the method of assessment appears to be key—direct measures of attention such as the assessment of eye movement, exhibit larger craving correlations [30] and greater reliability [31-34] than indirect measures such as reaction time. Assessment within naturalistic settings has also independently improved the reliability [35] and validity [36] of attentional bias measurement; yet, the naturalistic constraints of these methods prohibit advanced clinical application of these paradigms. New technological advances in VR implementation allow for the assessment of eye movement in a noninvasive and cost-effective manner and demonstrate early success in distinguishing smokers and nonsmokers on the basis of eye fixations to smoking cues in a virtual world [26].
Spontaneous eye blink rate (EBR) represents another, much less studied, potential objective correlate of cue-induced craving. EBR has been closely linked with striatal dopaminergic function and has been advanced as a reliable [37], more cost-effective, and minimally invasive alternative to positron emission tomography (PET) to assess dopaminergic functioning [38]. Dopamine release in the basal ganglia (including the striatum) inhibits the spinal trigeminal complex, leading to increased EBRs, as demonstrated in both rat and human trials [39]. In line with this theory, preclinical research has shown that direct dopaminergic agonists and antagonists increase [40] and decrease EBRs [39-41], respectively. Furthermore, a PET study in monkeys found a strong positive correlation between EBRs and dopamine (D2) or D2-like (D3) receptor availability in the striatum [42]. Given the observed modulation of striatal dopamine during cue-elicited substance craving [43,44], it may be possible to detect NTP cue-induced dopamine changes through EBR measurement. Nonetheless, no studies to date have investigated this hypothesis.
Lastly, pupillometry represents an additional potential objective craving correlate. Pupil dilation is an indirect measure of norepinephrine (NE) release from the locus coeruleus and is associated with reward processing [45], including sensitivity to rewards [46], and engagement of cognitive resources [47]. Pupillary responses also seem to index changes in the allocation of attention and have been advanced as an ideal measure for related constructs that may not pass the threshold for overt behavior or conscious appraisal [48]. To our knowledge, only one study has investigated pupillometry as a measure of response to substance cue-exposure. Kvamme et al [49] found that pupillary bias toward alcohol versus neutral cues, but not subjective craving reports, predicted relapse to alcohol use in a sample of detoxified patients with alcohol dependence [49], suggesting that cue-induced changes in pupillometry may ultimately serve as a useful biomarker for addiction research and clinical care.
This study was intended to outline the methods underlying the development of a novel VR-NTP cue-exposure paradigm with embedded eye-characteristic assessments. Preliminary analyses on a pilot sample of participants are also provided as a proof of concept for the potential utility of this paradigm for the induction of subjective craving in the laboratory, assessment of potential biomarkers of craving (ie, attentional bias, EBR, and pupillary dilation), and prediction of NTP use behaviors.
Methods
NTP Cue VR Paradigm Development
The NTP Cue VR paradigm uses a virtual reality environment built using Unity. The HTC Vive Pro Eye VR headset (HTC) was used to enable VR capabilities and collect eye-related data. HTC’s SRanipal SDK [50] was used in conjunction with Tobii’s (Tobii Technology) Tobii XR SDK [51] to provide access to various data from the eye tracker. Specifically, Tobii XR SDK handled object selections, determining what participants were looking at, with its Gaze-to-Object Mapping (G2OM) algorithm, while the rest of the data were retrieved from the SRanipal SDK. The participants were free to move around (via teleportation) and interact with various objects within the VR environment using 2 hand-held Vive controllers. Surveys (a visual analogue scale [VAS] with a range of 0-100) assessing depressed mood and anxiety were presented at the start of the paradigm (following the initial training and test scenes) and additional surveys assessing subjective craving (“How much are you craving nicotine right now?”) and scene relevance (“How relevant was that scene to your own life?”) were presented between scenes within the headset. A VAS survey was chosen as the in-task measurement of subjective craving owing to its high face-validity, ability to capture the dynamic fluctuations in craving [52], and low burden on participants, especially over frequently repeated assessment. Survey responses were made by adjusting a slide bar using one of the controllers. Participants were instructed to “Just explore everything around you until the scene changes” and “During the task, we will be measuring what you pay attention to, and we will be asking you to rate your craving level between each scene.”
Three Active scenes (Driving, Patio, and Outdoor BBQ) and three Neutral scenes (Bus, Waiting Room, and Library) were developed and included in the final paradigm (see Figure 1 for screenshots of the scenes). The Active scenes include NTP-related cues, while in the Neutral scenes, all cues are neutral. Active cues include ashtrays, lighters, JUUL devices, cigarettes (individual and packs), Puffbars, hookahs, as well as the presence of human models engaged in smoking or vaping behaviors. Neutral cues (eg, water bottles, cellphones, pens or pencils, magazines, and candies) vary depending on the scene context. All cues are interactable such that the participants are able to pick up, throw, and collide the items with other items in the scene. All scenes (Active and Neutral) include the presence of at least one animated human model. Smoke and vapor effects are incorporated with the animated human models in the Active scenes to increase the immersiveness of the experience. All scenes include background music and audio effects consistent with the scene and the participants’ interaction.
NTP Cue VR Paradigm Procedure
The NTP Cue VR paradigm begins with 3 “test scenes,” which are approximately 3 minutes in duration, depending on participant comfort and abilities with the VR hardware. The first scene is the Practice Room. This is a square room with cubes systematically placed around corners of the room. The participants are asked to gaze at each of the boxes to confirm that the eye-tracking is functioning as intended. Then, the participants are asked to practice using the controllers to teleport to 4 different locations in the room. The second scene is the Practice Slider room, which instructs the participants how to answer the survey questions and provides the opportunity to practice adjusting the slider to answer the scales. The third test scene is the Blink Calibration room. In this scene, the participants are asked to blink 5 times after being prompted by an audio signal. The purpose of this room is to collect pupil diameter data when the participants actively blink to assist with increasing the accuracy of blink detection algorithms. Following the completion of the initial test scenes, the 2 mood surveys are presented, and the 6 scenes (3 Active and 3 Neutral) are pseudorandomized within scene type such that the general scene order is maintained (Active, Neutral, Active, Neutral, Active, and Neutral). The participants are then placed in each scene for 5 minutes. The entire paradigm is approximately 30 minutes in duration.
Data Collection
There are 2 types of data recorded within each scene, regular time series and event-based data that is recorded at event onset. Regular time series data are collected at every 10-millisecond interval (100 Hz), independent of the frame time. The following data are recorded periodically: (1) timestamp, (2) raw gaze intersection point, (3) position and forward direction of the participants' headset, and (4) pupil diameter and eye openness (calculated by SRanipal SDK). The following events and corresponding timestamps are recorded when they occur: (1) blinks, including number of blinks and the object of gaze at the time of the blink; (2) button presses on the controller, including time, button pressed, and object of interaction (if applicable); and (3) object of gaze when eye gaze switches to a new object.
Gaze Statistics Calculation
Raycasting from the eye position was initially used to enable object selection in the direction of gaze. However, this raycasting method did not perform well in our experiments, especially for very small objects, owing to the limited precision and accuracy of the eye tracker, microsaccades, etc. Therefore, for small objects of interest, we utilized the G2OM algorithm provided by the Tobii XR SDK, which is a machine learning–based object selection algorithm that aims to improve small object– and fast-moving object–tracking. Based on our testing, this algorithm improved object selection over the naïve method but still lacked selection quality. Thus, to further improve object selection, we introduced an additional mechanism to “lock” the object selection when an object is manipulated such that whenever a participant actively picks up a virtual object, the object selection algorithm will always select the picked object until the participant releases the object. If the participant is not interacting with an object, the G2OM algorithm is employed, or if no small objects are within the field, naïve raycasting is employed.
To calculate eye-gaze statistics toward active and neutral cue objects, 4 dictionaries corresponding to 4 different types of objects (Active, Neutral, Miscellaneous, and Background) are initialized prior to the start of participant involvement in the paradigm. These dictionaries are then used to store the cumulating gaze fixation or dwell time durations as values for individual objects belonging to each object and type. When a participant gazes at an object, the object is searched in the dictionary on the basis of its name and type. If the object was encountered before, the current fixation time is added to its cumulative fixation time. If the object had not been encountered before, a new entry is created for the object. The fixation time is then calculated as the difference between the timestamp of current entry and that of the next line of entry.
Following the completion of the paradigm, total fixation time indices are produced, which reflect the sum of values within each dictionary (Active, Neutral, Miscellaneous, and Background). The mean fixation time indices are also created, which reflect the total fixation time divided by the number of objects (number of keys) gazed at by the participant.
Blink Detection
Initially, we tested a measurement of eye openness, as calculated by the HTC SRanipal SDK, as an indicator for blink detection. However, given the lack of established thresholds of eye openness for blink detection, we instead chose to rely on estimates of pupil diameter. Consistent with previous studies, an eyeblink is herein defined as complete eyelid closure with the pupil covered for 50-500 milliseconds [53,54]. For any given timepoint, we consider a missing pupil diameter reading as a possible complete eyelid closure where the pupil is completely covered by the eyelid. These eye closure durations are blink candidates. If either pupil is covered for less than 50 milliseconds, the candidate is discarded as it is more likely owing to noise or an eye tracker limitation. If either pupil is covered for more than 500 milliseconds, the candidate is also discarded as this is more consistent with a microsleep [54,55]. Using this blink detection definition, the blink count for the majority of the current participants fell within 12-40 blinks per minute, which appears to align with the consensus of spontaneous blink rates in the literature [55-58].
Participant Recruitment and Screening Procedures
Participants for this ongoing study are recruited through flyers and web-based (eg, Facebook, Craigslist, and San Diego Reader) advertisements posted in the San Diego community. Interested individuals call the laboratory and complete a telephone-screening interview to determine initial eligibility. Inclusion criteria for the ongoing study are the following: (1) age >18 years, (2) nondaily (average use on 4-27 days per month in the past 3 months) or daily NTP use (average use on 7 days per week in the past 3 months), and (3) an NTP use history of ≥1 year. Exclusionary criteria are the following: (1) medical or psychiatric history affecting brain development (ie, history or treatment of neurologic disorders, severe head trauma with loss of consciousness for >2 minutes, or current severe DSM-5 psychiatric disorders other than tobacco use disorders), (2) nonfluency in English, (3) visual problems that may make task completion difficult (eg, severe motion sickness, blindness, and glasses).
Eligible participants are then invited for the in-person laboratory assessment and instructed to bring their NTP products with them for use immediately after the assessment to control for effects related to expectations of imminent substance availability [59]. They are asked to abstain from cannabis and alcohol use for at least 24 hours, and from NTP use for at least 1 hour, prior to testing.
Ethical Considerations
Upon arrival to the laboratory, participants receive a full explanation of the study procedures and provide written, informed consent. The study protocol was approved by the University of California, San Diego Human Protections Program institutional review board (protocol 180719) and is in accordance with the Helsinki Declaration of 1975, as revised in 2000.
Psychological Measures
Following consent procedures, participants undergo an extensive clinical interview and complete several self-report questionnaires covering demographic, psychological health (Mini International Neuropsychiatric Interview [MINI] for DSM-5 [60]), and substance use (90-day Timeline Follow-Back [TLFB] [61], PATH Tobacco Dependence [TD] [62], Customary Drinking and Drug Use Record [CDDR] [63], and Tobacco Craving Questionnaire-Short-Form [TCQ-SF] [64]) domains. The TLFB has high test-retest reliability for intervals ranging from 30 to 360 days prior to the interview date, with an intraclass correlation coefficient=0.92 for “Total number of cigarettes smoked per interval” [65]. Thus, past 90-day NTP use episode count from the TLFB was used in the quantitative analyses presented below. All study interview and self-report data were collected and managed using REDCap electronic data capture tools hosted at the University of California, San Diego.
Participants then undergo the NTP Cue VR paradigm, which includes repeated (postscene) assessments of subjective nicotine craving and scene relevance to the individual participant (VAS; see NTP Cue VR Paradigm Development). Upon completion of the paradigm, additional assessments on VR-related outcomes such as VR presence (Igroup Presence Questionnaire [IPQ] [66]) and VR-related simulator or motion sickness (Simulator Sickness Questionnaire [SSQ] [67]) are administered. The IPQ total score was calculated using a simple averaging method to obtain a single average perceived presence score ranging 0-100. Similarly, the SSQ was scored in concordance with procedures outlined to assess VR-specific sickness (Virtual Reality Sickness Questionnaire [VRSQ] [68]), which involves a simple averaging method to obtain a single average sickness score with a range of 0-100.
Statistical Analysis of Pilot Data
These analyses include the first 31 participants to complete the study protocol; however, data were missing for some subjects on a subset of indices owing to technological difficulties (as indicated by the degrees of freedom for each test presented in the results section). Owing to safety restrictions related to COVID-19, no biological verification of abstinence was conducted. Group differences are not being investigated in the present pilot analyses since the goal of this study is to describe the development and general validity of the paradigm and to maximize statistical power. Statistical analyses were conducted using a repeated measures (ie, paired samples) t test (2-tailed) or Pearson correlation framework. The threshold of significance was set at P<.05 for all analyses. SPSS Statistics for Windows (version 27; IBM Corp) software was used for all analyses.
Results
Results Overview
Demographic information is presented in Table 1. In general, the sample is predominantly male (61%) and White (61%), and 61% had no or very limited (one time) previous experience with VR.
Table 1.
Variable | Value | ||
Age (years), mean (SD) | 30.77 (16.33) | ||
Sex (male), % | 61.3 | ||
Ethnicity (White), % | 61.3 | ||
Education (college level), % | 74.2 | ||
Previous experience with virtual reality, n | |||
|
Never | 12 | |
|
Once | 7 | |
|
A few times | 9 | |
|
Many times | 3 | |
Nicotine and tobacco product use days (in the past 90 days), mean (SD) | 60.10 (33.39) | ||
Nicotine and tobacco product use episodes (in the past 90 days), mean (SD) | 772.29 (1008.20) | ||
Tobacco Craving Questionnaire score at baseline, mean (SD) | 105.32 (9.46) | ||
VRa presence (Igroup Presence Questionnaire score), mean (SD) | |||
|
Spatial presence | 65.74 (16.46) | |
|
Involvement | 62.08 (21.59) | |
|
Experienced realism | 52.22 (23.87) | |
|
Total | 60.05 (9.66) | |
VR-related sickness (Virtual Reality Sickness Questionnaire score), mean (SD) | |||
|
Oculomotor | 16.94 (14.60) | |
|
Disorientation | 15.56 (15.43) | |
|
Total | 16.25 (13.94) |
aVR: virtual reality.
Subjective Craving
The paired samples t test, which investigated subjective craving during the paradigm, revealed a significant effect of scene condition on craving (t30=4.24, P<.001; Cohen d=0.76, 95% CI 0.36-1.16), with Active scenes (mean 42.77, SD 34.07) eliciting greater subjective craving than Neutral scenes (mean 29.42, SD 25.54; Figure 2). Pairwise comparisons tested among all Active scenes revealed that craving ratings were greater after the Driving scene (mean 48.77, SD 35.67) than after the Outdoor BBQ scene (mean 43.50, SD 35.79; P=.01) and the Patio scene (mean 40.00, SD 33.69; P=.01); yet, no difference in ratings was observed between the Outdoor BBQ and Patio scenes (P=.33).
Attentional Bias
The paired samples t test, which investigated eye-gaze fixation time during the paradigm, revealed a significant effect of cue-type on fixation time during the Active scenes (t30=–4.76, P<.001; Cohen d=–0.85, 95% CI –1.26 to –0.44), with greater mean fixation time toward Neutral cues (mean 12,888.52, SD 6314.20 milliseconds) compared to Active cues (mean 5807.98, SD 3002.78 milliseconds). Additional t tests within each Active scene (see Figure 3) revealed a greater Active (mean 4364.32, SD 2541.85 milliseconds) versus Neutral (mean 1962.32, SD 812.64 milliseconds) cue fixation time in the Patio scene (t29=5.05, P<.001; Cohen d=0.92, 95% CI 0.49-1.35), and a greater Active (mean 3060.69, SD 2183.26 milliseconds) versus Neutral (mean 2111.13, SD 972.85 milliseconds) cue fixation time in the Outdoor BBQ scene (t29=2.24, P=.03; Cohen d=0.41, 95% CI 0.03-0.78). However, we observed a lower Active (mean 10,238.44, SD 6037.01 milliseconds) versus Neutral (mean 34,723.50, SD 19,114.72 milliseconds) cue fixation time in the Driving scene (t30=–5.83, P<.001; Cohen d=–1.05, 95% CI –1.48 to –0.60).
Pupil Diameter
The paired samples t test, which compared mean pupil diameters, revealed a smaller pupil diameter in response to Active cues (mean 3.87, SD 0.78 mm) than for Neutral cues (mean 3.97, SD 0.71 mm; t28=–2.01, P=.05; Cohen d=–0.37, 95% CI –0.75 to 0.01) averaged across Active scenes. Additional t tests within each Active scene (see Figure 4) revealed a greater Active (mean 3.95, SD 0.63 mm) versus Neutral (mean 3.83, SD 812.64 mm) cue pupil diameter in the Patio scene (t27=3.95, P<.001; Cohen d=0.75, 95% CI 0.32-1.16), and a trend for greater Active (mean 3.78, SD 0.61 mm) versus Neutral (mean 3.71, SD 0.59 mm) cue pupil diameter in the Outdoor BBQ scene (t27=1.60, P=.12; Cohen d=0.30, 95% CI –0.08 to 0.68). As observed for attentional bias, a lower Active (mean 4.16, SD 1.13 mm) versus Neutral (mean 4.43, SD 0.80 mm) cue pupil diameter was observed in the Driving scene (t28=–2.07, P=.05; Cohen d=–0.38, 95% CI –0.76 to –0.003).
Spontaneous Eye-Blink (EBR)
The paired samples t test revealed no significant differences in EBR during Active and Neutral scenes (t30=0.49, P=0.62; Cohen d=0.09, 95% CI –0.26 to 0.44). Within Active scenes only, pairwise comparisons revealed that the Outdoor BBQ scene (mean 197.86, SD 96.80) was associated with a greater EBR than the Patio scene (mean 173.83, SD 76.90; P=.04; Cohen d=0.41, 95% CI 0.02-0.78). No differences were observed between the Driving scene (mean 193.47, SD 85.98) and the Outdoor BBQ (P=.53; Cohen d=0.12, 95% CI –0.24 to 0.47) or Patio scene (P=.21; Cohen d=0.24, 95% CI –0.13 to 0.60).
Relationship to NTP Subjective Craving and Use
Exploratory Pearson correlations were investigated to provide an initial estimate of the potential for these objective metrics to serve as an indicator of subjective craving and past NTP use. Attentional bias (mean Active vs Neutral Cue fixation time across Active scenes), pupil diameter, and EBR were not found to significantly correlate with in-task subjective craving ratings (attentional bias: rDriving=–0.09, rPatio=0.16, rOutdoor BBQ=0.26, P>.05 for all; pupil diameter: rDriving=0.16, rPatio=0.01, rOutdoor BBQ=0.06, P>.05 for all; EBR: rDriving=0.12, rPatio=0.28, rOutdoor BBQ=0.19, P>.05); however, attentional bias was found to positively correlate with past 90-day NTP use episodes at a trend level (r=0.33, P=.06; see Figure 5). Similar positive correlations were observed for each scene separately (r=0.19-0.31). No relationships were observed between past 90-day NTP use and pupil diameter (r=–0.27, P=.17) or EBR (r=–0.05, P=.79).
Discussion
Principal Findings
This report describes our approach to the development of a novel NTP cue VR paradigm designed to simultaneously induce and assess potential eye-based objective correlates of nicotine craving in naturalistic and translatable virtual settings. The preliminary statistical analyses support the potential of this paradigm in its ability to induce subjective craving while instilling a moderate sense of presence in the virtual world and only low levels of VR-related sickness.
The preliminary results outline a potential context-specific effect of NTP-related attentional bias and pupil dilation in this pilot sample. Consistent with the literature on attentional bias [23-26] and pupil dilation [49], we observed greater Active NTP versus Neutral control cue-related effects in 2 of the 3 Active scenes (Patio and Outdoor BBQ). The similarity observed in the pattern of effects between attentional bias and pupil dilation provides early evidence of a potential cross-validation of these metrics. No effects were observed for the EBR metric; however, the size of this effect, if present at all, may be smaller than we are currently able to detect with the limited sample.
The observed reversal of attentional bias and pupil dilation toward neutral cues in the Driving scene warrants further investigation, given the large effect size. Potential explanations for this include the presence of especially engaging neutral cues in the Driving scene, as a 360° video of a busy city street is presented in the background, which participants report as entertaining to watch. Despite the overall bias toward neutral cues reflected in the global attentional bias metric, and within the Driving scene alone, participants with greater attentional bias toward NTP cues (even if negative) were found to endorse greater NTP use in the previous 90 days. This effect appears to be driven by the higher-frequency NTP users in our sample and is consistent with the literature supporting the validity of attentional bias as a clinically important indicator of nicotine addiction [27]. Additional analyses are planned to assess direct and indirect relationships between scene eye-related outcomes and relevance to the individual, scene-specific craving level, randomization of scenes, engagement with specific cues, and NTP use groups (ie, nondaily vs daily NTP users) once more data are collected.
Strengths and Limitations
This pilot study has several strengths and limitations. Strengths include the development of a cutting-edge VR cue-reactivity task that incorporates the latest technological advances in graphic design to increase translatability to the real-world and simultaneous assessment of multiple potential eye-related indices of cue-reactivity in a 3D virtual environment. Limitations include the absence of biological verification to confirm self-reported NTP use and the inability to investigate NTP use profiles in the analyses owing to limited power. Importantly, given the limited sample size, we caution against over interpretation of our results. It remains unknown whether the absence of significant results, particularly with respect to the correlations between objective eye-related indices and subjective craving ratings, are the result of limited power to detect these relationships or true independence of these indices. However, we believe that the general pattern of scene-related effects on attentional bias and pupil dilation are encouraging and warrant further study. The identification of reliable objective correlates (“biomarkers”) of craving would allow for greater examination of the underlying neurobiological processes involved, and inform new avenues for the development of psychological and pharmacological treatments.
Conclusions
To our knowledge, this is the first attempt to investigate eye-tracking indices (attentional bias, pupillometry, or EBR) within a VR substance cue-exposure paradigm. Taken together, the results of this preliminary data analysis suggest that this paradigm may prove useful for laboratory-based studies of NTP cue-reactivity and provide a platform for further investigation of eye-based markers of psychophysiological processes that may subserve the subjective craving experience. Once thoroughly tested and validated, this paradigm could function as a translatable platform for which experimental manipulations and craving interventions could be tested.
Acknowledgments
This work was supported by the California Tobacco-Related Disease Research Grants Program Office of the University of California grant T30IP0962 and the National Institutes of Health grant UL1TR001442.
Abbreviations
- CDDR
Customary Drinking and Drug Use Record
- D2
dopamine
- DSM-5
Diagnostic and Statistical Manual of Mental Disorders (5th edition)
- EBR
spontaneous eye blink rate
- G2OM
Gaze-to-Object Mapping
- ICD-10
International Classification of Diseases (10th edition)
- IPQ
Igroup Presence Questionnaire
- MINI
Mini International Neuropsychiatric Interview
- NE
norepinephrine
- NTP
nicotine and tobacco product
- PET
positron emission tomography
- SSQ
Simulator Sickness Questionnaire
- TCQ-SF
Tobacco Craving Questionnaire-Short-Form
- TD
PATH Tobacco Dependence
- TLFB
90-day Timeline Follow-Back
- VAS
visual analog scale
- VR
virtual reality
- VRSQ
Virtual Reality Sickness Questionnaire
Footnotes
Conflicts of Interest: None declared.
References
- 1.Robinson T, Berridge KC. The neural basis of drug craving: an incentive-sensitization theory of addiction. Brain Res Brain Res Rev. 1993;18(3):247–291. doi: 10.1016/0165-0173(93)90013-p.0165-0173(93)90013-P [DOI] [PubMed] [Google Scholar]
- 2.Robinson TE, Berridge KC. Incentive-sensitization and addiction. Addiction. 2001 Jan;96(1):103–114. doi: 10.1046/j.1360-0443.2001.9611038.x. [DOI] [PubMed] [Google Scholar]
- 3.American Psychiatric Association . Diagnostic and statistical manual of mental disorders (5th edition) Arlington, VA: American Psychiatric Publishing; 2013. [Google Scholar]
- 4.International statistical classification of diseases and related health problems (10th edition) Geneva: World Health Organization; 2010. [Google Scholar]
- 5.Bagot KS, Heishman SJ, Moolchan ET. Tobacco craving predicts lapse to smoking among adolescent smokers in cessation treatment. Nicotine Tob Res. 2007 Jun;9(6):647–652. doi: 10.1080/14622200701365178.779329730 [DOI] [PubMed] [Google Scholar]
- 6.Taggar JS, Lewis S, Docherty G, Bauld L, McEwen A, Coleman T. Do cravings predict smoking cessation in smokers calling a national quit line: secondary analyses from a randomised trial for the utility of 'urges to smoke' measures. Subst Abuse Treat Prev Policy. 2015 Apr 14;10:15. doi: 10.1186/s13011-015-0011-8. https://substanceabusepolicy.biomedcentral.com/articles/10.1186/s13011-015-0011-8 .10.1186/s13011-015-0011-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Baker TB, Mermelstein R, Collins LM, Piper ME, Jorenby DE, Smith SS, Christiansen BA, Schlam TR, Cook JW, Fiore MC. New methods for tobacco dependence treatment research. Ann Behav Med. 2011 Apr;41(2):192–207. doi: 10.1007/s12160-010-9252-y. http://europepmc.org/abstract/MED/21128037 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Tiffany S, Warthen M, Goedeker K. The Functional Significance of Craving in Nicotine Dependence. In: Caggiula AR, Bevins RA, editors. The Motivational Impact of Nicotine and its Role in Tobacco Use. New York, NY: Springer; 2008. Sep 19, pp. 171–197. [Google Scholar]
- 9.Hone-Blanchet A, Wensing T, Fecteau S. The use of virtual reality in craving assessment and cue-exposure therapy in substance use disorders. Front Hum Neurosci. 2014;8:844. doi: 10.3389/fnhum.2014.00844. doi: 10.3389/fnhum.2014.00844. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Shiffman S, West R, Gilbert D, SRNT Work Group on the Assessment of Craving Withdrawal in Clinical Trials Recommendation for the assessment of tobacco craving and withdrawal in smoking cessation trials. Nicotine Tob Res. 2004 Aug;6(4):599–614. doi: 10.1080/14622200410001734067.L5Y064PT3BHJDHER [DOI] [PubMed] [Google Scholar]
- 11.O'Brien CP, Childress AR, McLellan AT, Ehrman R. Classical conditioning in drug-dependent humans. Ann N Y Acad Sci. 1992 Jun 28;654:400–415. doi: 10.1111/j.1749-6632.1992.tb25984.x. [DOI] [PubMed] [Google Scholar]
- 12.See RE. Neural substrates of conditioned-cued relapse to drug-seeking behavior. Pharmacol Biochem Behav. 2002 Mar;71(3):517–529. doi: 10.1016/s0091-3057(01)00682-7.S0091305701006827 [DOI] [PubMed] [Google Scholar]
- 13.Bordnick PS, Traylor A, Copp HL, Graap KM, Carter B, Ferrer M, Walton AP. Assessing reactivity to virtual reality alcohol based cues. Addict Behav. 2008 Jun;33(6):743–756. doi: 10.1016/j.addbeh.2007.12.010.S0306-4603(07)00348-6 [DOI] [PubMed] [Google Scholar]
- 14.Wilson SJ, Sayette MA. Neuroimaging craving: urge intensity matters. Addiction. 2015 Feb;110(2):195–203. doi: 10.1111/add.12676. http://europepmc.org/abstract/MED/25073979 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Traylor AC, Parrish DE, Copp HL, Bordnick PS. Using virtual reality to investigate complex and contextual cue reactivity in nicotine dependent problem drinkers. Addict Behav. 2011 Nov;36(11):1068–1075. doi: 10.1016/j.addbeh.2011.06.014.S0306-4603(11)00194-8 [DOI] [PubMed] [Google Scholar]
- 16.Acker J, MacKillop J. Behavioral economic analysis of cue-elicited craving for tobacco: a virtual reality study. Nicotine Tob Res. 2013 Aug;15(8):1409–1416. doi: 10.1093/ntr/nts341. http://europepmc.org/abstract/MED/23322768 .nts341 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Baumann SB, Sayette MA. Smoking cues in a virtual world provoke craving in cigarette smokers. Psychol Addict Behav. 2006 Dec;20(4):484–489. doi: 10.1037/0893-164X.20.4.484.2006-22487-014 [DOI] [PubMed] [Google Scholar]
- 18.Ryan JJ, Kreiner DS, Chapman MD, Stark-Wroblewski K. Virtual reality cues for binge drinking in college students. Cyberpsychol Behav Soc Netw. 2010 Apr;13(2):159–162. doi: 10.1089/cyber.2009.0211.10.1089/cyber.2009.0211 [DOI] [PubMed] [Google Scholar]
- 19.Bordnick PS, Copp HL, Traylor A, Graap KM, Carter BL, Walton A, Ferrer M. Reactivity to cannabis cues in virtual reality environments. J Psychoactive Drugs. 2009 Jun;41(2):105–112. doi: 10.1080/02791072.2009.10399903. http://europepmc.org/abstract/MED/19705672 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Culbertson C, Nicolas S, Zaharovits I, London ED, De La Garza R, Brody AL, Newton TF. Methamphetamine craving induced in an online virtual reality environment. Pharmacol Biochem Behav. 2010 Oct;96(4):454–460. doi: 10.1016/j.pbb.2010.07.005. http://europepmc.org/abstract/MED/20643158 .S0091-3057(10)00206-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Monti PM, Rohsenow DJ, Hutchison KE. Toward bridging the gap between biological, psychobiological and psychosocial models of alcohol craving. Addiction. 2000 Aug;95 Suppl 2:S229–S236. doi: 10.1080/09652140050111799. [DOI] [PubMed] [Google Scholar]
- 22.Ostafin BD, Feyel N. The effects of a brief meaning in life intervention on the incentive salience of alcohol. Addict Behav. 2019 Mar;90:107–111. doi: 10.1016/j.addbeh.2018.10.035.S0306-4603(18)30840-2 [DOI] [PubMed] [Google Scholar]
- 23.Mogg K, Bradley BP, Field M, De Houwer J. Eye movements to smoking-related pictures in smokers: relationship between attentional biases and implicit and explicit measures of stimulus valence. Addiction. 2003 Jun;98(6):825–836. doi: 10.1046/j.1360-0443.2003.00392.x.392 [DOI] [PubMed] [Google Scholar]
- 24.Bradley B, Field M, Mogg K, De Houwer J. Attentional and evaluative biases for smoking cues in nicotine dependence: component processes of biases in visual orienting. Behav Pharmacol. 2004 Feb;15(1):29–36. doi: 10.1097/00008877-200402000-00004.00008877-200402000-00004 [DOI] [PubMed] [Google Scholar]
- 25.Kwak S, Na DL, Kim G, Kim GS, Lee J. Use of eye movement to measure smokers' attentional bias to smoking-related cues. Cyberpsychol Behav. 2007 Apr;10(2):299–304. doi: 10.1089/cpb.2006.9953. [DOI] [PubMed] [Google Scholar]
- 26.Gamito P, Oliveira J, Baptista A, Morais D, Lopes P, Rosa P, Santos N, Brito R. Eliciting nicotine craving with virtual smoking cues. Cyberpsychol Behav Soc Netw. 2014 Aug;17(8):556–561. doi: 10.1089/cyber.2013.0329. [DOI] [PubMed] [Google Scholar]
- 27.Waters AJ, Shiffman S, Sayette MA, Paty JA, Gwaltney CJ, Balabanis MH. Attentional bias predicts outcome in smoking cessation. Health Psychol. 2003 Jul;22(4):378–387. doi: 10.1037/0278-6133.22.4.378. http://europepmc.org/abstract/MED/12940394 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Ryan F. Detected, selected, and sometimes neglected: cognitive processing of cues in addiction. Exp Clin Psychopharmacol. 2002 May;10(2):67–76. doi: 10.1037//1064-1297.10.2.67. [DOI] [PubMed] [Google Scholar]
- 29.Franken IH. Drug craving and addiction: integrating psychological and neuropsychopharmacological approaches. Prog Neuropsychopharmacol Biol Psychiatry. 2003 Jun;27(4):563–579. doi: 10.1016/S0278-5846(03)00081-2.S0278-5846(03)00081-2 [DOI] [PubMed] [Google Scholar]
- 30.Field M, Munafò MR, Franken IHA. A meta-analytic investigation of the relationship between attentional bias and subjective craving in substance abuse. Psychol Bull. 2009 Jul;135(4):589–607. doi: 10.1037/a0015843. http://europepmc.org/abstract/MED/19586163 .2009-09537-005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Soleymani A, Ivanov Y, Mathot S, de Jong PJ. Free-viewing multi-stimulus eye tracking task to index attention bias for alcohol versus soda cues: Satisfactory reliability and criterion validity. Addict Behav. 2020 Jan;100:106117. doi: 10.1016/j.addbeh.2019.106117.S0306-4603(19)30178-9 [DOI] [PubMed] [Google Scholar]
- 32.Marks KR, Pike E, Stoops WW, Rush CR. The magnitude of drug attentional bias is specific to substance use disorder. Psychol Addict Behav. 2015 Sep;29(3):690–695. doi: 10.1037/adb0000084. http://europepmc.org/abstract/MED/25961149 .2015-20855-001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Marks KR, Roberts W, Stoops WW, Pike E, Fillmore MT, Rush CR. Fixation time is a sensitive measure of cocaine cue attentional bias. Addiction. 2014 Sep;109(9):1501–1508. doi: 10.1111/add.12635. http://europepmc.org/abstract/MED/24894879 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Marks KR, Pike E, Stoops WW, Rush CR. Test-retest reliability of eye tracking during the visual probe task in cocaine-using adults. Drug Alcohol Depend. 2014 Dec 01;145:235–237. doi: 10.1016/j.drugalcdep.2014.09.784. http://europepmc.org/abstract/MED/25456573 .S0376-8716(14)01850-X [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Spanakis P, Jones A, Field M, Christiansen P. A Stroop in the Hand is Worth Two on the Laptop: Superior Reliability of a Smartphone Based Alcohol Stroop in the Real World. Subst Use Misuse. 2019;54(4):692–698. doi: 10.1080/10826084.2018.1536716. [DOI] [PubMed] [Google Scholar]
- 36.Monem RG, Fillmore MT. Measuring heightened attention to alcohol in a naturalistic setting: A validation study. Exp Clin Psychopharmacol. 2017 Dec;25(6):496–502. doi: 10.1037/pha0000157. http://europepmc.org/abstract/MED/29251979 .2017-55847-008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Kruis A, Slagter HA, Bachhuber DRW, Davidson RJ, Lutz A. Effects of meditation practice on spontaneous eyeblink rate. Psychophysiology. 2016 May;53(5):749–758. doi: 10.1111/psyp.12619. http://europepmc.org/abstract/MED/26871460 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Jongkees BJ, Colzato LS. Spontaneous eye blink rate as predictor of dopamine-related cognitive function-A review. Neurosci Biobehav Rev. 2016 Dec;71:58–82. doi: 10.1016/j.neubiorev.2016.08.020.S0149-7634(16)30284-6 [DOI] [PubMed] [Google Scholar]
- 39.Kaminer J, Powers AS, Horn KG, Hui C, Evinger C. Characterizing the spontaneous blink generator: an animal model. J Neurosci. 2011 Aug 03;31(31):11256–11267. doi: 10.1523/JNEUROSCI.6218-10.2011. http://www.jneurosci.org/cgi/pmidlookup?view=long&pmid=21813686 .31/31/11256 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Kleven MS, Koek W. Differential effects of direct and indirect dopamine agonists on eye blink rate in cynomolgus monkeys. J Pharmacol Exp Ther. 1996 Dec;279(3):1211–1219. [PubMed] [Google Scholar]
- 41.Desai RI, Neumeyer JL, Bergman J, Paronis CA. Pharmacological characterization of the effects of dopamine D(1) agonists on eye blinking in rats. Behav Pharmacol. 2007 Dec;18(8):745–754. doi: 10.1097/FBP.0b013e3282f14ee6.00008877-200712000-00005 [DOI] [PubMed] [Google Scholar]
- 42.Groman SM, James AS, Seu E, Tran S, Clark TA, Harpster SN, Crawford M, Burtner JL, Feiler K, Roth RH, Elsworth JD, London ED, Jentsch JD. In the blink of an eye: relating positive-feedback sensitivity to striatal dopamine D2-like receptors through blink rate. J Neurosci. 2014 Oct 22;34(43):14443–14454. doi: 10.1523/JNEUROSCI.3037-14.2014. http://www.jneurosci.org/cgi/pmidlookup?view=long&pmid=25339755 .34/43/14443 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Volkow ND, Wang G, Telang F, Fowler JS, Logan J, Childress A, Jayne M, Ma Y, Wong C. Cocaine cues and dopamine in dorsal striatum: mechanism of craving in cocaine addiction. J Neurosci. 2006 Jun 14;26(24):6583–6588. doi: 10.1523/JNEUROSCI.1544-06.2006. http://www.jneurosci.org/cgi/pmidlookup?view=long&pmid=16775146 .26/24/6583 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Zijlstra F, Booij J, van den Brink W, Franken IHA. Striatal dopamine D2 receptor binding and dopamine release during cue-elicited craving in recently abstinent opiate-dependent males. Eur Neuropsychopharmacol. 2008 Apr;18(4):262–270. doi: 10.1016/j.euroneuro.2007.11.002.S0924-977X(07)00206-4 [DOI] [PubMed] [Google Scholar]
- 45.Anderson BA, Yantis S. Value-driven attentional and oculomotor capture during goal-directed, unconstrained viewing. Atten Percept Psychophys. 2012 Nov;74(8):1644–1653. doi: 10.3758/s13414-012-0348-2. http://europepmc.org/abstract/MED/22810561 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Muhammed K, Manohar S, Ben Yehuda M, Chong TT, Tofaris G, Lennox G, Bogdanovic M, Hu M, Husain M. Reward sensitivity deficits modulated by dopamine are associated with apathy in Parkinson's disease. Brain. 2016 Oct;139(Pt 10):2706–2721. doi: 10.1093/brain/aww188. http://europepmc.org/abstract/MED/27452600 .aww188 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Bijleveld E, Custers R, Aarts H. The unconscious eye opener: pupil dilation reveals strategic recruitment of resources upon presentation of subliminal reward cues. Psychol Sci. 2009 Nov;20(11):1313–1315. doi: 10.1111/j.1467-9280.2009.02443.x.PSCI2443 [DOI] [PubMed] [Google Scholar]
- 48.Laeng B, Sirois S, Gredebäck G. Pupillometry: A Window to the Preconscious? Perspect Psychol Sci. 2012 Jan;7(1):18–27. doi: 10.1177/1745691611427305.7/1/18 [DOI] [PubMed] [Google Scholar]
- 49.Kvamme TL, Pedersen MU, Overgaard M, Rømer Thomsen K, Voon V. Pupillary reactivity to alcohol cues as a predictive biomarker of alcohol relapse following treatment in a pilot study. Psychopharmacology (Berl) 2019 Apr;236(4):1233–1243. doi: 10.1007/s00213-018-5131-1. http://europepmc.org/abstract/MED/30607476 .10.1007/s00213-018-5131-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Eye and Facial Tracking SDK. VIVE Developers. [2022-02-08]. https://developer-express.vive.com/resources/vive-sense/eye-and-facial-tracking-sdk/
- 51.Tobii XR SDK. [2022-02-08]. https://vr.tobii.com/sdk/
- 52.Wewers ME, Rachfal C, Ahijevych K. A psychometric evaluation of a visual analogue scale of craving for cigarettes. West J Nurs Res. 1990 Oct;12(5):672–681. doi: 10.1177/019394599001200508. [DOI] [PubMed] [Google Scholar]
- 53.Caffier PP, Erdmann U, Ullsperger P. Experimental evaluation of eye-blink parameters as a drowsiness measure. Eur J Appl Physiol. 2003 May;89(3-4):319–325. doi: 10.1007/s00421-003-0807-5. [DOI] [PubMed] [Google Scholar]
- 54.Schleicher R, Galley N, Briest S, Galley L. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics. 2008 Jul;51(7):982–1010. doi: 10.1080/00140130701817062.793008494 [DOI] [PubMed] [Google Scholar]
- 55.Wang Y, Toor SS, Gautam R, Henson DB. Blink frequency and duration during perimetry and their relationship to test-retest threshold variability. Invest Ophthalmol Vis Sci. 2011 Jun 28;52(7):4546–4550. doi: 10.1167/iovs.10-6553.iovs.10-6553 [DOI] [PubMed] [Google Scholar]
- 56.Doughty MJ. Consideration of three types of spontaneous eyeblink activity in normal humans: during reading and video display terminal use, in primary gaze, and while in conversation. Optom Vis Sci. 2001 Oct;78(10):712–725. doi: 10.1097/00006324-200110000-00011.00006324-200110000-00011 [DOI] [PubMed] [Google Scholar]
- 57.Tsubota K, Nakamori K. Dry eyes and video display terminals. N Engl J Med. 1993 Feb 25;328(8):584. doi: 10.1056/NEJM199302253280817. [DOI] [PubMed] [Google Scholar]
- 58.Abusharha A. Changes in blink rate and ocular symptoms during different reading tasks. Clin Optom (Auckl) 2017;9:133–138. doi: 10.2147/OPTO.S142718. doi: 10.2147/OPTO.S142718.opto-9-133 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Jones A, Hogarth L, Christiansen P, Rose AK, Martinovic J, Field M. Reward expectancy promotes generalized increases in attentional bias for rewarding stimuli. Q J Exp Psychol (Hove) 2012;65(12):2333–2342. doi: 10.1080/17470218.2012.686513. [DOI] [PubMed] [Google Scholar]
- 60.Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, Hergueta T, Baker R, Dunbar GC. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59 Suppl 20:22–33;quiz 34-57. [PubMed] [Google Scholar]
- 61.Sobell L, Sobell M. Timeline follow back user's guide: A calendar for assessing alcohol and drug use. Toronto, ON: Addiction Research Foundation; 1996. [Google Scholar]
- 62.Strong DR, Messer K, Hartman SJ, Conway KP, Hoffman AC, Pharris-Ciurej N, White M, Green VR, Compton WM, Pierce J. Measurement of multiple nicotine dependence domains among cigarette, non-cigarette and poly-tobacco users: Insights from item response theory. Drug Alcohol Depend. 2015 Jul 01;152:185–193. doi: 10.1016/j.drugalcdep.2015.03.040. http://europepmc.org/abstract/MED/26005043 .S0376-8716(15)00206-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63.Brown SA, Myers MG, Lippke L, Tapert SF, Stewart DG, Vik PW. Psychometric evaluation of the Customary Drinking and Drug Use Record (CDDR): a measure of adolescent alcohol and drug involvement. J Stud Alcohol. 1998 Jul;59(4):427–438. doi: 10.15288/jsa.1998.59.427. [DOI] [PubMed] [Google Scholar]
- 64.Heishman SJ, Singleton EG, Pickworth WB. Reliability and validity of a Short Form of the Tobacco Craving Questionnaire. Nicotine Tob Res. 2008 Apr;10(4):643–651. doi: 10.1080/14622200801908174. http://europepmc.org/abstract/MED/18418787 .792239917 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Robinson SM, Sobell LC, Sobell MB, Leo GI. Reliability of the Timeline Followback for cocaine, cannabis, and cigarette use. Psychol Addict Behav. 2014 Mar;28(1):154–162. doi: 10.1037/a0030992.2012-34893-001 [DOI] [PubMed] [Google Scholar]
- 66.Schubert TW. The sense of presence in virtual environments: Zeitschrift für Medienpsychologie. 2003 Apr;15(2):69–71. doi: 10.1026//1617-6383.15.2.69. [DOI] [Google Scholar]
- 67.Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. Int J Aviat Psychol. 1993 Jul;3(3):203–220. doi: 10.1207/s15327108ijap0303_3. [DOI] [Google Scholar]
- 68.Kim HK, Park J, Choi Y, Choe M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Appl Ergon. 2018 May;69:66–73. doi: 10.1016/j.apergo.2017.12.016.S0003-6870(17)30282-X [DOI] [PubMed] [Google Scholar]