Abstract
We evaluated an Internet-based contingency management intervention to promote smoking cessation. Participants in the contingent group (n = 39) earned vouchers contingent on video confirmation of breath carbon monoxide (CO) ≤ 4 parts per million (ppm). Earnings for participants in the noncontingent group (n = 38) were independent of CO levels. Goals and feedback about smoking status were provided on participants’ homepages. The median percentages of negative samples during the intervention in the noncontingent and contingent groups were 25% and 66.7%, respectively. There were no significant differences in absolute CO levels or abstinence at 3- and 6-month follow-ups. Compared to baseline, however, participants in both groups reduced CO by an estimated 15.6 ppm during the intervention phases. The results suggest that the contingency for negative COs promoted higher rates of abstinence during treatment, and that other elements of the system, such as feedback, frequent monitoring, and goals, reduced smoking.
Keywords: cigarette smoking, contingency management, delay discounting, Internet
Information technology permeates our everyday lives. Recently, researchers and clinicians have harnessed the Internet and mobile phones to deliver behavioral health interventions for a range of problems (Boyer, Smelson, Fletcher, Ziedonis, & Picard, 2010; Gainsbury & Blaszczynski, 2011; Goodwin, Velicer, & Intille, 2008; Marsch & Dallery, 2012), including smoking cessation (e.g., Cobb, Graham, Byron, Niaura, & Abrams, 2011; Japuntich et al., 2006; McDaniel & Stratton, 2006; Seidman et al., 2010). For example, Dallery and colleagues developed and tested an Internet-based contingency management intervention to promote smoking cessation (Dallery & Glenn, 2005; Dallery, Glenn, & Raiff, 2007; Dallery, Meredith, & Glenn, 2008; Dallery & Raiff, 2011; Glenn & Dallery, 2007; Stoops et al., 2009). Breath carbon monoxide (CO) was verified by observing participants providing their samples via a web camera. Monetary vouchers, which could be exchanged for goods and services available over the Internet, were delivered contingent on negative breath CO samples. There was excellent adherence to the two-sample per day breath CO collection schedule. In one study, for example, participants provided more than 97% of 1,120 total samples scheduled during the 4-week study (Dallery et al., 2007). Furthermore, compared to a CO monitoring baseline condition in which 3% of the breath CO samples were negative for recent smoking, more than 65% of the 400 breath CO samples collected during treatment were negative (CO ≤ 7 parts per million [ppm]).
Breath CO serves as the objective index of smoking status in most contingency management interventions for smoking cessation (Sigmon, Lamb, & Dallery, 2008). Because breath CO has a half-life of about 6 to 8 hr, samples should be collected at least twice per day to increase the validity of measures of smoking status (Crowley, MacDonald, Zerbe, & Petty, 1991; Javors, Hatch, & Lamb, 2005; cf Yoon, Higgins, Bradstreet, Badger, & Thomas, 2009). Further, CO samples should be collected for a sustained period to initiate and possibly maintain abstinence (e.g., at least for several weeks to initiate abstinence; Gourlay, Forbes, Marriner, Pethica, & McNeil, 1994; Kenford et al., 1994; Yoon et al., 2009). Given these requirements, an in-person method to collect CO samples, either at participants’ homes or in a treatment clinic, would constitute a significant barrier to treatment. The Internet-based system appeared to obviate the barriers to treatment entailed by frequent CO monitoring (Dallery & Raiff, 2011).
Internet-based contingency management has been shown to promote abstinence in heavy (Dallery, Glenn, & Raiff, 2007), adolescent (Reynolds, Dallery, Shroff, Patak, & Leraas, 2008), and rural (Stoops et al., 2009) smokers. Stoops et al. (2009) conducted a small, two-group, randomized study to evaluate the impact of contingent vouchers relative to vouchers delivered regardless of CO status (the noncontingent CO group) in a sample of rural smokers. Participant earnings in the noncontingent CO group were yoked, or matched, to participants in the contingent CO group. Thus, the total amount earned in both groups was equivalent; the difference between groups was whether vouchers were contingent on smoking abstinence. During the 6-week treatment period, 36.6% of CO samples were negative (CO ≤ 4 ppm) in the contingent CO group, and 11.3% were negative in the noncontingent CO group. Between-groups differences in abstinence were not maintained at a 12-week follow-up.
The primary objective of the current study was to evaluate further the efficacy of Internet-based contingency management to promote smoking cessation. As in the Stoops et al. (2009) study, participants were randomized to a contingent CO or a noncontingent CO group. We also conducted follow-up assessments at 3 and 6 months after the main treatment phase, which is longer than in previous Internet-based contingency management studies. Because factors associated with treatment response to this novel intervention have not been explored, a secondary objective was to assess predictors of outcomes during treatment based on demographics, smoking characteristics, and individual differences in temporal discounting (i.e., impulsive choice; Bickel et al., 2010; M. E. Carroll, Anker, Mach, Newman, & Perry, 2010; Dallery & Raiff, 2007; Yoon et al., 2007).
METHOD
Participants
Participants were healthy smokers recruited through print media and word of mouth in north-central Florida. Participant characteristics are presented in Table 1. To be included in the study, smokers had to be between the ages of 18 and 60 years, smoke > 10 cigarettes per day, blow a CO at intake of ≥ 10 ppm, report a minimum 2-year smoking history, own a computer and have Internet access, and express a desire to quit smoking (a score of 8 or greater on a scale of 1 to 10 in response to the question, “How much do you want to quit smoking right now?”). Applicants were excluded if they showed evidence of current alcohol dependence or drug use (verified by self-report or urinalysis for cocaine, benzodiazepines, and opiates), reported smoking marijuana more than twice per month, reported a history of medical or psychiatric illness that, in our judgment, would interfere with the study, or resided with another smoker who smoked inside the home. Women were disqualified if they were pregnant (assessed via urine testing). The local institutional review board approved all study procedures.
Table 1.
Characteristic | Contingent group (n = 39) |
Noncontingent group (n = 38) |
---|---|---|
Age (years) | 39.3 ± 13.20 | 40.1 ± 13.28 |
Female (%) | 49 | 47 |
Caucasian (%) | 74.4 | 81.6 |
Cigarettes per day | 20.4 ± 8.74 | 20.6 ± 9.65 |
Weight (kg) | 80.4 ± 16.7 | 79.3 ± 19.1 |
Intake CO (ppm) | 25.1 ± 15.82 | 25.2 ± 13.14 |
Years smoking | 19.64 ± 12.37 | 19.68 ± 12.74 |
Nicotine dependence (FTND) | 5.13 ± 1.90 | 5.24 ± 2.54 |
Readiness to change (URICA) | 9.9 ± 1.27 | 10.1 ± 1.52 |
Delay discounting (AUC) | 0.421 ± 0.270 | 0.454± 0.228 |
Note. Values are means ±SD unless otherwise indicated.
Materials
Participants were loaned a CO monitor and a web camera if needed. In seven cases, a laptop computer was loaned to participants if their computer did not work properly. CO monitors were initially calibrated by the manufacturer and then at least every 6 months thereafter (per the manufacturer’s recommendations) using a calibration kit provided by the distributor. Copies of participants’ driver’s licenses were obtained, and they were asked to sign an off-campus property certificate and contract stating that they would return the equipment.
Screening
Interested persons were screened over the phone for basic qualifying criteria such as smoking severity, desire to quit smoking, and age. Qualified applicants were scheduled for an in-person screening. During the screening, participants underwent an informed consent process during which a member of the research team described the study procedures in detail. Participants were informed that they could earn a maximum $530.00 in voucher earnings during the study. The vouchers could be exchanged for goods or gift certificates purchased over the Internet (e.g., Amazon.com) at any point during the study. Participants were informed that they could purchase desired items as soon as they earned enough money to cover the cost of the item plus shipping and handling. Participants were also informed about environmental sources that could elevate CO readings. In particular, participants were instructed to avoid secondhand or environmental smoke, and avoid smoking any other substances (e.g., marijuana). A CO reading above 4 ppm was always considered positive, regardless of the cause. If a participant believed the sample was a false positive, he or she could contest the result. If the samples before and after the contested sample were negative, the contested sample was considered negative and vouchers were awarded (but for the purposes of data analysis, the sample was considered positive).
In addition, participants completed a brief psychosocial history and several questionnaires. The psychosocial history contained questions related to demographics, smoking history, prior drug use and abuse, general health, medication use, and the Fagerstrom Test for Nicotine Dependence (FTND; Fagerstrom & Schneider, 1989), which is a 6-item questionnaire that assesses nicotine dependence. Scores can range from 0 to 10, with 0 representing very low dependence and 10 representing very high dependence. Urine and CO samples were collected, and the urine samples were analyzed for the presence of cocaine, benzodiazepines, and opiates. Weight was measured on a regularly calibrated electronic scale.
Participants also completed the University of Rhode Island Change Assessment (URICA; Prochaska & DiClemente, 1983), which is a self-report measure that assesses four stages of change (precontemplation, contemplation, action, and maintenance). Participants responded on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). The subscales are combined arithmetically to yield a continuous score that can be used to assess self-reported readiness to change.
Finally, participants completed a delay-discounting task on a computer. The task was based on prior research that used similar procedures (Dallery & Raiff, 2007; Johnson & Bickel, 2002). Participants clicked one of two response buttons presented on the computer screen. The left button corresponded to an adjusting amount of money available immediately, and the right button corresponded to $100.00 after one of eight delays (1 week, 2 weeks, 1 month, 4 months, 8 months, 1 year, 5 years, and 10 years) presented in random order. An algorithm was used to adjust the value of the immediate adjusting amount of money so that an indifference point, or the point at which each option was approximately equated in terms of value, was determined at each delay (for detailed information about the computer algorithm, see the Supporting Information or contact the first author). All choices during the discounting task were hypothetical in the sense that participants did not actually receive the amounts they chose. The only instructions for this task were located above the two response alternatives and said, “Please choose the option you prefer, based on how you feel right now.” A summary measure of delay discounting, the area under the curve (AUC), was calculated by normalizing the delays and indifference points and then summing the areas under each pair of the eight indifference points. Specifically, the AUC was calculated by summing the results of the following equation: x2−x1 [(y1 +y2)/2], where x1 and x2 are successive delays and y1 and y2 are the present values associated with those delays. The AUC measure can thus range from 0 (maximum discounting) to 1 (no discounting). The AUC yields a normally distributed measure of discounting (see Myerson, Green, & Warusawitharana, 2001, for details).
Design
The study used a two-group randomized design. The primary difference between the two groups was whether vouchers were delivered contingently (contingent CO group) or non-contingently (noncontingent CO group) based on smoking abstinence. Participants also were informed about the availability of vouchers on a sample-by-sample basis in the contingent CO group during the intervention. The first five participants were assigned to the contingent CO group to ensure that there were enough participants to use in the yoking procedure (see below). The remaining participants were randomly assigned to the contingent CO or noncontingent CO group after the screening phase (Signorini et al., 1993). Participants were stratified based on smoking severity (>30 cigarettes per day, yes–no) and gender (Borrelli, Spring, Niaura, Hitsman, & Papandonatos, 2001). Each participant in the noncontingent CO group was yoked to a single participant (selected randomly) in the contingent CO group. The yoked participant received the same vouchers as his or her matched participant in the contingent CO group (e.g., Higgins, Wong, Badger, Ogden, & Dantona, 2000; Silverman et al., 1998). Noncontingent CO participants were told that the only way to earn vouchers was by submitting CO videos, but that the vouchers would be delivered on an unpredictable schedule and they may or may not actually earn any vouchers. They were also told that earning vouchers would not be based on self-reports of smoking, cotinine levels, or for meeting CO goals (described below). Vouchers were delivered immediately after a CO video was submitted, but the amount of the voucher was determined by the participant’s yoked contingent counterpart.
Internet-Based CO Monitoring and Contingency Management
A software engineer developed a web-based application called Mōtiv8 for managing contingency management programs and collecting CO video samples. Study participants logged into the Mōtiv8 website using a username and password and completed the CO video sample collection by following simple on-screen and printed instructions. The web-based sample-collection process, explained in detail below, progressed as follows: participant homepage, bandwidth selector, video post result entry, voucher display, back to homepage.
The participant’s homepage, which was accessed over a secure, encrypted connection, consisted of a graph of the participant’s sample results (showing cumulative smoking abstinence), voucher earnings history, a box showing messages from the system administrator, a “post video” button, and a display showing his or her previous sample’s date and time and the earliest date and time at which he or she could provide their next sample (see Figure 1). The procedure specified that an 8-hr interval must elapse between samples. When the participant clicked the “post video” button, the program directed him or her to the bandwidth selector interface. The participant selected whether he or she was using a high-speed connection (a digital subscriber line, cable modem, or satellite) or a low-speed connection (dial up over a phone line).
The video post interface guided the participant through the process of uploading a CO video sample and posting a result. On the video post screen, the participant was instructed to click a button, which turned on the webcam. A box (320 by 240 pixels) then displayed what the webcam was seeing, allowing the user to orient the camera properly to capture the result. The user was then instructed to click “record” when he or she was ready to leave a sample (see Figure 1a). When the participant wished to stop recording, he or she was instructed to click the “stop” button. The video stream from the webcam was sent directly to the study server over a secure, encrypted connection so that it could not be tampered with by the participant.
A properly recorded video contained the following components: (a) participant resetting the CO meter and showing that it was zeroed; (b) participant holding his or her breath while the meter counted down from 15 s; (c) participant exhaling into the CO meter; and (d) participant showing the results indicated by a digital display on the CO meter to the camera.
After recording the sampling procedure, the participant entered the numerical result indicated on the CO meter and posted the result in a text box. The result was fed into the voucher program, where it was processed by the Mōtiv8 incentive tracking system and the result and appropriate voucher amount were stored. The appropriate voucher interface was generated, and the participant was automatically directed to the voucher display. For the voucher display, the screen displayed different messages based on a positive or negative result and group assignment. If the participant was in the contingent CO group, the participant’s earnings and account balance were displayed, along with the amount he or she could earn for the next negative sample. The latter information about potential earnings for the next sample was not included for participants in the noncontingent CO group but was otherwise the same.
The user could then click a “continue” button to return to his or her homepage, where the latest graph, account summary, and earliest possible date and time for providing the next CO sample were displayed. The average time to upload a video after accessing the Mōtiv8 website was approximately 90 s. All videos were reviewed, usually twice per day, by trained staff, and the results entered by participants were compared to the results shown on the videos. Staff also made sure that the steps for submitting a video (described above) were properly followed and that participants were not attempting to fake a video. Visual cues, such as watching the chest expand before (during the initial deep breath) and after the exhalation, as well as audio cues, such as hearing the exhalation through the meter, were used to verify the authenticity of the CO submission process. If there were concerns about authenticity, the participant was contacted, and in some cases was asked to turn his or her head sideways during exhalation to show that his or her lips were around the mouthpiece of the meter during exhalation. Of the 4,774 samples, 39 were problematic, mainly for technical reasons, but four were discrepancies between the user input and the video-confirmed CO reading. None of these discrepancies, however, resulted in a voucher being awarded erroneously.
Procedure
After group assignment, research staff met with participants in their homes to provide equipment and instruct them regarding the various phases of the study as well as the CO sampling procedures. Participants completed a quiz demonstrating that they had read and understood the instructions for their group assignment (Silverman, Chutuape, Bigelow, & Stitzer, 1999).
Baseline
During the first 3 days, participants were asked to submit two video clips showing CO values each day. There were no cut-off requirements during this phase. An average CO was calculated at the end of this phase.
Tapering
For participants in the contingent CO group, reductions in breath CO were required for voucher delivery for each of the eight submissions during this 4-day phase. For participants in both groups, reduction goals were determined based on the average of the baseline CO levels, such that at the end of the phase the final CO cutpoint was ≤ 4 ppm. For example, if the average baseline CO was 36 ppm, then each successive sample would have to be 4 ppm lower than the previous sample so that the goal for the eighth sample was 4 ppm. Meeting the cut-off for each submission resulted in a $3.00 voucher if the participant was in the contingent CO group. Participants in the noncontingent CO group earned vouchers based on their matched contingent CO counterpart, contingent on submitting videos but independent of meeting their goals. Also, for all participants, the quantitative progress graph on each participant’s homepage showed his or her own personal tapering goals as horizontal dashed lines in the graph above the respective sample number (see Figure 1). In addition, for all participants, after leaving a sample, the goal for the next CO sample was indicated in a text box. The final day of tapering (Day 7) was designated as the quit date.
Abstinence induction
During the remainder of the voucher procedure (21 days), CO samples from participants in the contingent CO group were judged to be either positive or negative. Samples ≤ 4 ppm were considered negative. Evidence of continued abstinence resulted in an increasing schedule of voucher magnitude for contingent participants. The initial voucher value was $3.00 and increased in value by $0.25. In addition, every third consecutive negative CO resulted in a $5.00 bonus. Thus, the first negative sample resulted in a $3.00 voucher, the second in $3.25, the third in $8.50, the fourth in $3.75, and so on. Participants in the noncontingent CO group earned vouchers based on their matched voucher counterpart, contingent on video submissions but independent of abstinence. At the end of the abstinence induction phase, participants completed a Treatment Acceptability Scale (results reported by Raiff, Jarvis, Turturici, & Dallery, 2013).
Thinning
A 21-day thinning phase followed the main treatment phase. Monitoring and the possibility of voucher delivery were decreased to twice per week (one sample on Mondays and one sample on Thursdays, any time after 12:00 p.m. but before 12:00 a.m.). The voucher values simply continued based on the previous schedule for the contingent CO group. As with the previous two phases, participants in the noncontingent CO group earned vouchers based on their matched counterpart, contingent on video submissions but independent of abstinence.
Counseling
Brief supportive counseling was provided in person during the tapering phase, by phone after the quit day and after Week 4, and then another in-person session at the end of Week 7. Counselors followed a semistructured script based on Public Health Service guidelines (Fiore, 2000), and were not blind to group assignment. An identical script was used in each phase. Counselors were trained by a licensed psychologist.
Follow-up
Follow-up visits were scheduled at the completion of the thinning phase and at 3 and 6 months after completion of the thinning phase. Research assistants visited participants in their homes or participants came to the clinic. During these visits, participants provided CO and urine samples, and they were asked to report whether they had smoked cigarettes, even a puff, during the past 7 days (yes or no). Abstinence was defined by CO ≤ 4 ppm, urine cotinine < 80 ng/ml, and an answer of “no” to whether they had smoked during the past 7 days. Urine cotinine was assessed by a highly sensitive gas chromatography technique. Cotinine was excluded in the point prevalence measure for participants who reported nicotine replacement (e.g., a patch). This occurred for one participant at 4 weeks and one participant at 3 and 6 months in the contingent CO group, and for one participant in the noncontingent CO group at 6 months.
RESULTS
The main dependent variable for the study was breath CO level. Missing samples were considered positive. Figure 2 shows individual subject CO data for both groups during all phases of the study, with each row representing a different participant. Positive samples are indicated by a thin line, negative samples are represented by a thick line, and missing samples are indicated by blank spaces. Few samples were negative during baseline, suggesting that self-monitoring alone did not promote smoking abstinence. During tapering, slightly more CO samples met the 4 ppm cut point for abstinence compared to baseline. Visual analysis also suggests that there were higher rates of negative CO samples during the treatment phase in both groups compared to baseline.
Average CO levels (±SD) across baseline, tapering, abstinence induction, and thinning phases in the contingent CO group were 25.4 ppm (±17.4), 13.9 ppm (±12.3), 6.8 ppm (±11.8), 7.7 ppm (±13.8), and average COs in the noncontingent CO group were 23.5 ppm (±12.4), 16.3 ppm (±11.3), 8.0 ppm (±7.5), 9.3 ppm (±9.3). We used a mixed-effects linear model to assess the effects of group assignment and treatment phase on CO levels. Our response variable was CO level (ppm), and our fixed factors were group (contingent or noncontingent), phase (baseline, tapering, abstinence induction, or thinning) and the group-by-phase interaction, as well as subject weight, age, race, gender, FTND score, URICA score, and AUC. We modeled a random intercept for each subject and assumed a first-order autoregressive correlation structure across the four phases.
Mixed-model analysis showed no significant effect of group and no group-by-phase interaction, indicating that group status was not a reliable predictor of daily CO. However, CO level varied significantly by phase across both treatment groups. Our model estimated that CO level in the tapering phase was, on average, 9.0 ppm lower than at baseline (p < .0001, 95% CI =[−11.2, −6.8]), CO level in abstinence induction was 15.6 ppm lower than at baseline (p<.0001, 95% CI =[−17.8, −13.4]), and CO level in thinning was 14.5 ppm lower than at baseline (p < .0001,95% CI =[−16.7, −12.2]). Also, average CO level in thinning was estimated to be 5.5 ppm lower than in tapering (p < .0001, 95% CI =[−7.7, −3.2]), and CO level during abstinence induction was 6.6 ppm lower than during tapering (p < .0001, 95% CI =[−8.8, −4.4]). Average CO level during thinning was not significantly different than during abstinence induction (p =.3339).
Among patient demographic variables included in the mixed model (age, weight, race, gender, FTND, URICA, and AUC), CO level was significantly associated with FTND and marginally associated with AUC. Across both groups, average daily CO level was estimated to increase by 1.80 ppm for each additional point of the FTND score (95% CI =[0.501, 3.090], p =.0072), and CO level was estimated to decrease by 1.02 ppm for each additional increase of 0.1 of AUC (95% CI =[−2.111, −0.713], p =.0685).
Because contingencies were placed on the dichotomous outcome of smoking or abstaining, we also assessed group differences in the percentage of abstinence during treatment (abstinence induction and thinning phases combined). Figure 3 (top) shows the median percentage of negative samples in both groups (shaded bars) as well as the percentage of negative samples for each participant (filled circles). The median percentages of negative samples in the noncontingent CO and contingent CO groups were 25% and 66.7%, respectively (p =.048 by two-sided Mann-Whitney test). Figure 3 (bottom) shows the median number of days of sustained abstinence using the same conventions as in the top panel. Median days of sustained abstinence in the noncontingent CO and contingent CO groups were 2.25 and 5.50, respectively (p =.0731 by two-sided Mann-Whitney test).
Adherence to the video submission protocol was high in both groups. The median percentages of submitted samples in the noncontingent CO and contingent CO groups were 89.5% and 91.9%, respectively. In addition, participants returned the equipment (CO monitor, laptop, webcam) at a high rate: All laptops were collected, and only three CO monitors were not returned.
Out of the total maximum earnings of $530.00 for continuous smoking abstinence, the average amount earned in the contingent CO group was $245.70. Because participants were yoked, nearly the same magnitude of earnings was received by individuals in the noncontingent CO group ($223.88).
At the 3-month follow-up, 18% of the participants in the contingent group were abstinent, and 7.7% in the noncontingent CO group were abstinent, based on the definition of abstinence described above. At the 6-month follow-up, 8% in the contingent CO group were abstinent, and 15.8% in the noncontingent CO group were abstinent. χ2 tests indicated that these differences were not statistically significant. Participants lost to follow-up were considered positive for smoking (indicated by blank spaces in Figure 3 at the 3- and 6-month time points; note that Figure 3 shows CO data only; abstinence defined here is per the point prevalence definition noted above).
DISCUSSION
The present study suggests that Internet-based contingency management can promote higher rates of negative CO samples during treatment relative to a group in which vouchers are delivered regardless of CO levels (Stoops et al., 2009). Of all the CO samples, 67% were negative for smoking in the contingent CO group and 25% were negative in the noncontingent CO group. These differences were not related to between-groups differences in adherence to the video-based CO sampling procedure: Both groups provided equivalently high rates of CO sub-missions via the Mōtiv8 system. The main procedural difference between groups was the contingency for abstinence; both groups received nearly identical instructions, goals, and feedback based on smoking status.
Although there were differences in the percentages of negative COs between groups during treatment, there were no significant differences in sustained abstinence during treatment (Figure 3) and no significant differences between groups in absolute CO levels. Nevertheless, CO levels did decrease substantially in both groups relative to baseline. Figure 2 shows that following a baseline condition (CO monitoring) in which virtually all participants continued to smoke, a meaningful number of participants in both groups were able to achieve some sustained period of abstinence during treatment. These results suggest that factors other than the contingencies for abstinence influenced smoking. We can only speculate that this behavior change was due to frequent monitoring (twice daily), goal-based instructions for reducing CO, automated and immediate feedback, graphed progress (see Figure 1), implicit social contingencies for abstinence, or some combination of these factors. Other studies that have used noncontingent comparison groups did not include most of these factors (e.g., Higgins, Wong, Badger, Ogden, & Dantona, 2000; Silverman et al., 1998), particularly the frequent monitoring, immediate feedback, and quantitative and graphed goals. Participants in this study also indicated that they found these features to be some of the most helpful in promoting abstinence (reported in Raiff et al., 2013).
Thus, the procedures in the noncontingent CO comparison group most likely entailed a number of active ingredients that promoted behavior change. In addition, adherence to the video submission protocol was high in both groups, which may suggest that monetary consequences, even when they are not contingent on CO, play a role in promoting abstinence. That is, to the extent monetary consequences increased engagement with the self-monitoring, feedback, and goal-setting procedures, they may have also increased abstinence. It is also possible the monetary consequences adventitiously reinforced abstinence (cf. Schroeder, Gupman, Epstein, Umbricht, & Preston, 2003). Overall, the pattern of results suggests that the contingency based on abstinence per se was important, but other factors were also operative in generating negative COs.
The findings that group differences in negative COs were not paralleled by group, mean differences in the absolute CO levels, and point prevalence measures at follow-up are consistent with the Stoops et al. (2009) study. These findings may suggest that frequent monitoring over the Internet is more likely to capture smoking abstinence or to record false negatives. It would have been desirable to characterize smoking status more fully using other biochemical measures, such as salivary cotinine, throughout the study. Future researchers could use Internet-based methods to obtain salivary cotinine results. The process to obtain salivary cotinine may become more rapid and affordable as new nanomaterial electrochemical biosensors are developed (e.g., Nian et al., 2012). The use of CO and cotinine could be combined such that CO could be used early in treatment to initiate abstinence, and then salivary cotinine could be used during maintenance, as Higgins and colleagues have done with their in-person contingency management procedure with pregnant smokers (e.g., Higgins et al., 2004). A combined procedure would optimize the chances of detecting and reinforcing early abstinence, which cotinine cannot do given its much longer half-life (about 20 hr) compared to CO.
Higher rates of delay discounting (i.e., AUC) were marginally associated with smoking status during treatment. Although there is considerable interest in the relation between discounting and smoking status (e.g., Dallery & Raiff, 2007; Koffarnus, Jarmolowicz, Mueller, & Bickel, 2013; Yoon et al., 2007), and whether discounting could be used to tailor treatment (Stanger, Budney, & Bickel, 2012), the relatively weak association between the two in the present study suggests that delay discounting may have limited utility, at least under some conditions. In contrast, a measure of nicotine dependence, the FTND, was significantly associated with CO levels during the study (Dallery et al., 2007; Kozlowski, Porter, Orleans, & Pope, 1994). Whether, and how, measures of nicotine dependence could be used to tailor contingency management, or incorporate additional interventions (Gray et al., 2011), awaits further research.
The present study included a brief tapering, or shaping, period before the abstinence induction condition. Several studies have used similar procedures to initiate abstinence gradually in hard-to-treat smokers (Lamb, Kirby, Morral, Galbicka, & Iguchi, 2010; Lamb, Morral, Kirby, Iguchi, & Galbicka, 2004) and heavy smokers (Dallery et al., 2007). However, it is unclear whether shaping improves cessation outcomes relative to an abrupt transition to abstinence. Research with cocaine-dependent individuals that used a shaping procedure found no meaningful differences in abstinence between participants who were exposed to shaping and those who were not exposed (Correia, Sigmon, Silverman, Bigelow, & Stitzer, 2005). In the context of smoking cessation, one study suggests that shaping may enhance outcomes, relative to a more abrupt transition to abstinence, in hard-to-treat smokers (Lamb et al., 2010). It is plausible that the more frequent assessments of smoking relative to cocaine use (daily rather than three times per week) enhanced the effects of shaping with this population. A CO-based measurement procedure can detect relatively small changes in smoking, and such changes can be reinforced shortly after they occur. The Correia et al. (2005) and Lamb et al. (2010) studies differed in several other respects, and thus the differences in success could be the result of other factors such as the duration of the shaping period or the particulars of the shaping schedule employed, to name a few.
The lack of group differences in point prevalence at follow-up may not be surprising given that absolute CO levels and durations of sustained abstinence did not significantly differ between groups during treatment. Several factors with respect to enhancing and maintaining treatment gains should be considered. First, the duration of the current treatment period, at 7 weeks, was relatively short. This duration was shorter than most contingency management interventions that have targeted other drugs of abuse (Higgins et al., 2000; Silverman, Higgins, Brooner,&Montoya, 1996) and was shorter than many other psychosocial and pharmacological interventions that have targeted cigarette smoking (Fiore, 2000). Although treatment duration does not appear to be related to effects on abstinence while contingency management is ongoing (Lussier, Heil, Mongeon, Badger, & Higgins, 2006), it is possible that longer durations may translate into greater long-term maintenance. Second, the magnitude of voucher earnings may have been too low to generate high enough sustained rates of abstinence during treatment. Higher earnings have been associated with greater long-term success rates (Petry & Roll, 2011). Third, aside from the brief counseling sessions and the 3-week thinning period during which vouchers were available twice weekly for abstinence, the intervention did not include explicit procedures to maintain abstinence and prevent relapse after the intervention was removed. Future research should investigate the potential for intermittent schedules to maintain abstinence for long durations (e. g., several months). It might be desirable to include a random schedule of CO monitoring to prevent the possibility of timing smoking and then providing a CO measure after some period of abstinence, which might be more likely with a predictable schedule of CO monitoring such as the one used in the present study.
Additional methods to enhance treatment gains should be considered, such as combining Internet-based contingency management with pharmacotherapy and cognitive behavioral therapy (K. M. Carroll & Rounsaville, 2007; Cavallo et al., 2007; Fiore, 2000; Krishnan-Sarin et al., 2006). In addition, Meredith, Grabinski, and Dallery (2011) developed a group contingency model in which small groups of smokers worked together as a team to initiate abstinence and earn vouchers. Communication between members was enabled via a discussion board integrated into the Mōtiv8 architecture. The group contingency model could be sustained after the vouchers are withdrawn, thereby permitting a low-cost method to maintain social contingencies for abstinence.
The present study adds to the growing literature on Internet-based methods to promote smoking cessation. Although disparities still exist in access to Internet-based health care, there is some evidence that the access gap can be reduced through Internet-based methods (McDaniel & Stratton, 2006). Video-enabled mobile phones could also be employed in future versions of the intervention to further narrow the access gap (Hertzberg et al., in press). Furthermore, with the advent of lower cost methods to detect smoking (Schepis et al., 2008) and novel methods to fund contingency management interventions (Amass & Kamien, 2008; Dallery et al., 2008), Internet-based contingency management may become increasingly practical and scalable. Regardless of the particular method, information technology will continue to mitigate barriers to evidence-based behavioral treatments to promote smoking cessation and other healthy lifestyle choices.
Supplementary Material
Acknowledgments
We thank Dan Neal for statistical analysis; Matthew Locey for programming the delay-discounting procedure; Alana Rojewski, Steven E. Meredith, Rachel N. Cassidy, and Philip J. Erb for data collection and participant management; and Zaday Sanchez and Patrick Kurdila for data entry. The research and preparation of this paper were supported by National Institutes of Health Grants P30DA029926 and R01DA019580.
Footnotes
SUPPORTING INFORMATION
Additional supporting information may be found in the online version of this article at the publisher’s web-site.
Appendix S1 The logic of the computer algorithm to assess delay discounting.
Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.
Contributor Information
Jesse Dallery, University of Florida.
Bethany R. Raiff, Rowan University.
Michael J. Grabinski, Red 5 Group, LLC, New York
REFERENCES
- Amass L, Kamien JB. Funding contingency management in community treatment clinics: Use of community donations and clinic rebates. In: Heil SH, editor. Contingency management in substance abuse treatment. New York, NY: Guilford Press; 2008. pp. 280–297. [Google Scholar]
- Bickel WK, Jones BA, Landes RD, Christensen DR, Jackson L, Mancino M. Hypothetical intertemporal choice and real economic behavior: Delay discounting predicts voucher redemptions during contingency-management procedures. Experimental and Clinical Psychopharmacology. 2010;18:546–552. doi: 10.1037/a0021739. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Borrelli B, Spring B, Niaura R, Hitsman B, Papandonatos G. Influences of gender and weight gain on short-term relapse to smoking in a cessation trial. Journal of Consulting and Clinical Psychology. 2001;69:511–515. doi: 10.1037//0022-006x.69.3.511. [DOI] [PubMed] [Google Scholar]
- Boyer EW, Smelson D, Fletcher R, Ziedonis D, Picard RW. Wireless technologies, ubiquitous computing and mobile health: Application to drug abuse treatment and compliance with HIV therapies. Journal of Medical Toxicology: Official Journal of the American College of Medical Toxicology. 2010;6:212–216. doi: 10.1007/s13181-010-0080-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Carroll KM, Rounsaville BJ. A perfect platform: Combining contingency management with medications for drug abuse. The American Journal of Drug and Alcohol Abuse. 2007;33:343–365. doi: 10.1080/00952990701301319. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Carroll ME, Anker JJ, Mach JL, Newman JL, Perry JL. Delay discounting as a predictor of drug abuse. In: Bickel WK, editor. Impulsivity: The behavioral and neurological science of discounting. Washington, DC: American Psychological Association; 2010. pp. 243–271. [Google Scholar]
- Cavallo DA, Cooney JL, Duhig AM, Smith AE, Liss TB, McFetridge AK, Krishnan-Sarin S. Combining cognitive behavioral therapy with contingency management for smoking cessation in adolescent smokers: A preliminary comparison of two different CBT formats. The American Journal on Addictions/American Academy of Psychiatrists in Alcoholism and Addictions. 2007;16:468–474. doi: 10.1080/10550490701641173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cobb NK, Graham AL, Byron MJ, Niaura RS, Abrams DB. Online social networks and smoking cessation: A scientific research agenda. Journal of Medical Internet Research. 2011;13:279–293. doi: 10.2196/jmir.1911. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Correia CJ, Sigmon SC, Silverman K, Bigelow G, Stitzer ML. A comparison of voucher-delivery schedules for the initiation of cocaine abstinence. Experimental and Clinical Psychopharmacology. 2005;13:253–258. doi: 10.1037/1064-1297.13.3.253. [DOI] [PubMed] [Google Scholar]
- Crowley TJ, MacDonald MJ, Zerbe GO, Petty TL. Reinforcing breath carbon monoxide reductions in chronic obstructive pulmonary disease. Drug and Alcohol Dependence. 1991;29:47–62. doi: 10.1016/0376-8716(91)90021-p. [DOI] [PubMed] [Google Scholar]
- Dallery J, Glenn IM. Effects of an Internet-based voucher reinforcement program for smoking abstinence: A feasibility study. Journal of AppliedBehavior Analysis. 2005;38:349–357. doi: 10.1901/jaba.2005.150-04. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dallery J, Glenn IM, Raiff BR. An Internet-based abstinence reinforcement treatment for cigarette smoking. Drug and Alcohol Dependence. 2007;86:230–238. doi: 10.1016/j.drugalcdep.2006.06.013. [DOI] [PubMed] [Google Scholar]
- Dallery J, Meredith S, Glenn IM. A deposit contract method to deliver abstinence reinforcement for cigarette smoking. Journal of Applied Behavior Analysis. 2008;41:609–615. doi: 10.1901/jaba.2008.41-609. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dallery J, Raiff BR. Delay discounting predicts cigarette smoking in a laboratory model of abstinence reinforcement. Psychopharmacology. 2007;190:485–496. doi: 10.1007/s00213-006-0627-5. [DOI] [PubMed] [Google Scholar]
- Dallery J, Raiff BR. Contingency management in the 21st century: Technological innovations to promote smoking cessation. Substance Use & Misuse. 2011;46:10–22. doi: 10.3109/10826084.2011.521067. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fagerstrom K, Schneider NG. Measuring nicotine dependence: A review of the Fagerstrom Tolerance Questionnaire. Journal of Behavioral Medicine. 1989;12:159–182. doi: 10.1007/BF00846549. [DOI] [PubMed] [Google Scholar]
- Fiore MC. A clinical practice guideline for treating tobacco use and dependence: A US public health service report. JAMA: Journal of the American Medical Association. 2000;283:3244–3254. [PubMed] [Google Scholar]
- Gainsbury S, Blaszczynski A. A systematic review of Internet-based therapy for the treatment of addictions. Clinical Psychology Review. 2011;31:490–498. doi: 10.1016/j.cpr.2010.11.007. [DOI] [PubMed] [Google Scholar]
- Glenn IM, Dallery J. Effects of Internet-based voucher reinforcement and a transdermal nicotine patch on cigarette smoking. Journal of Applied Behavior Analysis. 2007;40:1–13. doi: 10.1901/jaba.2007.40-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Goodwin MS, Velicer WF, Intille SS. Telemetric monitoring in the behavior sciences. Behavior Research Methods. 2008;40:328–341. doi: 10.3758/brm.40.1.328. [DOI] [PubMed] [Google Scholar]
- Gourlay SG, Forbes A, Marriner T, Pethica D, McNeil JJ. Prospective study of factors predicting outcome of transdermal nicotine treatment in smoking cessation. BMJ (Clinical Research Ed.) 1994;309:842–846. doi: 10.1136/bmj.309.6958.842. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gray KM, Carpenter MJ, Baker NL, Hartwell KJ, Lewis AL, Hiott DW, Upadhyaya HP. Bupropion SR and contingency management for adolescent smoking cessation. Journal of Substance Abuse Treatment. 2011;40:77–86. doi: 10.1016/j.jsat.2010.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hertzberg JS, Carpenter VL, Kirby AC, Calhoun PS, Moore SD, Dennis MF, Beckham JC. Mobile contingency management as an adjunc-tive smoking cessation treatment for smokers with posttraumatic stress disorder. Nicotine and Tobacco Research. in press doi: 10.1093/ntr/ntt060. Epub ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Higgins ST, Heil SH, Solomon LJ, Bernstein IM, Lussier JP, Abel RL, Badger GJ. A pilot study on voucher-based incentives to promote abstinence from cigarette smoking during pregnancy and postpartum. Nicotine and Tobacco Research. 2004;6:1015–1020. doi: 10.1080/14622200412331324910. [DOI] [PubMed] [Google Scholar]
- Higgins ST, Wong CJ, Badger GJ, Ogden DEH, Dantona RL. Contingent reinforcement increases cocaine abstinence during outpatient treatment and 1 year of follow-up. Journal of Consulting and Clinical Psychology. 2000;68:64–72. doi: 10.1037//0022-006x.68.1.64. [DOI] [PubMed] [Google Scholar]
- Japuntich SJ, Zehner ME, Smith SS, Jorenby DE, Valdez JA, Fiore MC, Gustafson DH. Smoking cessation via the Internet: A randomized clinical trial of an Internet intervention as adjuvant treatment in a smoking cessation intervention. Nicotine & Tobacco Research. 2006;8:S59–S67. doi: 10.1080/14622200601047900. [DOI] [PubMed] [Google Scholar]
- Javors MA, Hatch JP, Lamb RJ. Cut-off levels for breath carbon monoxide as a marker for cigarette smoking. Addiction. 2005;100:159–167. doi: 10.1111/j.1360-0443.2004.00957.x. [DOI] [PubMed] [Google Scholar]
- Johnson MW, Bickel WK. Within-subject comparison of real and hypothetical money rewards in delay discounting. Journal of the Experimental Analysis of Behavior. 2002;77:129–146. doi: 10.1901/jeab.2002.77-129. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kenford SL, Fiore MC, Jorenby DE, Smith SS, Wetter D, Baker TB. Predicting smoking cessation—who will quit with and without the nicotine patch. Journal of the American Medical Association. 1994;271:589–594. doi: 10.1001/jama.271.8.589. [DOI] [PubMed] [Google Scholar]
- Koffarnus MN, Jarmolowicz DP, Mueller ET, Bickel WK. Changing delay discounting in light of the competing neurobehavioral decisions theory: A review. Journal of the Experimental Analysis of Behavior. 2013;99:32–57. doi: 10.1002/jeab.2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kozlowski LT, Porter CQ, Orleans CT, Pope MA, Heatherton T. Predicting smoking cessation with self-reported measures of nicotine dependence: FTQ, FTND, and HSI. Drug and Alcohol Dependence. 1994;34:211–216. doi: 10.1016/0376-8716(94)90158-9. [DOI] [PubMed] [Google Scholar]
- Krishnan-Sarin S, Duhig AM, McKee SA, McMahon TJ, Liss T, McFetridge A, Cavallo DA. Contingency management for smoking cessation in adolescent smokers. Experimental and Clinical Psychopharma-cology. 2006;14:306–310. doi: 10.1037/1064-1297.14.3.306. [DOI] [PubMed] [Google Scholar]
- Lamb RJ, Kirby KC, Morral AR, Galbicka G, Iguchi MY. Shaping smoking cessation in hard-to-treat smokers. Journal of Consulting and Clinical Psychology. 2010;78:62–71. doi: 10.1037/a0018323. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lamb RJ, Morral AR, Kirby KC, Iguchi MY, Galbicka G. Shaping smoking cessation using percentile schedules. Drug and Alcohol Dependence. 2004;76:247–259. doi: 10.1016/j.drugalcdep.2004.05.008. [DOI] [PubMed] [Google Scholar]
- Lussier JP, Heil SH, Mongeon JA, Badger GJ, Higgins ST. A meta-analysis of voucher-based reinforcement therapy for substance use disorders. Addiction. 2006;101:192–203. doi: 10.1111/j.1360-0443.2006.01311.x. [DOI] [PubMed] [Google Scholar]
- Marsch LA, Dallery J. Advances in the psychosocial treatment of addiction: The role of technology in the delivery of evidence-based psychoso-cial treatment. Psychiatric Clinics of North America. 2012;35:481–493. doi: 10.1016/j.psc.2012.03.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- McDaniel AM, Stratton RM. Internet-based smoking cessation initiatives: Availability, varieties, and likely effects on outcomes. Disease Management & Health Outcomes. 2006;14:275–285. [Google Scholar]
- Meredith SE, Grabinski MJ, Dallery J. Internet-based group contingency management to promote abstinence from cigarette smoking: A feasibility study. Drug and Alcohol Dependence. 2011;118:23–30. doi: 10.1016/j.drugalcdep.2011.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Myerson J, Green L, Warusawitharana M. Area under the curve as a measure of discounting. Journal of the Experimental Analysis of Behavior. 2001;76:235–243. doi: 10.1901/jeab.2001.76-235. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nian H, Wang J, Wu H, Lo JG, Chiu KH, Pounds JG, Lin Y. Electrochemical immunoassay of cotinine in serum based on nanoparticle probe and immunochromatographic strip. Analytica Chimica Acta. 2012;713:50–55. doi: 10.1016/j.aca.2011.11.028. [DOI] [PubMed] [Google Scholar]
- Petry NM, Roll JM. Amount of earnings during prize contingency management treatment is associated with posttreatment abstinence outcomes. Experimental and Clinical Psychopharmacology. 2011;19:445–450. doi: 10.1037/a0024261. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Prochaska JO, DiClemente CC. Stages and processes of self-change of smoking: Toward an integrative model of change. Journal of Consulting and Clinical Psychology. 1983;51:390–395. doi: 10.1037//0022-006x.51.3.390. [DOI] [PubMed] [Google Scholar]
- Raiff BR, Jarvis BP, Turturici M, Dallery J. Acceptability of an Internet-based contingency management intervention for smoking cessation: Views of smokers, nonsmokers, and healthcare professionals. Experimental and Clinical Psychopharmacology. 2013;21:204–213. doi: 10.1037/a0032451. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Reynolds B, Dallery J, Shroff P, Patak M, Leraas K. A web-based contingency management program with adolescent smokers. Journal of Applied Behavior Analysis. 2008;41:597–601. doi: 10.1901/jaba.2008.41-597. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schepis TS, Duhig AM, Liss T, McFetridge A, Wu R, Cavallo DA, Krishnan-Sarin S. Contingency management for smoking cessation: Enhancing feasibility through use of immunoassay test strips measuring cotinine. Nicotine & Tobacco Research. 2008;10:1495–1501. doi: 10.1080/14622200802323209. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schroeder JR, Gupman AE, Epstein DH, Umbricht A, Preston KL. Do noncontingent vouchers increase drug use? Experimental and Clinical Psychopharmacology. 2003;11:195–201. doi: 10.1037/1064-1297.11.3.195. [DOI] [PubMed] [Google Scholar]
- Seidman DF, Westmaas JL, Goldband S, Rabius V, Katkin ES, Pike KJ, Sloan RP. Randomized controlled trial of an interactive internet smoking cessation program with long-term follow-up. Annals of Behavioral Medicine. 2010;39:48–60. doi: 10.1007/s12160-010-9167-7. [DOI] [PubMed] [Google Scholar]
- Sigmon SC, Lamb RJ, Dallery J. Tobacco. In: Heil SH, editor. Contingency management in substance abuse treatment. New York, NY: Guilford Press; 2008. pp. 99–119. [Google Scholar]
- Signorini DF, Leung O, Simes RJ, Beller E, Gebski VJ, Callaghan T. Dynamic balanced randomization for clinical trials. Statistics in Medicine. 1993;12:2343–2350. doi: 10.1002/sim.4780122410. [DOI] [PubMed] [Google Scholar]
- Silverman K, Chutuape MA, Bigelow GE, Stitzer ML. Voucher-based reinforcement of cocaine abstinence in treatment-resistant methadone patients: Effects of reinforcement magnitude. Psychopharmacology. 1999;146:128–138. doi: 10.1007/s002130051098. [DOI] [PubMed] [Google Scholar]
- Silverman K, Higgins ST, Brooner RK, Montoya ID, Cone EJ, Schuster CR, Preston KL. Sustained cocaine abstinence in methadone maintenance patients through voucher-based reinforcement therapy. Archives of General Psychiatry. 1996;53:409–415. doi: 10.1001/archpsyc.1996.01830050045007. [DOI] [PubMed] [Google Scholar]
- Silverman K, Wong CJ, Umbricht-Schneiter A, Montoya ID, Schuster CR, Preston KL. Broad beneficial effects of cocaine abstinence reinforcement among methadone patients. Journal of Consulting and Clinical Psychology. 1998;66:811–824. doi: 10.1037//0022-006x.66.5.811. [DOI] [PubMed] [Google Scholar]
- Stanger C, Budney AJ, Bickel WK. A developmental perspective on neuroeconomic mechanisms of contingency management. Psychology of Addictive Behaviors. 2012;27:403–415. doi: 10.1037/a0028748. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stoops WW, Dallery J, Fields NM, Nuzzo PA, Schoenberg NE, Martin CA, Wong CJ. An Internet-based abstinence reinforcement smoking cessation intervention in rural smokers. Drug and Alcohol Dependence. 2009;105:56–62. doi: 10.1016/j.drugalcdep.2009.06.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Yoon JH, Higgins ST, Bradstreet MP, Badger GJ, Thomas CS. Changes in the relative reinforcing effects of cigarette smoking as a function of initial abstinence. Psychopharmacology. 2009;205:305–318. doi: 10.1007/s00213-009-1541-4. [DOI] [PubMed] [Google Scholar]
- Yoon JH, Higgins ST, Heil SH, Sugarbaker RJ, Thomas CS, Badger GJ. Delay discounting predicts postpartum relapse to cigarette smoking among pregnant women. Experimental and Clinical Psychopharmacology. 2007;15:176–186. doi: 10.1037/1064-1297.15.2.186. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.