SMOKE AND MIRRORS: THE OVERNIGHT ABSTINENCE PARADIGM AS AN INDEX OF DISRUPTED COGNITIVE FUNCTION

Abstract

Rationale:

Smoking abstinence is known to cause decrements in cognition, but the effects are small and variable. One way to reduce variance may be to aggregate measures or visits. Although trait-like individual differences in smoking abstinence effects on cognition are theorized to predict relapse, the test-retest reliability (TRR) assumed in trait models has not been evaluated.

Objectives:

The objective of this study was to assess the value of aggregating measures to determine effect sizes (ES) of smoking vs. abstinence on measures of cognition, and to assess the short-term TRR of abstinence effects on cognition.

Methods:

Thirty adult smokers completed the typical overnight abstinence paradigm twice; each visit pair consisted of one smoking visit and one abstinent visit. Measures of attention, working memory, and inhibitory control were obtained in each visit.

Results:

There were small to medium ESs for smoking abstinence on individual cognitive measures during the first abstinence experience (‘visit pair’). Aggregating the measures within the visit pair and across visit pairs additively increased the ES of smoking vs. abstinence. Although TRRs were acceptable between smoking visits and between abstinent visits, TRRs for abstinence effects (smoking vs. abstinent visit differences) on cognition were consistently weak.

Discussion:

The ability of the typical overnight abstinence paradigm to reflect disrupted cognition at the group level can be substantially improved by aggregating across cognitive outcomes and/or multiple study visits. However, the patterns of poor TRR of smoking-abstinence differences in cognition cautions against their use as trait-like markers in studies of relapse or treatment response.

Introduction

Cigarette smoking remains the leading cause of preventable death in the United States, accounting for approximately one of every five deaths each year (Rostron, 2013). Even with the most efficacious treatments (Cahill, Stead, & Lancaster, 2012), most cessation attempts result in relapse within the first few weeks (e.g., Piasecki, 2006). Individual differences in the intensity and duration of withdrawal appear to play a major role in relapse, with more severe withdrawal symptoms predicting return to smoking (e.g., Piasecki et al., 1998, 2003a; Javitz et al., 2012). Consequently, withdrawal has become an important treatment target and withdrawal attenuation may mediate individual differences in treatment response (e.g. Piper et al., 2008).

Though most of the withdrawal literature emphasizes craving, affective and physical components, there is increasing interest in the cognitive aspect of withdrawal (see Ashare et al., 2014). That is, acute abstinence from nicotine may impair cognition, thereby making it more difficult to abstain. Among the many types of cognitive processes (e.g. Heishman et al., 2010), sustained attention, working memory, and inhibitory control may be especially important for maintaining goal-directed behavior in general and cigarette abstinence in particular (e.g. Ashare et al., 2014; de Wit, 2009; Gould, 2010; Kassel, 1997; Richards et al., 2011). In addition, these processes are posited as mediators linking genes and psychiatric disorders to smoking onset, maintenance, and relapse (e.g., Lerman et al., 2009; Wing et al., 2012) and as targets for treatment development (e.g. Ashare & Schmidt, 2014; Sofuoglu, 2010).

Although there is evidence that nicotine abstinence impairs each of these cognitive domains, the findings are inconsistent and the effects tend to be small (see reviews by Ashare et al., 2014; McClernon et al., 2015). The small effects may be due to small samples and ceiling effects on performance. Here, we address the possibility that the effects of abstinence on cognition are not reliable even within subjects. Most studies assess participants only in a single assessment of smoking vs abstinence. Under these conditions, day-to-day variability in sleep, affective state, and stress may contribute to measurement error (e.g., Durmer & Dinges, 2005; Mitchell & Phillips, 2007; Sandi, 2013; Weafer, Baggott & de Wit, 2013). Thus, low reliability may attenuate the observed effect size of abstinence on cognition, resulting in Type II errors (see Online Resource 1; e.g., Cohen, 1989; Shadish et al., 2002).

Online Resource 1.

Basic Demographic and Smoking Characteristics in Recent Studies of the Cognitive Effects of Overnight Abstinence from Smoking

Study	N	Age mean yrs	Sex % Male	CPD min	Abstinence min hours	Treatment-Seeking?
Ashare & Hawk (2012)	56	41	48	10	~10	No
Greenstein & Kassel (2009)	23	30	-	10	12	No
Harrison et al. (2009)	30	32	50	10	5,17	No
Hirshman et al. (2004)	20	23	55	-	24	No
Koznik et al. (2010)	15	31	60	10	24	No
Leventhal et al. (2010)	203	37	50	15	12	No
McClernon et al. (2008)	26	32	46	15	~10	No
Mendrek et al. (2006)	15	35	60	15	13	No
Myers et al. (2008)	28	36	50	15	12	No
Xu et al (2005)	8	35	63	10	14	No
Mean	42	33	54	12	14
Median	25	34	50	10	12

Notes. Age represents mean; CPDmin = minimum self-reported cigarettes smoked per day. The studies cited reflect the recent literature at the time the present study was planned. Study references are provided in the manuscript.

One way to decrease measurement error is to aggregate across multiple assessments (Furr & Bacharach, 2008; Watson, 2004). We evaluated the impact of two types of aggregation, across cognitive outcomes within visit and across visits within individual outcome measures. Participants completed a standard overnight abstinence paradigm (Visit Pair 1, a smoking and an abstinent visit; order counterbalanced) two times (Visit Pair 2). We employed one-week inter-visit intervals, which are both typical of the field and meet recommendations by psychometricians to focus first on short-term reliability (Furr & Bacharach, 2008; Watson, 2004). In each visit, subjects completed several measures of inhibitory control, sustained attention, and working memory to assess the effect of smoking abstinence on cognition. We also examined the abstinence effects across both Visit Pairs combined, compared to only the first pair, and across the different measures, compared to single measures, within a pair.

It is also possible that modest abstinence effects on cognition could reflect heterogeneity across individuals. Indeed, such individual differences are hypothesized to predict relapse within a group of smokers (e.g., Ashare et al., 2014; Heishman et al., 2010) and to explain higher relapse rates in smokers with psychopathology (e.g., McClernon & Kollins, 2008; Wing et al., 2012). Similarly, trait-like variability in abstinence effects on cognition has been posited as an endophenotype mediating gene-smoking relationships (e.g. Lerman et al., 2009), and some researchers recommend using individual differences in withdrawal-related cognitive deficits to tailor treatment (e.g. Ashare & Schmidt, 2014; Sofuoglu et al., 2013). To study individual differences effectively, however, the measures must have good test-retest reliability (e.g., Gottesman and Gould, 2003). However, despite the importance of this psychometric issue for the field (see also Weafer et al., 2013; Wray et al., 2014), it is striking that we could not find any prior empirical data addressing it. Thus, to identify individual differences in the effect of abstinence on cognition, we took into account the stability of the abstinence effect during Visit Pair 1 and Visit Pair 2. We also evaluated the test-retest reliability of abstinence effects on expired-air CO and self-reported craving and withdrawal.

To best inform the literature on cognition and smoking abstinence, the present study reflected typical methods in the area. Our informal summary of the study characteristics (see Online Resource 1) guided many design features of the study including: a) our abstinence manipulation, which compared smoking as usual to acute overnight abstinence (rather than a longer period of abstinence that might be more powerful; e.g., McClernon et al., 2015), b) our focus on non-treatment-seeking smokers (c.f. Patterson et al., 2010; Rhodes et al., 2012); c) our criterion of 10 cigarettes smoked per day (CPD; though heavier smokers may exhibit greater effects of acute abstinence); and d) our use of common, relatively brief cognitive paradigms (rather than focus on maximizing the sensitivity of the tasks to smoking abstinence).

Methods

Participants

Participants were 30 healthy adults who reported smoking at least 10 cigarettes per day and were not trying to quit smoking. Sample size was based on two criteria. First, the median sample size in the literature is approximately 28 (see Online Resource 1), and as noted above we designed the study to be typical of the literature. Second, test-retest reliability was a central aim of the study. Although rs ≥ 0.70 indicate adequate test-retest reliability (Endler et al., 1998; Watson, 2004), our sample size of 30 provided adequate power (.80) to detect test-retest rs ≥0.50.

Participants were recruited from community flyers and internet advertisements. Exclusion criteria included self-reported lifetime schizophrenia or bipolar disorder, current depression, treatment for substance use, or use of antipsychotics, antidepressants, anticonvulsants, anxiolytics, stimulants, or opiates, and uncorrected vision or hearing problems.

Procedures

Following a telephone pre-screen, participants completed an orientation visit during which they provided informed consent (all procedures were approved by the UB HSIRB), completed a measure of nicotine dependence, and were screened for baseline breath carbon monoxide (CO) of at least 10 ppm (Benowitz et al., 2002).

Participants next completed four lab visits, each a week apart from the next. Visit Pair 1 consisted of one smoking as usual (SAU) visit and one abstinent visit (order counterbalanced across participants). Most visits began at 10 a.m. or 1 p.m. and each participant started all visits at the same time of day. For abstinent visits, participants were instructed to refrain from smoking after midnight, and expired-air CO was used to ensure at least 50% reduction in smoking exposure relative to the orientation visit. Next, participants smoked a cigarette ad lib (SAU visits) or had a short break (abstinent visits), after which they completed craving and withdrawal measures. Participants completed half of the cognitive tasks (order counterbalanced across participants), took a short break to smoke or read magazines, and then completed the remaining cognitive tasks. Remuneration was provided at the end of each 2-hour visit.

Measures

Four cognitive tasks yielded seven cognitive outcomes. All tasks were presented on a 46-cm CRT via E-Prime software (PST, Pittsburgh, PA).

Stop Signal Task (SST).

The task began with 200 practice trials to develop a prepotent ‘Go’ response (see Logan and Cowan, 1984). A 500-ms fixation cross preceded presentation of the ‘Go’ stimulus, a white arrow pointing to the left or the right for 1000 ms, which was followed by a 1500-ms blank screen. Participants indicated the direction the arrow pointed using a PST response box. Participants then completed a 32-trial ‘Stop and Go’ practice and 3 64-trial task blocks in which a stop signal (100-ms, 1000-Hz tone) occurred on 25% of trials. In each block, the initial stop signal occurred 250 ms after ‘Go’ stimulus onset. To obtain approximately 50% inhibition on stop trials, we employed the dynamic timing approach of Band et al. (2003): successful inhibition on a stop trial resulted in a 50-ms increase in the stop delay on the next stop trial (making it harder to inhibit), whereas failure to inhibit resulted in a 50-ms decrease in the stop delay on the next stop trial (making it easier to inhibit). Data reduction was consistent with our previous work (e.g., Ashare & Hawk, 2012; Rhodes et al., 2012). Stop Signal Reaction Time (SSRT), the primary dependent measure in the stop task, estimates the speed of inhibitory control by subtracting the Mean Stop Delay (MSD) from the mean RT on go-trials (Mean GoRT). Smaller values indicate better inhibitory control.

The Go practice block also provided an RT-based metric of sustained attention. Reaction times (RT) on a range of tasks are positively skewed due to intermittent long RTs thought to reflect brief/micro lapses in attention (e.g., Acheson & de Wit, 2008; Leth-Steensen et al., 2000; Spencer et al., 2009). The mean RT reflects the degree of skew, whereas the median RT does not. Therefore, we focused on the average deviation from the median RT (MeanRT – MedianRT); larger DevMedian reflects greater lapses in attention.

Identical Pairs - Continuous Performance Task (IP-CPT).

The IP-CPT (Cornblatt & Erlenmeyer-Kimling, 1985; Halperin et. al., 2008) demands sustained attention as a series of 400 4-digit numbers appear briefly (150-ms; 1500-ms inter-stimulus interval). Participants use a key press to indicate the appearance of a stimulus identical to that immediately preceding it (p=.1). Percent hits (target detections) and percent false alarms (responses non-targets) were used as indicators of sustained attention and inhibitory control, respectively.

N-back Task.

Consistent with previous work (e.g., Loughead et al., 2009; Strand et al., 2012) participants were instructed to remember the location of a stimulus, a grey circle ~5 cm in diameter, as it appeared in 8 possible locations around the monitor perimeter of the computer screen. Stimuli appeared for 100 ms, followed by a 2900 ms response window. The n-back task included 3 conditions of varying difficulty levels: the 0-back, the 1-back, and the 2-back presented in ascending order. Participants were instructed to respond on every trial (100 trials per condition), indicating whether the stimulus was a target (30% of stimuli) or non-target (70%) using two buttons on the response box labeled “yes” or “no.”

During the 0-back, participants simply pressed “yes” if the stimulus appeared in the upper left corner of the computer screen and “no” for other locations. During the 1-back, participants were to press “yes” when the stimulus appeared in the same location as the stimulus immediately preceding it (30%) and “no” for any other location. During the 2-back, participants were to press “yes” when the stimulus appeared in same location as the stimulus “two back”; “no” for all other stimuli. Percent correct matches to targets was the primary outcome. (Although performance decreased as a function of load, load did not interact with abstinence. This pattern suggests abstinence disrupted short-term storage of information rather than the ability to mentally manipulate information. Thus, performance was averaged across loads.)

Visuo-Spatial Span Task (VSST).

The VSST, grounded in models of working memory (Baddeley, 1996; 2003; Hockey & Geffen, 2004), presents participants with an array of 10 squares scattered around the screen. During each trial, a sequence of “smiley” faces was presented within these squares at a rate of 1/s. During the Forward Span task (designed to measure short-term storage), participants used a computer mouse to click on the squares in the same order that the smiley faces appeared. During the Backward Span task (designed to measure mental manipulation), participants clicked on the squares in the reverse of the order in which the faces appeared. In each task, trials advanced in difficulty from a two-location sequence to a maximum of an eight-location sequence, with three trials in each difficulty level. The task terminated once a participant missed all three trials in the same difficulty level. For each direction (Forward and Backward), the total number of correct trials was computed. Due to experimenter error, VSST data were missing for two participants.

Cognition Composite.

For each cognitive outcome, the Subject × Visit distribution was z-transformed (positive z-scores represent better performance). The 7 z-scores were then averaged to form a cognition composite for each subject in each Smoke Condition × Visit Pair condition. Cronbach’s αs (0.78 to 0.85) suggested good to excellent internal consistency reliability.

Self-Report Measures.

Self-report measures included widely used measures of nicotine dependence (Fagerstrom Test for Nicotine Dependence, FTND; Heatherton, Kozlowski, Frecker, & Fagerstrom, 1991), withdrawal (short form of the Minnesota Withdrawal Scale (MNWS; Hughes & Hatsukami, 1986), and craving (Questionnaire of Smoking Urges - Brief, QSU-B; Cox, Tiffany, & Christen, 2001). The QSU-B consists of 2 factors (see Cox et al., 2001); Factor 1 reflects desire and intention to smoke, and Factor 2 reflects the belief that smoking will reduce negative affect. The MNWS assesses the degree to which smokers experience depressed or sad mood, sleep problems, irritability, anxiety, difficulty concentrating, restlessness or impatience, and increased appetite or hunger using a 0–4 scale (none, slight, mild, moderate, severe).

Data Analysis

What is the Impact of Smoking Abstinence on Individual Outcomes?

Mean-level abstinence effects on manipulation checks (expired-air CO, QSU Factors 1 and 2, and MNWS) and individual cognitive measures (CPT Percent Hits and False Alarms, SST practice DevMedian RT and SSRT, Spatial Span Forward and Backward trials correct, and n-back percent correct matches) were evaluated in univariate Smoking Condition (Smoking vs. Abstinent) × Visit Pair (1[Visits 1 and 2] vs. 2[Visits 3 and 4]) ANOVAs. Sex and condition order (Smoking 1^st vs. Abstinent 1^st) were included in all models, but are discussed only when they moderate the effect of smoking condition.

Does Aggregation Improve Abstinence Effect Sizes?

We compared effect sizes for individual cognitive outcomes during Visit Pair 1 (reflecting the typical lab study) to effect sizes following two forms of aggregation, aggregating across cognitive outcomes (i.e., the cognitive composite) and aggregating across visit pairs prior to computing abstinence effects.

How Strong is the Test-Retest Reliability of Individual Differences in Abstinence Effects on Cognition?

For each cognitive outcome, we computed the test-retest correlation (and 95% CI) between the abstinence effect difference scores from Visit Pairs 1 and 2.

Results

Participants were 17 male and 13 female adults with a mean age of 40.9 years (SD=14.7, range =19–63). Most participants were Caucasian (63.3%), reported high school or less education (60%), and reported an annual household income below $40,000 (82%). Participants smoked on average 15.9 cigarettes per day (SD = 7.2; range = 10–40) and were, on average, moderately dependent on cigarettes (Mean FTND = 4.1, SD = 2.3, range = 0–10).

What is the Impact of Smoking Abstinence on Individual Outcomes?

Table 2 presents means (SDs) for all manipulation checks and cognitive measures in each Visit Pair × Smoking Condition cell of the design. Not surprisingly, expired-air CO was markedly reduced during abstinence compared to smoking as usual visits [smoking condition F(1,27) = 123.6, p < .001]. Although the mean abstinence effect was somewhat greater during Visit Pair 2 compared to Visit Pair 1 [Visit Pair × Smoke Condition F(1,27) = 8.4, p=.007], the abstinence effect was quite large in both visit pairs [ps<.001] (see Table 1).

Table 2.

Test-Retest Reliability Correlation Coefficients (95% Confidence Interval) for the two Smoking Visits, the two Abstinence Visits, and the two Smoke minus Abstinence Difference Scores.

Dependent Variable	Smoking VP1,2	Abstinence VP1,2	Smoking-Abstinence Difference Score VP1,2
Biochemical Measure
Expired-Air CO (ppm)	.79 (.60 to .89)	.74 (.52 to .87)	.66 (.40 to .82)
Subjective Measures
Craving
QSU-B Factor 1	.76 (.56 to .88)	.65 (.39 to .82)	.40 (.05 to .66)
QSU-B Factor 2	.73 (.51 to .86)	.83 (.68 to .92)	.41 (.04 to .65)
Withdrawal
MNWS	.65 (.39 to .82)	.66 (.40 to .82)	.26 (−.10 to .57)
Cognitive Measures
Sustained Attention
CPT Hits (%)	.42 (.08 to .68)	.76 (.56 to .88)	.30 (−.06 to .59)
RT DevMedian (ms)	.72 (.49 to .86)	.62 (.35 to .80)	.29 (−.07 to .59)
Inhibitory Control
Stop Task SSRT (ms)	.66 (.40 to .82)	.75 (.54 to .87)	−.13 (−.46 to .24)
CPT False Alarms (%)	.44 (.10 to .69)	.47 (.14 to .71)	.36 (.01 to .64)
Working Memory
Forward Span (trials)	.81 (.64 to .91)	.82 (.66 to .91)	−.10 (−.44 to .26)
Backward Span (trials)	.73 (.51 to .86)	.68 (.43 to .83)	−.07 (−.41 to .29)
n-Back Matches (%)	.40 (.05 to .66)	.69 (.44 to .84)	−.34 (−.62 to .02)
Cognition Composite (z)	.87 (.74 to .94)	.91 (.82 to .96)	.13 (−.24 to .46)

Table 1.

Descriptive statistics and effect sizes (Cohen’s d _z’) for abstinence effects on all biochemical, Subjective, and cognitive measures, presented separately for Visit Pairs 1 and 2, as well as effect sizes for the aggregate of Visit Pairs 1 and 2.

	Visit Pair 1			Visit Pair 2			Aggregate
Dependent Variable	Smoking	Abstinent	ES	Smoking	Abstinent	ES	ES
Biochemical Measure
Expired-air CO (ppm)	25.8 (12.3)	6.9 (2.9)	1.73**	22.9 (9.7)	8.3 (3.7)	1.85**	1.94**
Subjective Measures
Craving
QSU-B Factor 1	18.6 (8.2)	24.6 (8.4)	.61**	14.9 (8.0)	21.6 (9.2)	.80**	.83**
QSU-B Factor 2	11.1 (6.5)	15.6 (7.5)	.63**	10.5 (6.4)	13.8 (6.8)	.53**	.70**
Withdrawal
MNWS	5.0 (6.7)	7.1 (6.3)	.35	4.7 (6.5)	5.6 (6.0)	.18	.16
Cognitive Measures
Sustained Attention
CPT Hits (%)	85.3 (12.1)	83.1 (14.2)	.20	87.3 (13.3)	83.0 (12.0)	.30	.32*
RT DevMedian (ms)	18.2 (18.4)	25.1 (22.4)	.43*	20.4 (22.1)	36.1 (36.8)	.50**	.58**
Inhibitory Control
Stop Task SSRT (ms)	205.3 (75.8)	232.3 (91.1)	.43*	188.3 (64.9)	224.0 (105.6)	.45*	.66**
CPT False Alarms (%)	5.9 (13.1)	5.7 (13.3)	.12	3.4 (5.1)	3.3 (4.9)	.07	.12
Working Memory
Forward Span (trials)	11.6 (3.7)	11.9 (4.2)	−.07	11.9 (3.4)	12.2 (3.6)	−.17	−.16
Backward Span (trials)	10.4 (3.0)	10.3 (3.7)	.04	10.8 (3.7)	11.2 (3.4)	−.18	−.07
n-back Matches (%)	85.9 (11.5)	83.7 (12.7)	.24	85.5 (14.9)	82.6 (12.8)	.24	.51**
Cognition Composite (z)	.05 (.65)	−.08 (.71)	.54**	.14 (.65)	−.06 (.75)	.50**	.67**

Note. Values represent mean (standard deviation), except for effect sizes (ESs), which are calculated as Cohen’s d_z’ (M_diff / SD_diff; see Cohen, 1989, p. 52);

^*p≤05

^**p≤.01.

As expected, abstinence increased QSU craving [smoking condition Fs(1,27)= 20.2 and 19.8, ps <.001, for Factor 1 and Factor 2, respectively]. Although craving was generally greater during Visit Pair 1 compared to Visit Pair 2 [Factor 1 means = 21.6 and 18.3; Factor 2 means = 13.6 and 12.3, Fs(1,27)=17.2 and 5.9, ps<.001 and .03], the effect of abstinence was comparable across visit pairs [Visit Pair × Smoking Condition Fs<1.6, ps>.21]. Withdrawal symptoms tended to be greater during abstinent compared to smoking as usual visits [F(1,27)=3.4, p=.08], an effect that did not interact with visit pair [F<1].

Several individual measures of cognition were disrupted during abstinence compared to smoking as usual: during abstinence, participants detected fewer targets on the IP-CPT (Percent Hits), exhibited larger average deviations from their median (DevMedian) RT on the ‘Go’ practice of the stop task and slower inhibitory processing on the stop task (SSRT), and exhibited poorer working memory on the n-back task (percent correct matches) [smoke condition Fs(1,27) = 4.4, 11.6, 11.8, and 8.03, ps=.05, .002, .002, and .01, respectively; Visit Pair × Smoke Condition Fs<3.5, ps>.07]. Abstinence did not significantly affect IP-CPT false alarms or spatial span forward or backward [all Fs<1.7, ps > .20].

As hypothesized, the cognitive composite decreased during abstinence compared to smoking as usual [F(1,27) = 15.2, p<.001; Visit Pair × Smoking Condition F(1,27)= 1.7, p=.20].¹

Does Aggregation Improve Abstinence Effect Sizes?

Table 1 provides the effect sizes (ESs) for abstinence (smoking vs. abstinent) for every outcome, separately for Visit Pairs 1 and 2 and for the aggregate of Visit Pairs 1 and 2. Expired-air CO and both QSU craving factors demonstrated robust abstinence effects that were modestly larger when aggregated across visit pairs compared to Visit Pair 1 alone. MNWS withdrawal exhibited weak abstinence effects in each visit pair and was unusual in that aggregation across visit pairs had no discernable impact.

We were most interested in the impact of abstinence on cognitive outcomes. In Visit Pair 1, which represents the typical paradigm in the field, abstinence effects on individual cognitive outcomes were, on average, small [mean d=0.20, median=0. 20, SD=0.19; Cohen, 1988]. The same was true of individual cognitive outcomes in Visit Pair 2 [mean d=.17, median=0.24, SD=.28]. Aggregation across the two visit pairs resulted in abstinence effects that were, on average, about 40% larger than those observed for Visit Pair 1 alone mean=.28; median=0.32, SD=.32]. For cognitive indices that were significantly disrupted by abstinence (CPT hits, DevMedian RT, SSRT, and n-back percent correct matches), aggregation across visit pairs was particularly beneficial [mean increase = 65.2%, median increase = 56.7%].

Aggregation across cognitive measures into the cognitive composite was also helpful. For each visit pair, the effect of abstinence on the cognitive composite [d=0.54 and 0.50 for Visit Pairs 1 and 2, respectively] was more than twice as large as we observed for the individual indices (see above). Double aggregation – averaging the composite across visit pairs yielded an even more impressive estimate of the impact of abstinence on cognition [d=0.67].

How Strong is the Test-Retest Reliability of Individual Differences in Abstinence Effects?

Table 2 displays test-retest (Visit Pair 1, 2) reliability (TRR) correlations (and confidence intervals), separately for smoking visits, abstinent visits, and, most importantly, the abstinence effect difference scores (smoking visit minus abstinent visit). Expired-air CO exhibited acceptable TRR between smoking visits and the two abstinent visits, and marginally acceptable reliability for the two abstinence effect difference scores. The pattern was similar for QSU craving scores (Factors 1 and 2); however, the TRR of the abstinence effect difference scores were weak (rs~.4). The MNWS exhibited acceptable TRR for smoking visits and for abstinence visits, but the abstinence effect difference score exhibited surprisingly poor TRR.

TRR for individual cognitive outcomes was adequate across the two smoking visits (except IP-CPT hits and false alarms; [mean r = .60; median r = .66]) and the two abstinent visits [mean r = .68; median r = .69]. However, TRR of the critical abstinence effect difference scores was consistently poor across individual cognitive outcomes [mean r = −.04; median = −.07]. Indeed, the 0.70 threshold for adequate TRR was outside the 95% CI for every measure, and the 95% CI included 0 for all measures except the IP-CPT false alarms.²

The cognition composite exhibited excellent TRR between smoking visits and between abstinent visits. However, the critical smoking-abstinent difference score for Visit Pair 1 was virtually unrelated to the smoking-abstinent difference score from Visit Pair 2 [r=.13], and the 95% CI did fell far short of the acceptable range for a trait individual difference measure.

Discussion

There is growing interest in the cognitive component of smoking withdrawal. Abstinence effects on cognition are theorized to be critical in smoking maintenance and relapse in the general population and in understanding the heightened risk for relapse among individuals with psychopathology and/or particular genetic characteristics (e.g. Ashare et al., 2014; Lerman et al., 2009). Despite this surge of interest, the literature on overnight abstinence effects on cognition is equivocal, and effects are often small. The present study is relatively novel in its psychometric approach to the topic, seeking both to evaluate the test-retest reliability of abstinence effects on cognition and to examine the degree to which aggregation across cognitive domains and visits increased the magnitude and reliability of abstinence effects on cognition.

To What Extent Does Smoking Abstinence Impact Cognition?

If we focus on our internal replication of the standard abstinence paradigm – examining individual cognitive outcomes for Visit Pair 1 – the findings are typical of the literature: mixed results with statistically significant abstinence effects for two of seven measures (see Table 1). Although we could focus on the two significant effects and ignore the remaining five non-significant effects, that is hardly good science (e.g., Simmons et al., 2011). If the null findings are due to low power, one viable approach to increasing power would be to markedly increase the sample size to improve the power to detect small effects (e.g., Cohen, 1989; Maxwell et al., 2008).

However, unless we believe that abstinence effects on cognition are truly small, we should also improve our methods of demonstrating and evaluating those effects. One straightforward approach is to increase the duration of abstinence in the typical study; increasing the duration of abstinence from 12 hours to 24 hours or more may result in larger deficits, on average (e.g., McClernon et al., 2015). Moreover, it seems important that the field strive to better model longer-term abstinence, including studies of cognition in people trying to remain abstinent during real quit attempts (e.g., Patterson et al., 2009, 2010).

We should also improve our selection of measures. For measures that exhibit poor reliability within each smoking condition and/or exhibit restricted range, making the measure insensitive to abstinence (e.g., CPT false alarms; see also Weafer et al., 2013), we should either eliminate them from future work or modify the task difficulty or duration to yield more psychometrically sound assessment.

We might also begin to better assess, minimize, or even directly manipulate factors that contribute to variability across visits. For example, abstinence order effects (SSRT in the present study; see also Leventhal et al., 2010) may be reduced by including an initial practice session (e.g., McClernon et al., 2015). Other state factors (fatigue, stress, motivation, affect) and gender may also influence cognitive performance (e.g. Durmer & Dinges, 2005; Mitchell & Phillips, 2007; Sandi, 2013), and statistical evaluation (e.g., for gender/menstrual cycle) or direct experimental control (e.g., of motivation; e.g., Schlienz et al., 2013) of these factors may increase the signal-to-noise ratio in future studies.

The present study points to an additional path forward: aggregation across multiple visits and/or across cognitive outcomes within visits. Conducting extra visits adds cost and complexity. However, two additional visits resulted in a more robust demonstration of the negative effects of abstinence on cognition than did the typical paradigm. As shown in Table 1, aggregating across both visit pairs doubled the number of cognitive measures that were sensitive to the effects of overnight abstinence (from 2 measures in Visit Pair 1 to 4 measures with both visit pairs) and resulted in effects that were at least 1/3 larger than observed during Visit Pair 1 alone (3 of the 4 crossed Cohen’s (1988) threshold (d=0.5) for a medium effect size).

Together these data suggest that if a study is focused on detection of a statistically significant effect, either doubling the sample size or doubling the number of visits per participant will meaningfully increase statistical power. However, if the goal is to demonstrate that abstinence effects on cognition are not always small, increases in the number of visits may yield more convincing evidence than a comparable increase in the number of participants. Of course, increases in sample size and the number of visits may provide an even more convincing estimate.

Finally, the present study demonstrates the advantages of aggregating across cognitive measures within a visit. Even with the problematic outcomes discussed above included in the composite, the cognitive composite exhibited good to excellent internal consistency reliability in each visit. The cognition composite exhibited moderate abstinence effects for each visit pair that were nominally larger than that observed for any single cognitive outcome. Double aggregation – across outcome measures and visits – yielded an abstinence effect that was medium-to-large (d=.67; Cohen, 1988). In future studies, aggregation across cognitive outcomes may both improve the abstinence effect size and markedly reduce the number of statistical tests conducted (and risk of Type I error).

Of course, aggregation across cognitive domains is not appropriate for studies focused on specific cognitive indices. Indeed, we considered aggregating within each domain (attention, inhibitory control, and working memory). However, we decided to pursue the general composite because: a) as stated above, it resulted in fewer statistical tests, and b) in general, aggregating across a larger number of “test items” results in greater improvement in the reliability of the “scale”, particularly when the number of “items” is relatively small. Thus, all else equal, aggregation across 7 “items” is preferable to aggregating across only 2–3 “items”. Of course, this logic could be extended to within-domain composites, where a composite based on 5–7 measures would be preferable to a composite formed from 2–3 measures – and, all else equal, an aggregate of 2–3 would be better than relying on a single outcome.

Although we are certainly not the first to recognize that increasing “test length” decreases measurement error, our study is novel in applying this approach to studying smoking abstinence effects on cognition. Future work may benefit not only from aggregation into composite cognition “scales” but also from item analysis to determine which “items” (cognitive measures) should be retained and from using the Spearman-Brown prophecy formula to predict the benefits of increasing the number of measures that form a composite (see, e.g., Wray et al., 2014).

How Strong is the Test-Retest Reliability of Individual Differences in Abstinence Effects on Cognition?

In contrast to the rather optimistic conclusion about the average impact of abstinence on cognitive performance – at least when data are aggregated – the present data suggest that the test-retest reliability of individual differences in overnight smoking abstinence effects on cognition is virtually non-existent. That is, the degree to which abstinence disrupted any given aspect of cognition in Visit Pair 1 was at best weakly related to how much a person showed the same effect in Visit Pair 2 (see Table 2). The best test-retest correlations were below 0.40, well below the recommended criterion of 0.70 (Endler et al., 1998; Watson, 2004).

It is important to consider the confidence interval around the observed correlation coefficients, particularly in relatively small samples (Watson, 2004). What is striking is even the upper limit of the 95%CIs around our estimates of the test-retest reliability of abstinence effects, fails to reach the 0.70 recommendation for adequate test-retest reliability. This is true even of the cognitive composite, which exhibited excellent test-retest reliability between the two smoking visits and the two abstinent visits (rs=.87 and .91) but essentially no test-retest reliability for the smoking versus abstinence difference score (r=.013; 95% CI = −.24 to .46). Although difference scores are frequently criticized as inherently less reliable than their individual components because the difference score incorporates measurement error from both components, it is equally important to note that difference scores are reliable to the extent that individual differences in true change exist (Rogosa, Brandt, & Zimowski, 1982; Rogosa & Willett, 1983).

Based on the present data, it seems clear that the smoking-abstinence differences derived from the typical abstinence paradigm should not be interpreted as reflecting trait-like individual differences. Although it is possible that the data will be more favorable in particular populations of smokers, the present findings suggest that trait interpretations must be supported by empirical demonstrations of trait-like test-retest reliability. Otherwise, progress in several areas in which withdrawal-related disruption of cognition is assumed to be trait-like, from relapse risk factor (e.g., Heishman et al., 2010) or endophenotype (e.g., Lerman et al., 2009) to treatment-matching variable (e.g., Ashare et al., 2013; Soflouglu et al., 2010), is likely to be slowed substantially.

This conclusion may appear inconsistent with findings in the literature reporting linkages between cognition and short-term abstinence. However, it turns out that most studies in this small literature report relapse associations with cognition assessed either when smoking as usual or during a period of abstinence rather than the difference from smoking to abstinence (c.f., Powell et al., 2010). As Ashare et al. (2014) note “although baseline deficits…predict relapse to smoking…no study that we know of has demonstrated a link between withdrawal-related deficits in attention and relapse” (italicized in Ashare et al., 2014, p. 3). Perhaps the absence of such demonstrations is not surprising in light of the poor test-retest reliability of individual differences in withdrawal-related cognitive effects in the present study.

Importantly, withdrawal-related change in cognitive function may not be the most clinically relevant metric to consider. Perhaps the overall level of cognitive function during abstinence is most critical, rather than how much change there is from smoking as usual (see also Powell et al., 2010). There is a parallel in recent considerations of cigarette craving (e.g. Sayette & Tiffany, 2013); “peak provoked craving” focuses on craving as a whole, regardless of the components that contribute to that craving (e.g., mood, abstinence, cues), as a more ecologically valid and clinically relevant construct. Returning to cognition, “cognitive valleys” may be most relevant to the ongoing self-control necessary to maintain abstinence; as the cognitive underpinnings of self-control slide downward (perhaps due to a combination of abstinence, a bad night’s sleep, and a stressful argument), the likelihood of relapse increases.

In the present study, nearly all of the cognitive indices demonstrated adequate test-retest reliability between abstinence visits (see Table 2), suggesting individual differences in cognitive performance were adequately captured. Indeed, the within-condition test-retest reliabilities compare favorably to the literature on healthy adults using the same or similar cognitive tasks (i.e., test-retest correlations range from .40 to .80; e.g., Hockey & Geffen, 2004; Rhodes et al., 2012; Weafer et al., 2013). Moving forward, studies should clearly specify whether the individual difference focus is on withdrawal-related changes or performance during abstinence; both are interesting clinically but they differ markedly in their conceptual implications and psychometric underpinnings.

It is also noteworthy that the present study revealed similar reliability patterns for subjective measures of craving and withdrawal. The strong within-smoking condition reliabilities in the present work are consistent with prior studies (e.g. Wray et al., 2014; Shiffman, West, & Gilbert, 2004). However, to our knowledge, we are the first to report the test-retest reliability of abstinence effects on these measures. As for cognitive measures, these reliabilities were poor suggesting considerable caution should be taken in considering abstinence effects on popular craving and withdrawal scales as indicators of individual differences.

In summary, the present work was novel in its psychometric approach to evaluating the cognitive and subjective impact of acute smoking abstinence. Future studies should strongly consider aggregation across individual outcomes to form psychometrically stronger composite indices of cognition; doing so may improve effect size estimates and reduce both Type I and Type II error rates. The present test-retest data call into serious question interpretation of variability in abstinence effects across participants as reflecting trait-like individual differences; future endophenotype and treatment selection studies must either develop more reliable indices of withdrawal-related cognitive deficits or focus on clinically relevant elements of cognition that already exhibit satisfactory ‘traitness’ (e.g., “trough cognition” during abstinence).