Use of an Automated Mobile Application to Assess Effects of Nicotine Withdrawal on Verbal Fluency: A Pilot Study

Abstract

Mild-to-moderate impairment in frontally mediated functions such as sustained attention, working memory and inhibition have been found to occur during tobacco withdrawal and may present a barrier to successful cessation. These findings have led to studies evaluating cessation treatments that target nicotine withdrawal related cognitive impairment. The instruments currently used to assess cognitive function provide detailed and specific information but have limitations including being time consuming, cumbersome, anxiety provoking and having poor ecological validity. We examined the feasibility of using a mobile computer application to test verbal fluency (VF) as a quick, easy-to-administer and more ecologically valid method of measuring the effects of short-term smoking abstinence on frontally mediated cognitive functions. Thirty participants completed two assessments – one during ad lib smoking and one after overnight abstinence. At each assessment, semantic and phonemic VF tests were administered using a mobile application and nicotine craving and withdrawal symptom severity was assessed. In repeated assessments, performance on both semantic and phonemic VF tests is expected to improve due to practice effects; however, significant improvements were observed only in semantic (p=0.012) but not phonemic (p=0.154) VF. Additionally, the change between assessments in phonemic (but not semantic) score was significantly associated with withdrawal (p=0.006) and craving (p=0.037) severity measured post-abstinence. This study demonstrates that nicotine withdrawal has differential effects on semantic versus phonemic VF suggesting impairments of working memory, attention, and inhibition. These effects were measured using methods easily employed in large groups of participants, potentially with remote test administration and automated scoring.

Introduction

The health consequences of smoking have been well described with the most recent estimates attributing approximately 480,000 deaths annually in the United States to smoking (USDHHS, 2014). Although therapies are available that significantly increase smoking cessation rates, long-term cessation rates using the best available therapies continue to be under 30% (Fiore et al., 2008). Among the factors thought to contribute to smoking lapse are cognitive changes that have been observed during the nicotine withdrawal period. These include mild-to-moderate impairment in a number of frontally mediated cognitive functions including sustained attention, working memory and response inhibition (al-Adawi & Powell, 1997; Ashare, Falcone, & Lerman, 2014; Dawkins, Powell, West, Powell, & Pickering, 2007; Powell, Dawkins, & Davis, 2002; Wesnes, Edgar, Kezic, Salih, & de Boer, 2013). The reversal of these withdrawal-associated cognitive deficits has been proposed as a potential target for medication development efforts aimed at increasing smoking cessation rates (Ashare et al., 2016; Ashare & Schmidt, 2014).

Instruments for assessing attention, working memory and response inhibition are included in most standard neuropsychological assessment batteries which consist of multiple tests designed to isolate changes in specific cognitive mechanisms. While these instruments provide detailed and specific information regarding the affected cognitive mechanisms, they are time consuming, cumbersome, anxiety-provoking, and tend to have poor ecological validity (Chaytor, Schmitter-Edgecombe, & Burr, 2006; Higginson, Arnett, & Voss, 2000; Nadolne & Stringer, 2001). Additionally, they need to be administered and evaluated by trained individuals in a face-to-face encounter. The addition of such instruments to studies evaluating potential smoking cessation interventions therefore adds substantial participant burden and study cost and precludes the remote or self-administration of these instruments. The use of these instruments is therefore limited primarily to studies specifically evaluating cognition as a primary outcome.

Verbal fluency is a broad measure of cognition with good ecological validity (Burgess, Alderman, Evans, Emslie, & Wilson, 1998; Doesborgh et al., 2002) that can be administered quickly (total testing time of less than 5 minutes) and relatively easily by an individual with limited or no training. This can be accomplished via the use of a computer application designed for a smartphone or tablet (Pakhomov, Marino, Banks, & Bernick, 2015).

Tests of verbal fluency consist of assessing phonemic verbal fluency (PVF) and semantic verbal fluency (SVF) in which the participant is asked to say as many words as she/he can in one minute that either start with a letter of the alphabet or belong to a semantic category, respectively. Word retrieval on both PVF and SVF tests relies on both frontal and temporal cognitive mechanisms; however, PVF relies more heavily on frontal function (working memory, inhibition of inappropriate responses and sustained attention) than SVF, and SVF relies more heavily on temporal function (semantic memory) than PVF. PVF and SVF tests have been widely used in neurological, psychiatric, mental health, and school settings and their construct and ecological validity have been extensively demonstrated (Burgess et al., 1998; Doesborgh et al., 2002; Henry & Crawford, 2004a, 2004b, 2004c, 2005; Henry, Crawford, & Phillips, 2004, 2005; Marino et al., 2012; Spek, Schatorje, Scholte, & van Berckelaer-Onnes, 2009; Witt, Elger, & Helmstaedter, 2013).

In particular, differential performance on SVF vs. PVF tests has been shown to be indicative of various types of cognitive impairments. Specifically, decreased SVF performance with preserved PVF performance is associated with impairment in temporal lobe brain mechanisms (e.g. semantic memory), whereas decreased PVF performance with preserved SVF performance is indicative of impairment in frontal lobe brain mechanisms (e.g., attention, working memory, and inhibition) (Baldo, Schwartz, Wilkins, & Dronkers, 2006; Grogan, Green, Ali, Crinion, & Price, 2009; Henry & Crawford, 2004a; Melrose et al., 2009; Mummery, Patterson, Hodges, & Wise, 1996). There is currently limited information available on whether nicotine withdrawal is associated with changes in verbal fluency. Determining if such effects occur is necessary in order to later assess if changes in these measures are associated with smoking cessation outcomes and if the outcomes improve with interventions that normalize these measures. Several prior studies of nicotine withdrawal effects included verbal fluency measures (al-Adawi & Powell, 1997; Dawkins et al., 2007; Powell et al., 2002; Richardson, Powell, & Curran, 2003; Smolka, Budde, Karow, & Schmidt, 2004); however, these studies relied only on PVF as they focused on assessing executive function and did not include SVF testing. The results with respect to effects of withdrawal on PVF reported in these studies are somewhat inconclusive and need to be investigated further.

The purpose of the present study is to assess the feasibility of using a mobile computer application (i.e., administered on a tablet computer) for measuring cognitive changes associated with nicotine withdrawal by assessing the effect of overnight abstinence from cigarettes on verbal fluency (both PVF and SVF). Based on prior literature showing negative effects of nicotine withdrawal on frontal brain mechanisms, we hypothesized that PVF performance would decline, SVF performance would be preserved and that these changes would be detectable with the application being tested.

Methods

Participants

Participants were recruited from the University of Minnesota and surrounding communities through flyers and advertisements in local print and on-line sources. To be eligible for the study, participants had to be between the ages of 18 and 64 and smoke primarily cigarettes, on average at least 8 cigarettes per day. Only native English speakers were enrolled to minimize variability due to possible effects of using non-native language. Participants using medications that may affect cognition or speech, with unstable medical or psychiatric conditions or having conditions affecting speech and language were excluded from the study. The study was approved by the University of Minnesota Institutional Review Board and written informed consent was obtained from all participants.

Procedure

We assessed participant eligibility initially via a telephone interview. Participants likely to be eligible for the study were invited to the first of two laboratory sessions, at which we obtained written informed consent, and confirmed inclusion and exclusion criteria (listed above in the Participants section). Smoking status was confirmed via an exhaled carbon monoxide (CO) concentration of ≥ 8 parts per million (ppm), whereas all other inclusion / exclusion criteria were based on self-report. In this repeated measures study, we measured withdrawal symptom severity, craving, PVF, and SVF during each of the two laboratory sessions. Participants smoked ad libitum prior to the first laboratory session but were to abstain from smoking and any other tobacco products overnight prior to the second laboratory session. Smoking abstinence was confirmed via an exhaled carbon monoxide (CO) concentration of < 8 ppm at the beginning of the second laboratory visit. One trial of each, PVF (letter F) and SVF (animals), was administered during each lab session with PVF administered first, followed immediately by SVF. The order of administration of PVF and SVF was the same in both lab sessions and was not counterbalanced based on evidence that the order is unlikely to affect the results (Taler, Johns, Young, Sheppard, & Jones, 2013). Craving and withdrawal symptoms were assessed at each lab session prior to verbal fluency testing.

Cognitive Assessment Instruments

Phonemic verbal fluency and semantic verbal fluency tests were administered using a validated computerized tool (Pakhomov et al., 2015). An iPad app was used to provide standardized test instructions (via iPad) and to audio-record the participant’s responses. PVF was assessed by asking participants to name as many words as they can that start with letter “F” during a one minute period. Standard PVF administration instructions were used including instructing participants to avoid different morphological forms of the same word or using proper nouns denoting persons, places, titles, etc. SVF was assessed by asking participants to name as many animals as they can during a one minute period. The same PVF and SVF stimuli (letter “F” and “animals” category) were administered during both the pre- and post-abstinence visits. We decided against using alternate forms of these instruments due to potential differences in their difficulty (Barry, Bates, & Labouvie, 2008).

The audio recordings were automatically uploaded to the System for Automated Speech and Language Analysis (SALSA) (Marek-Spartz, Knoll, Bill, Christie, & Pakhomov, 2014) for subsequent processing consisting of a) manual verbatim transcription, and b) automated analysis based on automatic speech recognition as described in detail elsewhere (Pakhomov et al., 2015). Disjoint subsets of the data were transcribed and scored by three individuals trained to use the SALSA system, transcribe speech verbatim, and score SVF and PVF tests according to established guidelines. All transcriptions and scores were verified and checked for consistency by a single individual (WT) twice, 5 days apart. Manually derived SVF and PVF scores were compared with automatically computed scores to determine the feasibility of using automatically generated scores to assess nicotine withdrawal induced cognitive changes.

Craving and Nicotine Withdrawal Assessment Instruments

Withdrawal symptom severity was measured using the Minnesota Nicotine Withdrawal Scale (MNWS) with severity score calculated by adding the scores of the 7 withdrawal symptom related items (Hughes & Hatsukami, 1986). Craving severity was measured using the brief ten item version of the Questionnaire of Smoking Urges (QSU) with severity score calculated by adding the scores of the 10 items (Cox, Tiffany, & Christen, 2001).

Statistical Analysis

Due to the within-subject design, we used linear mixed-effects models to test for differences between SVF and PVF scores obtained during the pre-abstinence visit to those obtained during the post-abstinence visit. Mixed models included age, sex and level of education of the participants as additional covariates. One-way, absolute agreement, single unit Intra-Class Correlation Coefficient (ICC) testing was used to evaluate agreement between manually and automatically derived SVF and PVF scores. Spearman rank correlation was used to test for association between SVF and PVF scores, and craving and withdrawal symptom severity. Internal consistency of the MNWS (withdrawal) and QSU (craving) instruments was measured using Chronbach’s alpha. All statistical tests were carried out using the R package (R Core Team, 2014)

Results

Study Sample

Thirty participants completed both laboratory sessions and were used in the analysis. Average age for participants was 49.0 years (SD 8.41), 14 participants (47%) were female, 18 participants (60%) self-identified as Caucasian and 7 participants (23%) as African American. Twenty-two participants (73%) had some post-high school education. Participants smoked an average of 14.7 (SD 4.3) cigarettes per day and had been daily smokers for an average of 29.4 (SD 10.8) years. An additional 4 subjects withdrew from the study after the first laboratory session and were not included in the analysis. The mean length of time between laboratory visits was 4.87 days (SD 6.48).

Manual analysis

As expected, withdrawal and craving scores were significantly higher during the visit occurring after overnight abstinence than during the pre-abstinence visit (Table 1). The internal consistency of these measures in our study (Chronbach’s alpha of 0.90 for MNWS and 0.92 for QSU) was similarly high to those published elsewhere (Cox et al., 2001; Etter & Hughes, 2006). Gender, age and level of education were not significantly associated with changes in SVF and PVF scores between lab visits (Table 2). Scores on the SVF were significantly (p = 0.012) higher during the second laboratory session relative to the first (Tables 1 and 2), consistent with the presence of a practice effect. No significant difference in PVF scores was found between labs (p > 0.1); however, the change in PVF score (lab 2 – lab1) was inversely associated with lab 2 withdrawal symptom (rho = −0.49, p = 0.006) and craving (rho = −0.38, p = 0.037) severity indicating that lower scores during lab 2 were associated with greater withdrawal and craving. No significant correlation was found when comparing change in withdrawal and craving scores vs. changes in PVF and SVF scores between lab sessions.

Table 1.

Semantic Verbal Fluency (SVF), Phonemic Verbal Fluency (PVF), Withdrawal and QSU scores at pre-abstinence and post-abstinence, and the difference between the two lab sessions.

		Pre-abstinence (Lab 1)	Post-abstinence (Lab 2)	p-value
SVF (animals)
Score		20.07 (5.60)	21.83 (6.53)	0.012
Errors		0.10 (0.31)	0.13 (0.43)	ns*
Repetitions		0.67 (1.10)	0.77 (1.10)	ns
PVF (letter F)
Score		13.16 (3.65)	14.03 (5.51)	ns
Errors		0.66 (0.88)	0.50 (0.86)	ns
Repetitions		1.10 (1.52)	0.96 (1.13)	ns
Withdrawal		5.13 (6.00)	7.46 (7.54)	0.007
QSU Score		34.63 (15.29)	48.47 (11.31)	< 0.001
Exhaled CO concentration (ppm)#		14.47 (6.71)	4.43 (1.43)	< 0.001

^*p-values > 0.1 reported as ns (not significant)

^#CO concentrations were measured in parts per million (ppm)

Table 2.

Parameters of mixed models used to test for differences between pre- and post-abstinence lab sessions in manually and automatically calculated verbal fluency scores.

	Manual scoring				Automatic scoring
	SVF		PVF		SVF		PVF
	β	t-value (p-value)	β	t-value (p-value)	β	t-value (p-value)	β	t-value (p-value)
Lab 1 vs. 2	1.77*	2.68 (0.012)	0.87	1.46 (0.154)	2.36#	3.37 (0.002)	1.07	1.49 (0.146)
Age	0.09	0.70 (0.490)	0.03	0.24 (0.813)	0.11	0.90 (0.377)	0.20	1.59 (0.124)
Sex	−1.28	−0.57 (0.573)	−3.26	−1.70 (0.101)	−1.47	−6.89 (0.497)	−3.11	−1.46 (0.157)
Level of Education	1.16	0.83 (0.413)	0.96	0.80 (0.429)	0.30	0.23 (0.822)	0.48	0.36 (0.723)

^*indicates significance at p < 0.05

^#indicates significance at p < 0.01

Potential for automation

Manually and automatically derived SVF and PVF scores were in good-to-excellent agreement – ICC of 0.88 (95% CI 0.80 – 0.92) and 0.77 (95% CI 0.64 – 0.86) for the SVF and PVF tests, respectively. The differences in manual and automatic scores were largely attributable to extraneous comments made by participants during testing.

One of the features of using automatic speech recognition (ASR) technology for automated verbal fluency scoring is the availability of probabilities for each word generated by the ASR system. These probabilities represent the confidence with which the ASR system selected the word from possible alternatives for a given portion of the speech signal. In previous work (Pakhomov et al., 2015), we have used this feature to develop a reliability filter that relied on the proportion of words produced by the ASR system with low confidence to identify speech samples that are not suitable for fully automated analysis and may require manual transcription. We applied this strategy to the current study sample and found that, for example, setting aside 30% of the samples for manual transcription improves the ICC to 0.93 for the SVF test and 0.84 for the PVF test on the remaining 70% of the samples (data not shown).

We also experimented with using automated scores (without reliability filtering – all samples were included) to replicate the mixed modeling results obtained with manual scores. The results of modeling after replacing manual SVF and PVF scores with automatically derived scores are presented in Table 2.

Discussion

The results of this pilot study suggest that a novel, easy-to-use mobile application is a feasible approach for measuring differential effects of nicotine withdrawal on semantic and phonemic verbal fluency. Previous work on verbal fluency testing for assessing differential performance on PVF relative to performance on SVF (cited in the Introduction) has found that declines in one with stable performance on the other can be used as markers of impairment in different cognitive mechanisms. In the current study, we found that abstinence from nicotine differentially impacts the practice effects observed in PVF relative to those observed in SVF. Tests of phonemic and semantic verbal fluency have been widely demonstrated to show reliable practice effects in repeated assessments of cognitively normal individuals. Practice effects are present in both phonemic (Bartels, Wegrzyn, Wiedl, Ackermann, & Ehrenreich, 2010; Beglinger et al., 2005; Bruggemans, Van de Vijver, & Huysmans, 1997; Calamia, Markon, & Tranel, 2012; Harrison, Buxton, Husain, & Wise, 2000; Macciocchi, 1990; Ross et al., 2007; Ruff, Light, Parker, & Levin, 1996; Zgaljardic & Benedict, 2001) and semantic fluency (Cooper et al., 2001; Cooper, Lacritz, Weiner, Rosenberg, & Cullum, 2004; Harrison et al., 2000) tests, particularly when these tests are administered over short time intervals. Absence of a practice effect on these tests when such an effect is expected is clinically significant (Lezak, Howieson, & Loring, 2004) and is a marker of cognitive impairment in neurodegenerative disease (Cooper et al., 2004; Duff et al., 2011).

The findings of the present study show a significant improvement between the pre-abstinence and the post-abstinence sessions in the SVF score but not in the PVF score. Due to differences in study designs, procedures and objectives, it is difficult to directly compare these results to other studies in which investigations of nicotine withdrawal effects included verbal fluency assessment. Two other studies most similar to ours (Powell et al., 2002; Smolka et al., 2004) reported presence of a small practice effect after a 12 hour abstinence from smoking on a different version of the PVF test (i.e., they used alternate forms consisting of 3 letters). The finding of no significant practice effect on PVF in our study may have been due to using more stringent criteria for confirming abstinence (resulting in average exhaled CO concentrations during post-abstinence of 4 ppm vs. ~ 9 ppm in the other studies).

A strength of our study is the concurrent evaluation of both SVF and PVF, which allows for observing a differential effect of these two measures. In prior literature, similar differential patterns of performance on these two tests were found in individuals treated with carbamazepine (Smith et al., 1994) and has been associated with fatigue after administration of a cognitively effortful task to patients with multiple sclerosis in contrast to healthy controls that showed a significant improvement (Krupp & Elkins, 2000). The lack of an expected practice effect on PVF may therefore be a clinically significant finding indicative of transient cognitive impairment.

A possible explanation for the differential effects of nicotine withdrawal on PVF but not SVF test performance has to do with the well-documented dissociation between mechanisms that underlie SVF and PVF performance. PVF performance relies predominantly on frontal cognitive mechanisms of working memory, attention, and inhibition, whereas SVF performance involves temporal mechanisms of semantic memory in addition to frontal mechanisms (Baldo et al., 2006; Grogan et al., 2009; Henry & Crawford, 2004a; Melrose et al., 2009; Mummery et al., 1996). Therefore, impairment in working memory, attention and inhibition (as have been reported to occur during nicotine withdrawal (Ashare et al., 2014)), would be expected to affect PVF to a greater extent than SVF. Indeed, whereas prior studies of PVF and SVF tests repeated over a short interval have demonstrated that both phonemic and semantic fluency scores improve by approximately 3 to 4 words on repeated assessments (Beglinger et al., 2005; Bruggemans et al., 1997; Calamia et al., 2012; Cooper et al., 2001; Cooper et al., 2004; Zgaljardic & Benedict, 2001), our results demonstrate that PVF was impacted by withdrawal to a greater extent than SVF.

Impaired cognition has been suggested as a potential target for smoking cessation medications (Ashare & Schmidt, 2014) and medications known to be effective at increasing smoking cessation rates (e.g. nicotine replacement therapy, bupropion, varenicline) have been shown to reverse (at least, in part) nicotine withdrawal related cognitive deficits (Ashare & McKee, 2012; Atzori, Lemmonds, Kotler, Durcan, & Boyle, 2008; Loughead et al., 2010; Myers, Taylor, Moolchan, & Heishman, 2008; Patterson et al., 2009). Testing the effects on cognition of potential smoking cessation medications may therefore be of substantial importance. Developing methods that can test nicotine withdrawal associated cognitive changes quickly, in a standardized way and with minimal training will allow for the broad usage of these tools. The current study demonstrates that use of such methods is feasible. Subsequent studies are needed to confirm that the observed effects on cognition are associated with changes in smoking behavior before these methods can be widely used in smoking cessation studies. Although not directly tested in the current study, the methods used in this study have the added potential advantages of being able to be applied outside of the clinic setting (i.e. via a smartphone or tablet application) and of being scored automatically. Future studies are needed to determine the feasibility of such applications.

This pilot study has several limitations that may affect the interpretation of the results including the relatively small sample size. We did not counterbalance the order of experimental conditions (smoking vs. abstinence), which could potentially introduce confounding between nicotine withdrawal and practice effects. Our findings, however, demonstrate differential performance on SVF and PVF tests. Both of these tests have been shown to exhibit small but reliable practice effects in a number of prior studies and meta-analyses (cited earlier in this discussion), therefore suggesting that the results in the current study are likely due to the effects of nicotine withdrawal on cognition rather than practice effects. Future studies using a non-smoker control group would further help disentangle practice effects from withdrawal. Although there is evidence that the order of administration of PVF and SVF does not affect the results (Taler et al., 2013), future studies may benefit from counterbalancing this order across experimental conditions as well. Another possible limitation is that the present study relied on a single letter in the administration of the PVF tests, whereas typical administration of this test combines scores from responses to three letters (e.g. “A-F-S”, “C-F-L”, etc.). We decided to use a single letter “F” in order to minimize the amount of testing time and based on reports in the literature that results obtained with letter “F” tend to be similar to those obtained with 3-letter combinations (Ruff et al., 1996; Tombaugh, Kozak, & Rees, 1999) and thus is frequently used in clinical settings as a bedside test (Lezak et al., 2004). While this pilot study was not designed to assess concurrent and predictive validity of SVF and PVF testing, these tests have been well established and widely used for assessment of cognitive function in populations affected by neurodegenerative disease, traumatic brain injury, developmental disorders, and drug toxicity (Burgess et al., 1998; Doesborgh et al., 2002; Henry & Crawford, 2004a, 2004b, 2004c, 2005; Henry et al., 2004, 2005; Marino et al., 2012; Spek et al., 2009; Witt et al., 2013). Future studies are needed to assess the concurrent and predictive validity of VF testing in the context of cognitive effects of smoking and nicotine withdrawal.

In summary, this study found that a mobile application that does not require specialized training or expertise to administer was able to capture differential changes in semantic and phonemic verbal fluency scores that are indicative of negative changes in cognition occurring during the nicotine withdrawal period. This study further found that the magnitude of these effects is associated with the magnitude of withdrawal symptoms. Future studies are needed to determine if cognitive deficits as measured by differential performance of verbal fluency tests is predictive of successful smoking cessation and therefore could potentially be a target for smoking cessation interventions, and whether the administration and scoring of these tests can be fully automated.

Public Significance.

Neuropsychological tests are widely used to assess cognitive effects of medical conditions and substances. In this study, we test the use of an easier to administer and lower cost tool for assessing verbal fluency during withdrawal from smoking. Our findings suggest that this tool can be used to measure changes in cognitive function during nicotine withdrawal and therefore justify the need for broader testing in larger populations in a variety of settings.