Which Method of Assessing Depression and Anxiety Best Predicts Smoking Cessation: Screening Instruments or Self-Reported Conditions?

Abstract

Introduction

Affective disorders and symptoms (ADS) are predictive of lower odds of quitting smoking. However, it is unknown which approach to assessing ADS best predicts cessation. This study compared a battery of ADS screening instruments with a single, self-report question on predicting cessation. Among those who self-reported ADS, we also examined if an additional question regarding whether participants believed the condition(s) might interfere with their ability to quit added predictive utility to the single-item question.

Methods

Participants (N = 2637) enrolled in a randomized controlled trial of web-based smoking treatments completed a battery of five ADS screening instruments and answered a single-item question about having ADS. Those with a positive self-report on the single-item question were also asked about their interference beliefs. The primary outcome was complete-case, self-reported 30-day point prevalence abstinence at 12 months.

Results

Both assessment approaches significantly predicted cessation. Screening positive for ≥ one ADS in the battery was associated with 23% lower odds of quitting than not screening positive for any (p = .023); those with a positive self-report on the single-item had 39% lower odds of quitting than self-reporting no mental health conditions (p < .001). Area under the receiver operating characteristic curve values for the two assessment approaches were similar (p = .136). Adding the interference belief question to the single-item assessment significantly increased the area under the receiver operating characteristic curve value (p = .042).

Conclusions

The single-item question assessing ADS had as much predictive validity, and possibly more, than the battery of screening instruments for identifying participants at risk for failing to quit smoking. Adding a question about interference beliefs significantly increased the predictive utility of the single-item question.

Implications

This is the first study to demonstrate that a single-item question assessing ADS has at least as much predictive validity, and possibly more, than a battery of validated screening instruments for identifying smokers at highest risk for cessation failure. This study also demonstrates adding a question about interference beliefs significantly adds to the predictive utility of a single, self-report question about mental health conditions. Findings from this study can be used to inform decisions regarding how to assess ADS in the context of tobacco treatment settings.

Introduction

It is well established that having an affective disorder or symptoms of an affective disorder can impact whether one smokes and their ability to stop smoking. Epidemiological data demonstrate that individuals with the most common affective disorders, such as depression and anxiety-related disorders, are 1.4–2.4 times more likely to be current smokers than individuals without mental health conditions.^1–4 At the population level, lifetime quit rates for smokers with common affective disorders and symptoms (ADS) are significantly lower (23.2%–39.4%) than quit rates among smokers without ADS (42.5%–49.6%).^1,2 Many treatment studies also demonstrate that smokers with ADS are less likely to quit smoking than their counterparts without ADS.^5–13 As a result, smokers with common ADS may also experience disproportionate rates of tobacco-related deaths and diseases.^4,14,15

Given the association between ADS and smoking cessation, assessing ADS in the context of tobacco treatment programs is important because identifying individuals who are at risk for cessation failure can guide treatment-related decisions. For example, assessing ADS can help ensure that targeted interventions are given to those who need them most. However, there are many ways to assess ADS and it is unknown which assessment method best predicts cessation or whether the higher respondent burden of lengthy symptom assessments is justified by increased predictive validity over brief self-report items that simply ask the presence of common ADS. Guidance on the predictive validity of different approaches to assessing ADS would be invaluable, particularly in settings where length of assessment is an important consideration (eg, hospital settings, primary care, telephone counseling, and digital health interventions).

As mentioned above, there are many ways to assess ADS. Although structured diagnostic interviews (eg, the Structured Clinical Interview for DSM Disorders¹⁶) can identify individuals who meet current and/or lifetime criteria for a wide range of mental health conditions, they require a trained professional to administer and are time-intensive. As a result, they are not feasible to implement in most settings. Two commonly used approaches which are more feasible and have been shown to predict cessation are: (1) brief, validated screening instruments that assess symptoms of one or more common ADS,^{8,10,13,17,18} and (2) a single-item question where smokers self-report having mental health conditions, including depression and anxiety.^19–23 However, to our knowledge, no study has compared the ability of these approaches to predict cessation outcomes. Moreover, in addition to asking smokers to self-report mental health conditions with a single item, many quitlines add an additional question regarding smokers’ beliefs that their mental health conditions may interfere with their ability to quit smoking. Evidence suggests that smokers who believe their mental health will interfere with their ability to quit have disproportionately low quit rates.^20,24 Some quitlines have used this additional question to determine which callers are offered a targeted protocol designed for smokers with mental health conditions.²⁵ However, no study has assessed whether the addition of the interference beliefs question adds predictive validity to the single-item question.

Thus, the primary aim of this study was to assess and compare the predictive validity of two common assessment approaches—a battery of validated screening instruments used to assess five of the most common depression and anxiety-related disorders versus a single-item, self-report of having depression and/or anxiety—to predict smoking cessation. Specifically, using data from a previously published randomized controlled trial (RCT) for smoking cessation,²⁶ we examined how well each assessment approach predicted 12-month cessation rates and whether either approach contributed any unique explanation of variance over the other. We hypothesized that the single-item self-report question would demonstrate at least as much predictive utility as the assessment battery. A secondary aim of this study was, among those who self-reported depression and/or anxiety on the single-item question, to examine if asking whether they believe the condition(s) might interfere with their ability to quit smoking adds predictive utility to the question. We hypothesized that that this follow-up question would significantly add to the predictive utility of the single-item question.

Methods

Participants

Participants were 2637 adult smokers from all 50 states who participated in a prior RCT of two web-delivered smoking interventions (NCT01812278).²⁶ To be eligible for the trial, participants had to meet the following criteria: (1) be ≥18 years old; (2) smoke ≥5 or cigarettes per day for the last year; (3) want to quit smoking within 30 days; (4) have Internet and email access; (5) not be participating in other smoking interventions or treatments; (6) have no prior use of the control intervention, Smokefree.gov; (7) have no previous participation in one of our previous smoking studies; (8) have no other household members participating in the study; (9) be willing to be randomized to treatment, complete three follow-up surveys, and provide contact information; (10) reside in the United States; and (11) be able to read English.

Procedures

A complete description of study methods can be found in the main outcomes paper.²⁶ Briefly, after completing the informed consent and baseline survey, participants were randomized to one of two web-based programs based either on Acceptance and Commitment Therapy (ACT) or standard care (the National Cancer Institute’s Smokefree.gov web site). Participants had access to their assigned web site for 12 months. The primary cessation outcome was assessed at 12 months postrandomization. Participants received a $25 incentive for completing the outcome survey, and an additional $10 incentive if they responded within 24 hours to the emailed survey link.²⁷ The 12-month data retention rate was 88% and did not differ by arm.

Measures

Battery of Validated Screening Instruments for Depression and Anxiety

At baseline, participants completed a battery of validated screening instruments for five of the most common depression and anxiety-related conditions—major depressive disorder, social anxiety disorder, posttraumatic stress disorder (PTSD), generalized anxiety disorder, and panic disorder. All assessments asked participants to report based on their current symptoms (past week − past month, depending on the measure). Instruments included: the Center for Epidemiologic Studies Depression scale (CES-D) (scores ≥16 were classified as screening positive)²⁸; the Mini-Social Phobia Inventory (Mini-SPIN; scores of ≥6 indicated a positive screen)²⁹; the 6-item PTSD Checklist (PCL-6; scores ≥14 indicated a positive screen)³⁰; the 7-Item Generalized Anxiety Disorder scale (GAD-7; scores ≥10 indicated a positive screen)³¹; and the Autonomic Nervous System Questionnaire (ANSQ; reporting ≥one panic attack within the past month that occurred in a situation in which they were not in danger or the center of attention indicated a positive screen).³² For the purposes of this study, we grouped participants by whether they did or did not have a positive screen on at least one of these measures.

Single-Item, Self-Report of Depression and/or Anxiety and Interference Beliefs

Participants also self-reported the presence of mental health conditions using a single item developed by the North American Quitline Consortium³³ (“Do you have any of the following mental health conditions?”). Response options included: anxiety disorder, depression disorder, bipolar disorder, schizophrenia, alcohol abuse, drug abuse, or none of the above. All participants selecting depression and/or anxiety were classified as having ADS; participants who selected “none of the above” were classified as negative for a mental health condition. Participants who self-reported at least one mental health condition on this question were asked a follow-up question developed by the North American Quitline Consortium about their interference beliefs (“If you checked any of the items above, do you believe that the condition(s) may interfere with your ability to quit smoking?” y/n).

Cessation Outcomes

The primary cessation outcome for this study was self-reported 30-day point prevalence abstinence (no smoking, not even a puff in the last 30 days) at 12 months postrandomization analyzed using complete-case methodology. As a secondary cessation outcome, we analyzed 12-month 30-day point prevalence abstinence using an intent-to-treat methodology by imputing participants with missing outcome data as smokers. We did not biochemically confirm abstinence as self-reported outcomes are recommended for large, population-level cessation trials where there is no face-to-face contact, demand characteristics for false reporting are minimal, and biochemical confirmation is not feasible due to large sample size and geographic diversity.^34,35

Statistical Analysis

Preliminary descriptive analyses were conducted to calculate rates of baseline depression/anxiety identified using both assessment methods (battery of validated screening instruments vs. single, self-report item). To evaluate the validity of the single-item question (ie, to ensure it was assessing ADS), we also examined what proportion of participants who self-reported a depression and/or anxiety also screened positive for an affective disorder on the screening battery.

To determine whether the assessment approaches were associated with cessation outcomes at 12 months (ie, had predictive validity), we used adjusted logistic regression models and report odds ratios and 95% confidence intervals. Covariates included treatment arm and factors used in stratified randomization (ie, gender, education, and smoking ≥21 cigarettes/day). Two approaches were used to compare the predictive validity of the assessment approaches. First, we calculated the area under the receiver operating characteristic (ROC) curve (AUC_ROC). AUC_ROC values range from 0.5 (chance-level ability to predict the outcome) to 1 (perfectly predicts outcome)³⁶ and are a commonly used metric for comparing predictive validity. We also estimated a multivariate regression model that included both assessment approaches to assess if they contribute unique explanation of variance over the other. To ensure that multicollinearity was not an issue in this last model, we first reviewed the reviewed variance inflation factors (all variance inflation factors <2) and correlation between the two assessment approaches (r = 0.57).^37–39

We conducted similar analyses to examine whether adding the interference beliefs question (ie, believing their mental health condition(s) might interfere with their ability to quit smoking) improved the predictive utility of the single-item, self-report question. We first examined the prevalence of interference beliefs among smokers who self-reported depression/anxiety as well as the rates of smoking cessation among those who did and did not report these interference beliefs. Adjusted logistic regression models compared the relative likelihood of quitting among both groups (interference beliefs and no interference beliefs among those who self-reported depression/anxiety) in comparison to those who did reported no mental health conditions. To assess the predictive validity of the interference beliefs question, we calculated the AUC_ROC value. Finally, we used a post hoc logistic regression to compare rates of cessation between those with and without interference beliefs. All statistical tests were two-sided, with α = 0.05, and analyses were completed using R version 3.6.1⁴⁰ and R library pROC.⁴¹

Results

Sample Characteristics

A complete description of the overall sample can be found in the main outcomes paper.²⁶ On average, participants were 46.2 years old (standard deviation = 13.4); 79% were female, 73% identified as White, 11% as Black/African American, and 8% as Hispanic. Approximately half (52%) of the participants were employed, 37% were married, and 28% had a high school education or less. Slightly more than half (55%) were classified as having high nicotine dependence (Fagerström Test for Nicotine Dependence ≥6), 33% smoked more than one pack of cigarettes per day, and the majority (80%) had been smoking for more than 10 years.

Rates of Baseline Depression and Anxiety

Using the battery of validated screening instruments, 73.5% (1938/2637) screened positive for at least one ADS and 21.7% (574/2637) screen positive for none. Specifically, 55.7% (1470/2637) of participants screened positive for depression, 52.4% (1383/2637) screened positive for PTSD, 43.4% (1145/2637) screened positive for panic disorder, 34.2% (903/2637) screened positive for generalized anxiety, and 30.2% (797/2637) screened positive for social anxiety. On the single self-report item, 35.7% (942/2637) of participants self-reported having a depression and/or anxiety condition and 59.4% (1566/2637) self-reported having no mental health conditions.

Importantly, a large proportion of participants who self-reported depression/anxiety on the single-item question also screened positive for an ADS in the assessment battery. Of the participants with complete screening information, 93.7% (871/930) of those who self-reported depression and/or anxiety screened positive for at least one ADS; 81.0% (574/709) of participants who self-reported depression also screened positive for depression on the CES-D; and 93.5% (614/658) of participants who self-reported anxiety also screened positive for at least one anxiety disorder.

Assessing Predictive Utility

See Table 1 for rates of smoking cessation and results from multivariate logistic regressions. Upon examining the quit rates associated with each assessment approach, the group with the lowest quit rates (20%) were those who self-reported depression/anxiety on the single-item question. Those who self-reported no mental health conditions and those who did not screen positive for any ADS had the highest quit rates (both 29%). Compared with smokers who did not screen positive for any ADS on the assessment battery, those who screened positive on at least one instrument were 23% less likely to be abstinent at 12 months (p = .023). This suggests that the battery of validated ADS screening instruments is predictive of smoking cessation. Using the single-item question, smokers who self-reported depression and/or anxiety were 39% less likely to quit smoking than those who self-reported no mental health conditions (p < .001). Thus, the single-item approach to assessing ADS is also predictive of smoking cessation.

Table 1.

Rates of Cessation and Predictive Validity for Both Indices of Affective Disorders and Symptoms

	12-Month quit rate	Multivariate logistic regression models
Predictors	% (num/den)	OR (95% CI)	p	AUCROC (95% CI)
Complete-case analyses
Screening battery
No positive (+) screens	29% (153/520)	—	—	—
Any + screen	24% (406/1686)	0.77 (0.62, 0.97)	.023	0.542 (0.52, 0.57)
Single-item, self-report
Negative self-report	29% (393/1370)	—	—	—
+ Depression and/or anxiety	20% (164/830)	0.61 (0.50, 0.75)	<.001	0.572 (0.55, 0.60)
Imputed outcomes
Screening battery
No positive (+) screens	27% (153/574)	—	—	—
Any + screen	21% (406/1938)	0.74 (0.60, 0.92)	.007	0.548 (0.52, 0.58)
Single-item, self-report
Negative self-report	25% (393/1566)	—	—	—
+ Depression and/or anxiety	17% (164/942)	0.62 (0.51, 0.77)	<.001	0.574 (0.56, 0.60)

Covariates in multivariate logistic regression models included: gender, education, smoking more than one pack of cigarettes per day, and treatment arm. AUC_ROC = area under the receiver operating curve; CI = confidence interval; OR = odds ratio. Imputed outcomes refer to the secondary cessation outcome in which we imputed values for participants with missing outcomes data as smokers.

Comparing Predictive Validity

Results from the ROC analyses are also presented in Table 1. The AUC_ROC values were .542 for the battery of validated screening instruments and .572 for the single-item, self-report assessment. Although the AUC_ROC value for the self-report assessment approach was descriptively higher than that of the battery of screening instruments, the values were not significantly different (p = .136). The same pattern of results emerged when we classified participants with missing cessation data as smokers (p = .187). Finally, in the multivariate logistic regression model that included both assessment approaches, only the single-item, self-report assessment remained a significant predictor of cessation (p < .001).

Interference Beliefs Among Those Who Self-Report Depression/Anxiety

Of the participants who self-reported depression and/or anxiety on the single-item question, 32.2% (303/942) indicated believing their mental health condition(s) might interfere with their ability to quit, while 67.8% (639/942) did not. As shown in Table 2, compared with those who self-reported having no mental health conditions, those who self-reported depression/anxiety but did not have interference beliefs were 27% less likely to be abstinent at 12 months (23% vs. 29%; p = .008). Those with interference beliefs were 61% less likely to be abstinent (13% vs. 29%; p < .001).

Table 2.

Rates of Cessation and Predictive Validity When Assessing Interference Beliefs in Addition to Self-Reported Depression/Anxiety

	12-Month quit rate
	% (num/den)	OR (95% CI)	p	AUCROC (95% CI)
Complete-case analyses
Negative self-report	29% (393/1370)	—	—	—
Positive self-report, without interference beliefs	23% (128/562)	0.73 (0.58, 0.92)	.008	0.578 (0.55, 0.61)
Positive self-report, with interference beliefs	13% (36/268)	0.39 (0.26, 0.55)	<.001
Imputed outcomes
Negative self-report	25% (393/1566)	—	—
Positive self-report, without interference beliefs	20% (128/639)	0.74 (0.59, 0.93)	.010	0.579 (0.55, 0.61)
Positive self-report, with interference beliefs	12% (36/303)	0.40 (0.27, 0.57)	<.001

Covariates in multivariate logistic regression models included: gender, education, smoking more than one pack of cigarettes per day, and treatment arm. AUC_ROC = area under the receiver operating curve; CI = confidence interval; OR = odds ratio. Imputed outcomes refer to the secondary cessation outcome in which we imputed values for participants with missing outcomes data as smokers.

The AUC_ROC value for the two-part question that assesses both self-reported depression/anxiety and interference beliefs was .578, which was significantly greater than the AUC_ROC value for the single-item only (p = .042). A similar pattern of results emerged when we classified participants with missing cessation data as smokers, although the difference between the AUC_ROC values was no longer significant (p = .199). Finally, among smokers who self-reported depression/anxiety, we conducted a post hoc analysis to assess whether those with interference beliefs are less likely to quit than those without interference beliefs. Results (not shown) indicated that those with interference beliefs had 47% (95% confidence interval = 0.35, 0.79) lower odds of quitting than those without interference beliefs (p = .002).

Discussion

The primary goal of this study was to compare how well two approaches to assessing ADS (a battery of validated screening instruments and a single, self-report question) predicted 12-month smoking cessation among participants in a large RCT of two smoking interventions. Both approaches to assessing ADS were significantly associated with cessation outcomes. Specifically, smokers who screened positive on at least one instrument in the assessment battery were 23% less likely to quit smoking than those who did not screen positive for any ADS; those who self-reported depression/anxiety on the single-item question had 39% lower odds of quitting relative to those who self-reported not having any mental health conditions. Although the AUC_ROC values for the two assessment approaches did not significantly differ, the AUC_ROC value was descriptively higher for the single-item assessment. Moreover, when included in the same model, the single-item question remained a significant predictor of cessation, but the battery of screening instruments did not. Taken together, the results are in line with our hypotheses and suggest that the single-item, self-report question assessing for ADS has at least as much predictive utility, and possibly more, than the battery of validated screening instruments for identifying those at highest risk for failing to quit smoking.

Our secondary aim was to determine whether asking about interference beliefs (ie, beliefs that one’s mental health might interfere with their ability to quit smoking) added predictive utility to the single-item, self-report question. Results demonstrated that smokers who reporting believing their mental health would be a barrier to cessation were the least likely to quit smoking of all groups. Adding this question significantly increased the predictive validity of the self-report question when the primary, complete-case cessation outcomes were examined. Importantly, this is the first study outside the context of telephone quitlines to demonstrate that among smokers who self-report depression and/or anxiety conditions, those who report interference beliefs are significantly less likely to quit smoking than those who do not. Future research is needed to better understand why this subgroup has disproportionately low quit rates as well as how to best address interference beliefs in the context of smoking interventions.

It is worth noting that the AUC_ROC values observed in this study were relatively small as they ranged from 0.542 to 0.579. Still, this suggests that these measures of ADS performed better than chance at predicting smoking abstinence at 1 year. It is also important to note that AUC_ROC values are typically used to assess the predictive validity of an instrument for detecting an underlying condition; however, these measures were developed to screen for ADS, not to predict smoking behaviors or abstinence. Thus, we would not expect that these measures would have high AUC_ROC values for predicting cessation. Importantly, the purpose of assessing AUC_ROC values in this study was to have a metric by which to compare the predictive validity of the assessment methods rather than to establish that any assessment modality had a high AUC_ROC value. As a point of reference, the AUC_ROC values observed in this study are similar to values in other studies that examined the predictive validity of instruments designed to measure smoking-specific constructs (eg, level of nicotine dependence).^42–46 For example, one study compared the ability of five cigarette dependence measures to predict smoking cessation 1 month after visiting a smoking cessation web site; AUC_ROC values in ranged from .50 to .58.⁴² Another study found that the AUC_ROC values for predicting 6-month cessation rates among quitline callers ranged from 0.62 to 0.63 for assessments of urge frequency and nicotine dependence, respectively.⁴⁴

Despite relatively low AUC_ROC values, results from this study suggest that both approaches to assessing ADS were significantly associated with quitting smoking and are able to predict abstinence better than chance. In light of the results of this study, we suggest that researchers and clinicians consider a few contextual factors—intervention setting, available resources, and proportion of smokers who will be classified as “at-risk” based on the assessment approach—when selecting an ADS assessment approach for smoking interventions and related research, as the best assessment approach will vary based on context. For example, in settings where length of assessment is important (eg, telephone quit lines, hospital settings, and digital health interventions), the single-item question may be best. However, in settings where targeted interventions are available for specific diagnostic categories (eg, interventions designed specifically for smokers with PTSD or elevated depression symptoms), using a battery of one or more diagnostic-specific assessments will be more useful as the single-item approach does not provide detailed information on certain diagnostic categories. It is also important to consider available resources and the different proportion of smokers who will likely be classified as “at-risk” between the two assessment approaches. For example, settings with limited resources to provide targeted intervention protocols may only be able to offer the targeted intervention to those at highest risk for cessation failure. The best approach in this case may be adding a question about interference beliefs to the single-item question because significantly fewer people self-report depression/anxiety than screen positive for ADS, and even fewer reported interference beliefs. Moreover, this latter group had the lowest likelihood of quitting. Alternatively, in settings where brevity of assessment is less important and/or a targeted intervention is easier to implement for a larger proportion of smokers, using a battery of validated screening instruments may be best in order to provide the targeted intervention to a larger group of at-risk smokers.

This study has a few key limitations. First, the battery of validated ADS screening instruments was limited to the measures used in the main trial. It is possible that different assessment batteries comprised of different instruments may have differential predictive validity. However, that we included several commonly used assessments for the most common ADS in our battery suggests that the level of predictive validity found for the current battery may generalize to others. Second, the approaches used to assess ADS in this study (ie, validated screening instruments and a single-item question) are not diagnostic tools, but rather approaches to screening for ADS. Although it is probable that individuals classified as having ADS based on these assessment approaches do have symptoms of one or more affective disorders, it is likely that not all of these individuals would meet full diagnostic criteria for an affective disorder. However, the instruments we used in the assessment battery have sound psychometric properties, and there is evidence that a large proportion (61%–74%) of individuals who self-report affective disorders also meet diagnostic criteria.^47,48 Furthermore, in the context of real-world smoking interventions, the purpose of assessing ADS would not be for diagnostic purposes, but instead to identify individuals who are least likely to quit and might benefit most from a targeted treatment approach or additional support. Third, we were unable to assess the added predictive validity of the interference beliefs question among those who screened positive on at least one measure in the assessment battery, but did not self-report depression/anxiety on the single-item question. Additionally, there was no biochemical verification of smoking abstinence in this study. Biochemically verifying abstinence is often considered unnecessary in population-level intervention studies with no face-to-face contact as doing so is often not feasible and can bias results.^34,35 However, there is evidence that some smokers may be more likely to self-report abstinence despite continued smoking,³⁵ including smokers with ADS.¹⁰ If this is the case, differences in quit rates may be larger between those with and without ADS in this study, which may have implications for the predictive utility of the assessment approaches. Finally, as the present study was conducted as a secondary analysis of data from a larger trial, our conclusions should be considered preliminary. Thus, future research is needed to determine if these results generalize outside of the context of web-based smoking interventions and to test the possibility that the single-item question has more predictive validity than a battery of validated ADS screening instruments as our findings are inconclusive.

Conclusions

In the context of web-based smoking interventions, this secondary analysis is the first study to demonstrate that a single-item question assessing ADS has at least as much predictive validity, and possibly more, than a battery of validated ADS screening instruments for identifying smokers at highest risk for failing to quit smoking. This study also demonstrates adding a question about interference beliefs significantly adds to the predictive utility of a single, self-report question about mental health conditions. Findings from this study can be used to inform decisions regarding how to assess ADS in the context of tobacco treatment settings.

Supplementary Material

A Contributorship Form detailing each author’s specific involvement with this content, as well as any supplementary data, are available online at https://academic.oup.com/ntr.

Funding

This work was supported by a grant from the National Cancer Institute at the National Institutes of Health (R01CA166646 to JBB).

Declaration of Interests

JLH has received research support from Pfizer. JBB has served as a consultant for GlaxoSmithKline and serves on the advisory board for Chrono Therapeutics. None of the other authors have competing interests to disclose.