Leveraging Machine Learning to Predict Quit Attempts Among Adult Cigarette Smokers

Abstract

Introduction

Cigarette smoking is the leading preventable cause of premature death in the US. Quitting can significantly reduce the risk of adverse health effects. However, much is unknown about the factors prospectively associated with the behavioral step of making a quit attempt. The present study leveraged supervised machine learning, data from a prior experimental study, and exploratory methods to investigate longitudinal associations between smoking history, cognitive and behavioral indices, and quit attempts reported at a four-week follow-up assessment.

Aims and Methods

Secondary analysis of a project that enrolled 278 adult cigarette smokers (18–69 years old) for a prospective study of predictors of making a quit attempt(s). Analyses were based on N = 231 with non-missing data through the one-month follow-up, separated into training (n = 185, of whom 36 made a quit attempt) and testing (n =46, of whom 9 made a quit attempt) subsets.

Results

In the testing subset, a random forest model with class weights to address class imbalance and a linear support vector machine model performed best of the models tested, albeit not very well (eg, F₁ values of 0.36 in the testing set). Feature importance metrics suggested that contemplation status and perceived risks of “I will eat more” and “I will miss the taste of cigarettes” were of highest importance in classifications. All models had better specificity than sensitivity. Poor performance in the testing set may have resulted in part from small sample size.

Conclusion

While reliable predictions of quit attempts were not obtained across the train-test split, the present study leverages bottom-up, data-driven approaches, and machine learning methods to explore the understudied behavioral step of initiating a quit attempt. Future studies should continue exploring predictors of quit attempts, along with how they may differ from predictors of sustained smoking cessation. Data-driven machine learning methods have the potential to progress precision medicine efforts by clarifying generalizable mechanisms underlying clinical outcomes.

Introduction

In 2024, almost one-fifth of US adults (18.8%) used tobacco products, of which cigarettes are most common (9.9%).1 Cigarette smoking is the leading preventable cause of premature death in the US, resulting in conditions such as cancer, heart disease, and stroke.2 Quitting smoking can significantly reduce the risk of adverse health effects and premature death.2 Realizing the benefits of smoking cessation for public health requires both improvements in smoking cessation methods and improvements in our efforts to motivate smokers to make quit attempts in the first place. In 2022, only 53.3% of adult cigarette smokers said they took the behavioral step of making a ≥24 hour quit attempt over the past year and only 8.8% were successful.3

There is an abundance of cognitive, behavioral, and pharmaceutical interventions aimed at assisting cigarette smokers to successfully quit,4,5 yet such interventions are ineffective for those who do not make at least one quit attempt. Thus, it is important to understand the factors associated with behaviorally choosing to make a quit attempt, which may differ from predictors of successful cessation among those who do attempt to quit. For example, one large international cross-sectional study reported that quit attempts were best predicted by country-level policy factors (eg, taxation, advertising bans), whereas individual-level factors (eg, presence of other smokers in one’s family; age of initiation of smoking) better predicted success in achieving smoking cessation.6

We analyze longitudinal data from a previously published randomized controlled trial7 that tested an induction aimed at increasing rates of quit attempts among adult cigarette smokers. The induction was not successful at increasing quit attempts. We therefore cast a wide net in conducting an exploratory secondary analysis collapsing across experimental conditions to focus on baseline individual difference characteristics predictive of making a quit attempt. We applied supervised machine learning with two aims: (1) predicting quit attempts, and (2) understanding the variables that were most important in making quit attempt predictions, with the goal of informing future hypothesis-testing research on predictors as well as ultimately the improvement of interventions aimed at increasing quit attempts.

Machine learning, also known as statistical learning, has become a popular analytic method for predicting clinical outcomes, as it can be leveraged to explore complex relationships (eg, multivariate interactions). Supervised machine learning is predictive, in which a model is built using examples with known outcomes to estimate outcomes in new samples (eg, predicting treatment response to medication).8,9 Several studies have used supervised learning to explore smoking behavior (see Bickel et al10 for systematic review). To illustrate this approach, one study explored predictors of successful smoking cessation via group cognitive-behavioral therapy.11 Decision tree classifiers were used to predict treatment response through 6‑month follow-up. Models were evaluated by training in one cohort (n=90), using 10-fold cross-validation, and testing with a second independent cohort (n=71). Classification accuracy in the validation cohort ranged from 48 to 82% depending on predictor model, time of follow-up, and abstinence indicator. Delay discounting, which is the tendency for people to prefer smaller immediate rewards over larger delayed rewards, was the first split for all four decision trees (ie, the best predictor of treatment response).

The present study utilizes machine learning as a secondary data analytic tool for exploring predictors of smoking quit attempts. We use data from research testing a looming vulnerability induction for smoking cessation.7 “Looming vulnerability” is a model that posits individuals become anxious when they perceive threats as growing larger and accelerating toward them in space or time and has been linked to smoking behaviors.12 We aim to build a supervised learning classification model that predicts whether an individual will make a quit attempt, operationally defined as self-reported smoking cessation between the baseline and follow-up assessments. Participants were prompted to report quit attempt periods of ≥24 hours. Logistic Regression, Random Forest (RF), and Support Vector Machine (SVM) models were selected for the binary classification task given their strong and consistent performance across a wide range of samples (eg, Fernandez-Delgado et al13), that they can come with variable importance metrics, and all are relatively intuitive. Moreover, they use very different methods for classification. For example, RF is a non-linear ensemble method that aggregates responses across many decision trees, while SVM is a linear model that uses hyperplanes to separate dimensional space and can be adapted to be non-linear.

Methods

Participants

Participants were 278 adult cigarette smokers (18–69 years old, M=49.79, SD=11.42; 52.2% male) recruited from the community via convenience sampling to participate in a study testing the efficacy of an experimental manipulation aimed at increasing quit attempts.7 Most participants were African American (77.3%) or Caucasian (16.9%) and non-Hispanic (86.0%). Inclusion criteria for the parent trial were: (a) aged >18 years, (b) smoking ≥10 cigarettes per day), and (c) expired air CO >9 ppm at the first assessment. Participants were excluded from the original study if they had (d) a total score >5 on the Modified Mini Screen (MMS),14 (e) a score of >0 on the suicidality item, (f) lacking fluency in English, or (g) not planning to live locally during the study.

Procedures

The study methods complied with the Declaration of Helsinki, and the project was approved by the Institutional Review Board of American University (Project #09001) and preregistered on ClinicalTrials.gov (NCT02522156). Data collection was conducted between January 2016 and April 2019 using Qualtrics and paper forms. Data were stored in anonymized fashion, linked to a participant by code number. At the baseline appointment, each participant provided informed consent on a written paper consent form, which included specifying that their data could be used in future reports of the results based on group (not individual) statistics. The remainder of the baseline appointment included completion of surveys and an expired air CO test. Eligible participants were randomly assigned to either receive an audiotape-guided imagery procedure aimed at inducing smoking cessation attempts or a matched control exercise. A follow-up assessment occurred four weeks later, for which participants returned to complete surveys, retook the CO test, and reported on smoking behaviors (eg, quit attempts, smoking cessation). Participants were compensated for their time (see7 for a detailed description of the study procedures).

Measures

See Supplemental Table 1 for internal consistency metrics.

Demographics and Smoking History

Participants reported their age, gender, race, ethnicity, education, occupation, and income. They also reported their smoking history (including marijuana), such as years smoked, quit attempt history, and past month smoking rate.

Nicotine Dependence at Baseline

The 6-item Fagerström Test of Nicotine Dependence (FTND) self-report measure assessed severity of dependence.15 An example item is, “How soon after you wake up do you smoke your first cigarette?” scored on a 4-point scale (0=After 60 minutes, 3=Within 5 minutes).

Sensation Seeking at Baseline

The 40-item revised Sensation Seeking Scale-V (SSS-V) self-report questionnaire captured sensation seeking.16 Items are dichotomous (0=I prefer quiet parties with good conversation, 1=I like “wild” uninhibited parties) and capture attitudinal and behavioral characteristics.

Contemplation of Quitting at Baseline

Contemplation of quitting was measured with the Contemplation Ladder (CL) and a single-item measure of intent.17 The CL is a picture of a ladder with rungs labeled from 0 to 10. Zero is below the bottom rung (0=No thought of quitting) and 10 is above the top rung (10=Taking action to quit, cutting down, or enrolling in a program). The 8-item Stage of Change (SoC) questionnaire was also assessed.18 Three items were retained for analyses: follow-up logic-branched items were collapsed and items capturing redundant information were dropped. Contemplation status was assessed with: “If you are interested in quitting smoking, when do you think you will next attempt to stop smoking?”

Self-Efficacy at Baseline

The 17-item Smoking Self-Efficacy Questionnaire (SSEQ) measured self-confidence in ability to resist smoking in high-risk situations.19 Responses range on a 3-point scale (1=Yes, I could control my smoking behavior, 2=No I could not control my smoking behavior; 3=Does not apply). If “yes” was selected, a confidence rating (10–100) followed.

Outcome Expectancies at Baseline

The Perceived Risks and Benefits Questionnaire (PRBQ, 40 items),20 Smoking Consequences Questionnaire-Adult (SCQ-A, 55 items),21 and a self-generated outcome (SGO) were used to measure smoking outcome expectancies. The PRBQ and SCQ-A assess the likelihood of possible consequences of smoking, with responses on a 7-point and 10-point scale, respectively. For the SGO, participants were asked to complete the sentence, “When I smoke cigarettes, I expect to…” and rated the likelihood of the outcome on a 7-point scale. Qualitative responses were coded by two independent raters into categories of positive reinforcement, negative consequences, negative reinforcement, neutral action, and other/do not know. Cohen’s kappa reliability ratings for baseline and follow-up were 0.83±0.07 and 0.73±0.08, respectively.

Loomingness of Smoking Health Consequences at Baseline

The Physical Health subscale of the 12-item Cigarette Smoking Consequences Looming Scale (CSCLS-P)22 which assesses perceptions of smoking consequences following exposure to imagined looming threats was retained. Responses range on a 5-point scale (1=Changes are decreasing with each cigarette, 5=Changes are increasing with each cigarette) with higher scores indicating a greater perception of looming smoking health consequences.

Subjective Responses to the Audio-Guided Imagery Exercises at Baseline

Post-induction, participants completed 9 questions assessing general and specific (ie, related to the audio-guided imagery exercises) anxiety on a 10-point scale (1=Not at all, 10=Extremely). This served as a manipulation check and was the only baseline questionnaire to occur following the experimental manipulation.

Visual Imagery at Baseline

The 16-item Vividness of Visual Imagery Questionnaire (VVIQ)23 was used to measure clarity and liveliness of visual imagery. Participants imagined various words (eg, sky, door), with eyes open and with eyes closed, and rated the vividness of their mental representations on a 5-point scale (1=No image at all, 5=Perfectly clear and as vivid as normal vision).

Quit Attempts at Follow-Up

At the follow-up visit, participants self-reported cigarette quit attempts since the baseline study visit: “Have you made any attempt to quit smoking since your last study appointment? Only include attempts that lasted at least 24 hours.” Responses were coded 0=no quit attempt and 1=quit attempt made.

Data Preprocessing

All data cleaning and analyses were conducted in R (version [v] 4.6.1).24 Machine learning models were developed with caret (v7.0.1), glmnet (v5.0), kernlab (v0.9.33), and randomForest (v4.7.1.2).

Participant Exclusions

Of the original 278 participants, 14 were excluded due to missing the outcome measure and 33 were removed for attrition before the follow-up appointments. A total of 231 participants were retained for analyses. Independent Samples T-tests and Chi-Square Tests of Independence for participant characteristics and baseline total scores indicated that participants missing the outcome question did not significantly differ from those who participated in the full study (p<0.05).

Variable Recoding

Baseline item-level information, rather than total scores, was analyzed to explore associations at a granular level. There were initially 314 independent variables. Following data pre-processing steps of collapsing logic-branched questions and removing redundant information and qualitative responses information, we retained 242 independent variables for subsequent feature selection analyses, including variables indicating participant’s experimental condition. For descriptive statistics on quit attempts made by experimental condition, see Haaga et al, 2020. See Supplemental A and Supplemental Figure 1 for further detail on variable-level preprocessing.

Data Splitting

The full sample was randomly split into a training (n=185) and testing (n=46) set, respectively. The split was stratified by quit attempt (training n=36; testing n=9).

Results

Sample Characteristics

Demographics, prior to variable recoding, for the full sample and the data split are outlined in Table 1. Thirty-six training set participants and nine testing set participants reported a quit attempt between the baseline and follow-up appointments. There were no training set differences for quit attempt status by age (t(183)=0.48, p=0.474), gender (p=0.816), race (p=0.146), or ethnicity (p=0.179).

Table 1.

Participant Characteristics

Variable	Full Sample (N=231)			Training Set (n=185)				Testing Set (n=46)
	n	%	M (SD)	n	%	M (SD)	Range	n	%	M (SD)	Range
Age			50.2 (11.2)			50.5 (11.0)	21-68			49.1 (12.1)	18-69
Gender
Male	123	53.2		97	52.4			26	56.5
Female	108	46.8		88	47.6			20	43.5
Race
American Indian or Alaska Native	8	3.5		4	2.2			4	8.7
Black or African American	178	77.1		150	81.1			28	60.9
White	38	16.5		25	13.5			13	28.3
Asian	4	1.7		4	2.2			0	0.0
Native Hawaiian or Other Pacific Islander	1	0.4		0	0.0			1	2.2
Ethnicity
Hispanic	8	3.5		4	2.2			4	8.7
Non-Hispanic	201	87.0		166	89.7			35	76.1
Education
> 7th grade	1	0.4		1	0.5			0	0.0
7th grade to 9th grade	3	1.3		1	0.5			2	4.3
Partial high school	14	6.1		12	6.5			2	4.3
High school diploma or General Equivalency Degree (GED)	80	34.6		64	34.6			16	34.8
Partial college (at least one year)	65	28.1		50	27.0			15	32.6
Technical school or associates degree	24	10.4		23	12.4			1	2.2
College diploma (Bachelor’s degree)	30	13.0		23	12.4			7	15.2
Graduate or professional degree	13	5.6		10	5.4			3	6.5
Job
Full-time (35 hours or more per week) for pay	39	16.9		35	18.9			4	8.7
Full-time student	6	2.6		4	2.2			2	4.3
Part-time for pay	51	22.1		41	22.2			10	21.7
Part-time student	5	2.2		4	2.2			1	2.2
Disabled	55	23.8		39	21.1			16	34.8
Leave of Absence	4	1.7		2	1.1			2	4.3
Unemployed	65	28.1		53	28.6			12	26.1
Retired	25	10.8		17	9.2			8	17.4
Homemaker	8	3.5		7	3.8			1	2.2
Income
Less than $10,000	73	31.6		57	30.8			16	34.8
$10,000-$19,000	51	22.1		41	22.2			10	21.7
$20,000-$29,000	37	16.0		30	16.2			7	15.2
$30,000-$39,000	23	10.0		18	9.7			5	10.9
$40,000-$49,000	16	6.9		14	7.6			2	4.3
$50,000-$74,999	17	7.4		13	7.0			4	8.7
$75,000-$99,000	5	2.2		3	1.6			2	4.3
$100,000-$149,000	5	2.2		5	2.7			0	0.0
$150,000-$199,999	2	0.9		2	1.1			0	0.0
$200,000 or more	1	0.4		1	0.5			0	0.0

Notes: Missing counts excluded. Job responses are not mutually exclusive.

Feature Selection

A data-driven filtering method was implemented to reduce dimensionality, assessing two-tailed bivariate relationships between each questionnaire item and the outcome. Categorical predictors were one-hot-encoded, and if any level of a given variable showed a significant association with the outcome, all levels of that variable were retained. A total of 20 features (with categorical variables counted at the root level, not by individual one-hot-encoded dummy) were retained based on statistical significance across these methods (p < 0.05; see Table 2). All preprocessing and variable selection steps were conducted using only the training set – the testing set was subsequently restricted to the same feature set. Because feature selection was conducted once using the full training set, we assessed the stability of this selection procedure using a stratified bootstrap analysis. See Supplemental C and Supplemental Table 4 for details and results of this analysis. Following feature selection, there was 1.7% missingness in the training set and 1.8% missingness in the testing set.

Table 2.

Retained Features

Questionnaire	Prompt	Scale	Item	Training Set Descriptives
				n	M(SD)	Mode
Smoking History	Number of past serious and deliberate attempts to STOP SMOKING cigarettes completely.	Written responses.	9	185	2.7(4.5)	–
	Have you ever sought treatment for help in quitting smoking?	1 = Yes 2 = No	7	179	–	1
Stage of Change (Contemplation Status question)	If you are interested in quitting smoking, when do you think you will NEXT attempt to stop smoking?	1 = this week 2 = in the next 30 days 3 = in the next 6 months 4 = In the next 12 months 5 = Do not plan to quit within the next 12 months	–	183	–	3
Stage of Change (Quit Attempts)	Have you ever made a serious and deliberate attempt to CUT DOWN on how much you smoke?	1 = Yes 2 = No	2	181	–	2
	Have you ever made a serious and deliberate attempt to STOP SMOKING cigarettes completely?		5	184	–	2
Education Level	What is the highest level of education you completed?	1 = Less than seventh grade 2 = Seventh grade to ninth grade 3 = Partial high school 4 = High school diploma or General Equivalency Degree (GED) 5 = Some college (at least one year) 6 = Technical school or associates degree 7 = College diploma (bachelor’s degree) 8 = Graduate or professional degree	–	184	–	2
PRBQ Rate how likely each item would be if you were to stop smoking.	I will eat more.	1 = No chance 2 = Very unlikely 3 = Unlikely 4 = Moderate chance 5 = Likely 6 = Very likely 7 = Certain to happen	1	182	4.9(1.7)	–
	I will prove I can achieve abstinence from cigarettes.		2	182	5.0(1.5)	–
	I will miss the taste of cigarettes.		14	174	4.5(1.8)	–
	I will no longer offend others by smoking.		17	179	5.7(1.5)	–
	I will get instant health benefits.		18	177	5.2(1.7)	–
	I will desire a cigarette.		23	179	5.1(1.5)	–
	I will feel more energetic.		29	177	5.3(1.3)	–
SCQ-A Rate how LIKELY or UNLIKELY you believe each consequence is for you if you smoked a cigarette right now.	My throat burns after smoking.	0 = Completely 1 = Extremely Unlikely 2 = Very Unlikely 3 = Somewhat Unlikely 4 = A Little Unlikely 5 = A Little Likely 6 = Somewhat Likely 7 = Very Likely 8 = Extremely Likely 9 = Completely Likely	3	184	3.7(2.4)	–
	Cigarettes make my lungs hurt.		14	181	4.9(2.6)	–
	I will enjoy the flavor of the next cigarette.		19	184	6.6(2.4)	–
	Smoking irritates my mouth and throat.		34	183	4.8(2.6)	–
SSEQ* How confident are you that you could control your smoking behavior in this situation?	You have had a fight with your boyfriend, girlfriend, spouse, or any friend and you are angry and upset.	1 = Does not apply 2 = No control 3 = Low control 4 = High control	6	184	–	2
	You have been out for an evening, you feel relaxed and want to end the evening with a smoke.		8	185	–	2
SSS-V	Please indicate on your answer sheet which of the choices most describes your likes or the way you feel. In some cases you may find items in which you do not like either choice. In these cases mark the choice you dislike least.	1 = I would like to try surf boarding 0 = I would not like to try surf boarding	17	185	–	0

Notes: *The SSEQ was recoded to collapse logic-branched items – the scale reflects the recoded values.

Abbreviations: PRBQ, Perceived Risks and Benefits Questionnaire; SCQ-A, Smoking Consequences Questionnaire (adult version); SSEQ, Smoking Self-Efficacy Questionnaire; SSS-V, Sensation Seeking Scale (fifth version).

Modeling

Logistic Regression with ridge Regularization (LogL2), Support Vector Machine (SVM) with Linear (SVM-L), Polynomial (SVM-P), and Radial Bias Function (SVM-R) kernels, and Random Forest (RF) algorithms were developed in the training set with 5 times repeated 5-fold cross-validation. Training cross-validation splits were across all models. Median and mode missing imputation was conducted within each training cross-validation fold to prevent data leakage. Testing set missing data was imputed with median or mode values from corresponding training set columns. Additional centering, scaling, and dummy coding was applied within each cross-validation fold for the LogL2 and SVM algorithms. Weights were applied to address class imbalance: LogL2 with Class Weights (LogL2-W), SVM with a Weighted RBF Kernel (SVM-RW), and RF with Class Weights (RF-W). We used a ratio of 4:1 weighting in favor of the minority class, corresponding to the inverse of the observed class frequencies in the training set.25 This ratio was applied via the classwt argument for RF, the Weight argument for the radial SVM, and the weights argument in ridge logistic regression, consistent across all models. The random forest algorithm was customized to simultaneously tune mtry and ntree in the caret package, enabling exploration of potential interactions between these hyperparameters on model performance. Models were tuned to optimize F1 score in the training set, which served as our primary performance metric; AUPRC is also reported as a complementary measure of predictive performance. An F₁ and AUPRC score of 1 indicates a perfect model, while scores close to the prevalence of positive cases in the sample represent model performance at chance levels.26 All bootstrapped 95% confidence intervals (CI_95%) were resampled with replacement 2000 times. See Supplemental B for more details on these metrics. For all models, the positive class was making a quit attempt, and the classification probability threshold was 0.5. See Supplemental Table 2 for a list of hyperparameters that were evaluated, including their tuning ranges.

The RF-W, and SVM-L models performed best in the testing set, with F₁ values of 0.36 (AUPRC=0.47, CI_95%=[0.21, 0.75]) and 0.36 (AUPRC=0.44, CI_95%=[0.20, 0.72]), respectively. The RF-W model showed severe overfitting with F₁ of 1.0 (AUPRC=1) when refit to the full training set. Under repeated 5-fold CV in the training set it had a pooled F₁ of 0.32 (AUPRC range = 0.51–0.57). The SVM-L model showed a smaller gap between its full-training-set refit (F₁=0.56; AUPRC=0.75) and its repeated cross-validation performance (pooled F₁=0.45; AUPRC range = 0.49–0.59). Based on the confusionMatrix function in the caret package, none of our models performed significantly better than the no-information rate (ie, always predicting the majority class; all ps > 0.05). Given this lack of significant improvement over the no-information baseline, we did not proceed to compare model performance against an active baseline (eg, logistic regression without regularization). See Table 3 for sample confusion matrices. See Figures 1–3 for F₁ and AUPRC comparisons by model and sample – reported training set metrics are from refitting models derived from repeated CV to the full training set. See Supplemental Table 3 and Supplemental Figures 2 and 3 for RF-W and SVM-L testing set Brier scores and calibration curves.

Table 3.

Model Confusion Matrices

Algorithm	Best Hyperparameter Settings	Training Set (n=185)					Training Repeated CV*					Testing Set (n=46)
		TP	FP	TN	FN	F1	TP	FP	TN	FN	F1	TP	FP	TN	FN	F1
LogL2	Alpha=0, lambda=0.1	15	2	147	21	0.57	54	33	712	126	0.41	1	0	37	8	0.20
LogL2-W	Alpha=0, lambda=5.8, weights: 4,1	28	20	129	8	0.67	125	107	638	55	0.61	2	6	31	7	0.24
SVM-L	C=0.01	15	3	146	21	0.56	63	35	710	117	0.45	2	0	37	7	0.36
SVM-P	C=0.01, scale=0.1, degree=2	18	2	147	18	0.64	70	37	708	110	0.49	1	0	37	8	0.20
SVM-R	C=9, sigma=0.001	15	1	148	21	0.58	64	30	715	116	0.47	1	0	37	8	0.20
SVM-RW	C=0.1, sigma=0.01, Weight: 4,1	29	25	124	7	0.64	129	127	618	51	0.59	3	7	30	6	0.32
RF	mtry=10, ntree=700	36	0	149	0	1.0	64	41	704	116	0.45	2	1	36	7	0.33
RF-W	mtry=4, ntree=100, classwt=4,1	36	0	149	0	1.0	36	10	735	144	0.32	2	0	37	7	0.36

Notes: *Predictions pooled across 5 times repeated 5-fold cross-validation (CV).

Abbreviations: LogL2, Logistic Regression with L2 Penalty; SVM, Support Vector Machine; RF, Random Forest; W, Weighted; L, Linear; P, Polynomial; R, Radial.

Figure 1.

Logistic regression with L2 penalty model performance comparisons.

Figure 2.

Support vector machine model performance comparisons.

Figure 3.

Random forest model performance comparisons.

Feature Importance

The relative importance of each feature in classifying quit attempts was assessed using permutation testing on the full training set. Baseline predictive performance (AUPRC) was computed using each fitted model’s predictions on the observed data. Then, for each feature in turn, that feature’s values were randomly permuted across cases, breaking its association with the outcome while leaving all other features and the fitted model unchanged, and AUPRC was recomputed using the same model applied to the permuted data. This permutation was repeated 50 times per feature to reduce sampling variability from any single random shuffle, and the mean drop in AUPRC across repetitions (baseline minus permuted) served as that feature’s importance score. A decrease in AUPRC after permutation indicated the feature contributed to predictive performance, while no change indicated the feature was uninformative and an increase in AUPRC indicated that the feature was unhelpful to the model. This approach was applied uniformly across model types so importance scores could be comparable. We only interpreted feature importances for the highest performing models in the testing set. See Supplemental D for considerations in interpreting feature importance metrics.

For the RF-W, the top three most influential features were Contemplation Status, PRBQ item “I will miss the taste of cigarettes”, and SCQ item “Cigarettes make my lungs hurt”. For the SVM-L, the top three most influential features were number of past quit attempts, Contemplation Status, and PRBQ item “I will eat more”. Post-hoc analyses were conducted exploratorily to examine the strength and direction of associations between the features of highest importance and quit attempt in the training set. A Chi-Square Test of Independence uncovered a significant trend (χ²=33.94, p<0.001, CI_95%=[0.24, 1.00], Cramer’s V=0.41) in responses on Contemplation Status and residuals, using an absolute value threshold of 1.96, indicated that participants who reported a contemplation status of planning to quit in the next week or next 30 days were more likely to make a quit attempt, while participants who reported “I don’t plan to quit within the next 12 months” were less likely to make a quit attempt. T-tests revealed that participants who reported a quit attempt by follow-up had significantly fewer baseline beliefs that quitting would cause increased eating compared to those who did not attempt to quit (t(180)=−2.37, p=0.019, CI_95%=[−1.36, −0.12], Cohen’s d=−0.45), lower baseline expectations of missing the taste of cigarettes (t(172)=−2.12, p=0.036, CI_95%=[−1.37, −0.05], Cohen’s d=−0.41), greater baseline beliefs that cigarettes make their lungs hurt (t(179)=3.03, p=0.003, CI_95%=[0.51, 2.41], Cohen’s d=0.56), and a history of more quit attempts (t(37.34)= 3.32, p=0.002, CI_95%=[1.69, 6.96], Cohen’s d=1.02). Of these associations, a history of more quit attempts was also associated with quit attempt in the testing set (t(44)= 3.2 2, p=0.002, CI_95%=[1.31, 5.70], Cohen’s d=1.20). Moreover, the residuals for contemplation status trended similarly in the testing set responses of “I plan to quit in the next week” and “I don’t plan to quit within the next 12 months”.

Discussion

The present study utilized supervised machine learning methods to predict the occurrence of quit attempts over a one-month follow-up period among adult cigarette smokers recruited for a clinical trial evaluating a method of inducing cessation attempts. Primary evaluation of machine learning models concerns predictive performance in the held-out testing sets. Overall prediction in our held-out testing set was poor, suggesting the models overfit to the training set. All models had low sensitivity, with the greatest number of correctly predicted participants as quit attempters being 3 of 9 (~33%). As expected, given the class imbalance, all models performed better on specificity than sensitivity. Differential class weighting adjustments for class imbalance in quit attempts did appear to marginally improve performance.

Feature importance metrics for the top performing RF-W and SVM-L models suggested that history of a greater number of quit attempts and contemplation of quitting were the most important variables for model predictions, respectively, which merit testing in future research to determine their replicability. Effect sizes for associations between these higher-ranked features and quit attempts in the training and testing sets were medium to large. These findings are consistent with prior literature where a greater number of past quit attempts and more contemplation of quitting has been prospectively linked to lower rates of smoking.27,28 For example, prior literature has found that smokers often engage in repeated quit attempts; however, there is great variability in estimations for the number of quit attempts that it takes to successfully quit,29 and number of quit attempts before successful cessation may vary by population characteristics.30 Notably, variables associated with nicotine dependence, such as age of first cigarette and number of years smoking, were either not related to quit attempts in the training set, and thus excluded during feature selection, or scored low on model variable importance. A longitudinal study using machine learning to predict smoking cessation in a large Dutch cohort found that smoking initiation after 18 years old, fewer years of smoking, and a greater number of cigarettes smoked 30 days prior to assessment predicted greater chances of smoking cessation.31 Nicotine dependence is one of the most robust predictors of smoking cessation and relapse,32 which highlights that prediction of making any quit attempt may differ from prediction of successful cessation and maintenance. However, caution must be taken in interpreting these feature importance metrics, given the overall performance of the models was poor.

One potentially relevant methodological limitation is that the sample may have been too small to provide adequate training examples for the algorithm to identify reliable underlying relationships. This is reflected in the wide 95% confidence intervals around the test-set AUPRC estimates. Repeated 5-fold cross-validation was implemented to provide the model with more training sample data combinations; however, the models were repeatedly trained with the same data, which may not have reflected the general population characteristics or those of the testing set. Estimates for an appropriate sample size vary and depend on factors such as effect size, class separability, models choice, and data quality.33

Also, predictive results may have been constrained by limitations of the dataset. Quit attempts may be better predicted with variables such as smoking rules at home and advice from medical professionals or delay discounting.11,34 Moreover, filtering by univariate feature significance identified variables most likely relevant to quit attempt status, but this method does not consider interactions and dependencies among features, and how those may influence the outcome. Relatedly, because this selection step was conducted once on the full training set prior to internal cross-validation, rather than nested within each fold, our internal cross-validated performance estimates should be interpreted as optimistic rather than unbiased estimates of generalizability. Our independent test-set metrics, which are unaffected by this issue, offer a more trustworthy estimate of generalization performance. Future research might consider employing alternate feature selection methods that explore feature space more comprehensively, identifying more complex relationships among the features, including wrapper methods,9,35 for which a larger sample and one with more positive cases (ie, quit attempters) is likely needed. Finally, there was some ambiguity in the length of participant’s quit attempt. For example, one participant responded yes to a ≥24 hour quit attempt and then reported the length of their attempt as “10” in an open-ended follow-up question.

Conclusions

The present study leveraged machine learning techniques to investigate predictors of smoking quit attempts longitudinally across an experimental trial. While model prediction sensitivity was low and we were ultimately unable to detect reliable associations for predicting quit attempts, this research explores an important link in the behavioral pathway to smoking cessation and suggests there is likely heterogeneity in the factors that influence smoking behaviors. Future studies should consider utilizing machine learning to differentiate predictors of successful and unsuccessful quit attempts. Further research would benefit from larger, more diverse samples, with more frequent repeated assessments to probe possible causal mechanisms between proximal cognitive states and quit attempts. These bottom-up, data-driven approaches have the potential to identify clinically useful profiles of individuals based on their behavioral outcomes and may offer a way to help understand the complex mechanisms underlying smoking behavior, ultimately providing ideas for research on the efficacy of decision rules used to match treatments to individual characteristics of smokers.

Funding Statement

This research was supported by the National Cancer Institute at the National Institutes of Health (Grant Number: 1R15CA198838-01). The funder played no role in the preparation of this manuscript.

Code and Data Availability

Data, along with preprocessing and analysis scripts, are publicly available on our GitHub repository (https://github.com/zelaky/LV).

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising, or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

Ms Zoë Laky reports Meeting/travel support from National Institute for Mental Health, American University, outside the submitted work. Ms. Zoë E. Laky and Drs. Maria Barouti, Amanda Kaufmann, and David A. Haaga declare that they have no competing interests, financial or otherwise.