Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Feb 1.
Published in final edited form as: Drug Alcohol Depend. 2019 Oct 29;207:107676. doi: 10.1016/j.drugalcdep.2019.107676

Risk Factors Associated with Subsequent Initiation of Cigarettes and E-cigarettes in Adolescence: A Structural Equation Modeling Approach

Natalie Kintz 1, Mengyu Liu 1,2, Chih-Ping Chou 1, Robert Urman 1, Kiros Berhane 1, Jennifer B Unger 1, Tess Boley Cruz 1, Rob McConnell 1, Jessica L Barrington-Trimis 1
PMCID: PMC6980983  NIHMSID: NIHMS1545511  PMID: 31816488

Abstract

Background

Previous youth tobacco research has identified multiple correlated risk factors for initiation of cigarette and e-cigarette use; whether these factors are independently associated with initiation is not known, due to challenges with disentangling the independent effects of these correlated risk factors.

Methods

Students in 11th/12th grade enrolled in the Southern California Children’s Health Study were surveyed in 2014 (baseline) and again in 2015 (N=1553). Structural equation models (SEM) were developed to investigate associations of susceptibility, marketing, and the social environment (as latent factors), and other tobacco use at baseline with cigarette or e-cigarette initiation between baseline and follow-up. Analyses were restricted to baseline never cigarette users (N=1293) for models evaluating cigarette initiation, and to never e-cigarette users (N=1197) for models evaluating e-cigarette initiation.

Results

In fully-adjusted prospective SEM models, latent factors for cigarette susceptibility, marketing, and the social environment, along with ever e-cigarette use and ever hookah use at baseline were independently associated with cigarette initiation between baseline and follow-up (P<0.05). Similarly, latent factors for e-cigarette susceptibility, marketing, and the social environment, along with ever hookah use at baseline were associated with e-cigarette initiation between baseline and follow-up (P<0.05); however, cigarette use at baseline was not associated with e-cigarette initiation in SEM models (P=0.16).

Conclusions

We identified independent effects of multiple risk factors in SEM models on initiation of cigarettes and e-cigarettes. E-cigarette use was associated with cigarette initiation, but cigarette use was not associated with e-cigarette initiation in fully adjusted models. Research to identify underlying causal mechanisms are warranted.

Keywords: e-cigarettes, cigarettes, adolescence, structural equation models (SEM), risk factors, prospective cohorts, longitudinal analysis, epidemiology

1. INTRODUCTION

Combustible tobacco use is the leading cause of preventable death in the United States (US Department of Health and Human Services, 2012). It is critically important to understand what risk factors independently and collectively contribute to cigarette use in adolescents and young adults in order to develop preventive strategies and regulatory policies to reduce cigarette use and the public health burden of cigarettes. Decades of tobacco control research have identified numerous environmental and psychosocial risk factors for the initiation of cigarette use among youth (US Department of Health and Human Services, 2012). However, most of the studies to date have typically evaluated only one or two key risk factors at a time, using traditional multivariate techniques. For example, self-reported susceptibility to cigarette use, exposure to tobacco marketing, and a supportive social environment (use of cigarettes in the home, friends’ use of and positive attitudes toward tobacco) have all been shown to be risk factors of combustible cigarette use in separate models (U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES, 2014; US Department of Health and Human Services, 2012). More recently, numerous studies have reported that e-cigarette use (Barrington-Trimis et al., 2016; Best et al., 2017; Bold et al., 2018; Conner et al., 2017; Hammond et al., 2017; Leventhal et al., 2016; Leventhal et al., 2015; Loukas et al., 2018; Lozano et al., 2017; Miech et al., 2017; Morgenstern et al., 2018; Primack et al., 2015; Soneji et al., 2017; Spindle et al., 2017; Treur et al., 2018; Unger et al., 2016; Wills et al., 2016a; Wills et al., 2016b; Wills et al., 2016c), cigar use (Kong et al., 2017) and hookah use (Soneji et al., 2015) have also been identified as risk factors for subsequent initiation of combustible cigarettes.

Many of the established risk factors for combustible cigarette smoking may also be relevant to e-cigarette initiation, including a supportive social environment (Barrington-Trimis et al., 2015; Urman et al., 2018), exposure to e-cigarette marketing (Cruz et al., 2018), and self-reported susceptibility to e-cigarette use (Bold et al., 2016; Cruz et al., 2018; Farrelly et al., 2015; Mantey et al., 2016). It is less clear if combustible cigarette smoking is associated with subsequent e-cigarette use, or whether youth and young adults are using e-cigarettes to transition from or to quit cigarette use. Currently, no clinical trials have been conducted to evaluate the efficacy of e-cigarettes as a smoking cessation aid among adolescent or young adult populations. Several observational studies have had mixed results. A recent prospective analysis found that cigarette smoking was not associated with future e-cigarette use (Bold et al., 2017); an earlier study found that cigarette use was associated with a 2-fold increase in e-cigarette initiation (Wills et al., 2016b). We recently reported that young adult cigarette smokers were substantially more likely to remain as cigarette smokers with or without concurrent e-cigarette use than to transition to exclusive e-cigarette use or quit all nicotine product use (Barrington-Trimis et al., 2018).

While these studies provide key insight into important risk factors, more comprehensive approaches are needed to evaluate the net impact of multiple correlated risk factors for cigarette or e-cigarette initiation in a single model, and to disentangle the unique contribution of each factor in the presence of other correlated risk factors. To address these gaps in the understanding of cigarette and e-cigarette initiation, we examined the influence of the social environment, marketing exposure, susceptibility to tobacco use, and use of other tobacco products on the initiation of both cigarette and e-cigarette use (independently) in a prospective study of adolescents during the transition to young adulthood. Specifically, we utilized a structural equation modeling (SEM) approach to parse the contributions of multiple correlated risk factors on cigarette and e-cigarette initiation, respectively (Figure 1).

Figure 1. Conceptual Model of Cigarette (or E-Cigarette) Initiation at follow-up in association with three product specific latent factors at baseline, baseline tobacco product use, and sociodemographic variables.

Figure 1.

Open in a new tab

Conceptual model of cigarette (or e-cigarette) susceptibility, cigarette (or e-cigarette) marketing, and cigarette (or e-cigarette) social environment latent factors, e-cigarette (or cigarette), hookah, and cigar use, and demographic variables at baseline on cigarette (or e-cigarette) initiation at follow-up. Squares represent observed variables, circles represent latent variables. Covariance estimates for all factors and covariates were included but not all were shown in the model for graphic simplicity.

2. METHODS

2.1. Participants

This study utilized data from the Southern California Children’s Health Study (CHS), a prospective cohort study of adolescents, for which recruitment and data collection procedures have been described in detail elsewhere (Barrington-Trimis et al., 2015; McConnell et al., 2006). Briefly, students in 11th or 12th grade (mean age =17.3 ±0.6) were asked to complete in-classroom, paper-based questionnaires under study staff supervision from January 2014 to June 2014 (baseline for the current report; N=2097). Participants completed an online follow-up questionnaire between February 2015 and March 2016 after they reached 18 years of age (N=1563; response rate = 75%; mean age = 18.8 ± 0.6). Analyses were restricted, in respective analyses, to those who reported never having used cigarettes at baseline to examine risks of cigarette initiation (N=1293), and to those who had never used e-cigarette at baseline to examine risks of e-cigarette initiation (N=1197).

2.2. Ethics Statement

The study was approved by the University of Southern California Institutional Review Board. Participants age 18 or older provided written informed consent. Written parental informed consent and student assent were obtained for all CHS participants prior to data collection in 2014.

2.3. Measures

2.3.1. Tobacco and Alternative Tobacco Use

Participants were asked to report the age they first used e-cigarettes, cigarettes, hookah, and cigars in each survey. Participants who reported that they had never used a product were classified as “never users.” Those who reported an age of first use of a product were considered “ever users” of that product. Participants who reported no use of a product at baseline, but reported use of a product at follow-up were considered to have initiated use between baseline and follow-up.

2.3.2. Latent Variables

Susceptibility to Tobacco and Alternative Tobacco Product Use

A latent susceptibility factor for cigarette or e-cigarette use was assessed using three items as indicator variables for each product. Participants were asked the following questions separately for cigarettes and e-cigarettes with four response options (definitely not, probably not, probably yes, and definitely yes): (1) At any time in the next year do you think you will use this product?; (2) If one of your best friends were to offer you this product would you use it?; and (3) Have you ever been curious about using this product? Each item was treated as a continuous variable (e.g., ranging from 0 [definitely not] to 3 [definitely yes]).

Marketing

A latent factor for exposure to cigarette or e-cigarette marketing was assessed using five marketing questions as indicator variables for each tobacco product. Participants were asked the following questions separately for cigarettes and e-cigarettes: (1) When you are using the Internet, how often do you see ads for this product?; (2) When you are reading newspapers or magazines, how often do you see ads for this product?; (3) When you go to a convenience store, supermarket, or gas station, how often do you see ads for this product?; (4) During the past 30 days, how often did you see an ad for this product that was outdoors on a billboard or could be seen from outside a store?; and (5) When you watch TV or go to the movies, how often do you see actors using this product? Since the marketing indicator variables were well distributed across the range of response options, each was treated as a continuous variable from 1–6 based on response (I do not use the marketing channel, never, rarely, sometimes, most of the time, and always).

Social Environment

A latent factor for the cigarette or e-cigarette social environment was assessed using three social environment questions as indicator variables for each product. Participants were asked the following questions, separately for cigarettes and e-cigarettes: (1) How would your best friends act toward you if you used this product? (very unfriendly, unfriendly, friendly, or very friendly, treated as a continuous variable [0–3]). (2) How many of your four closest friends use this product? (response options=0–4 friends, treated as a continuous variable [0–4]; responses of “not sure” were treated as missing in the analysis). (3) Does anyone who lives with you now use this product? (yes/no).

2.3.3. Sociodemographic characteristics

Self-administered questionnaires completed by parents of participants assessed gender, ethnicity (Hispanic white, non-Hispanic white, and other), and parental education (highest level of education of either parent: High school diploma or less, some college, college diploma or above), which were included as additional covariates (McConnell et al., 2006).

2.3.4. Statistical Analysis

Chi-square tests were used to evaluate the odds of cigarette or e-cigarette initiation by each sociodemographic characteristic and by baseline tobacco product use in separate models. Structural equation modeling (SEM) was used to examine the associations of latent factors (susceptibility, marketing, and social environment) and observed variables (ever use of cigarette or e-cigarette, hookah, and cigar at baseline, gender, ethnicity, and parental education) with initiation of e-cigarette or cigarette use (in separate models). The zero-ordered correlations for the latent factor indicator variables, baseline use of other tobacco products, and cigarette or e-cigarette initiation at follow-up were first evaluated using Pearson product moment coefficients for continuous-continuous, point-biserial coefficients for continuous-dichotomous, and phi coefficients for dichotomous-dichotomous observed variables. SEM models were then specified separately for cigarette and e-cigarette initiation (Figure 1). The minimally adjusted model only included three latent factors (social environment, susceptibility to tobacco use, exposure to marketing) specific to the outcome tobacco product, and baseline cigarette use for e-cigarette initiation model or baseline e-cigarette use for the cigarette initiation model. The maximally adjusted model additionally included ever hookah or cigar use at baseline as well as gender, ethnicity, parental education. Models were estimated using robust weighted le\ast squares (WLSMV) (Muthén and Muthén, 1998–2015). The variances for each factor were set to 1 to get estimates for each indicator variable. Model estimates were standardized to factors and dependent variables (MPlus STDY method) (Muthén and Muthén, 1998–2015) and P-values were reported. A level of significance of α < 0.05 was used in all statistical analyses. The goodness-of-fit of the model to the data was assessed using a chi-square goodness-of-fit test statistic, comparative fit index (CFI), and the root square error of approximation (RMSEA). Minimal CFIs of 0.90 were required for model acceptance, and values of 0.95 or greater were accepted as an indication of good model fit. RMSEAs of less than 0.06 were accepted as indicators of a good fitting model (Bentler, 1990; Browne and Cudeck, 1993; Hu and Bentler, 1998).

Missing data rates were generally low for tobacco product use, susceptibility, marketing, several social environment variables, and sociodemographic characteristics (<3%). Higher missing data rates were observed for friends’ use of cigarettes or e-cigarettes due to responses of “not sure”, which were treated as missing data (6% for both cigarette and e-cigarette). Parental education was missing for 10% of the entire sample. As such, a missing parental education category was used for the analysis. Missing data were handled by the full information maximum likelihood (FIML) estimation in Mplus. FIML estimates coverage of missing data at the covariance matrix level and uses all possible data points during the analysis (Allison, 2003). Sensitivity analyses evaluated differences in key sociodemographic characteristics for those who did (vs. did not) complete the follow-up survey (Supplemental Table 1). Analyses were performed using SAS 9.4 (SAS, 2013) and Mplus 7.4 (Muthén and Muthén, 1998–2015).

3. RESULTS

Among never cigarette users at baseline (N=1293), 16.4% initiated cigarette use between baseline and follow-up (Table 1). E-cigarette, hookah, and cigar use at baseline were strongly associated with cigarette initiation in unadjusted models (OR e-cigarette: 4.91, 95% CI: 3.42–7.05; OR hookah: 4.10, 95% CI: 2.94–5.73; OR cigar: 5.29, 95% CI: 3.06–9.15, respectively). Males were also more likely to initiate cigarette use (unadjusted OR=1.67; 95% CI: 1.23–2.25), but no associations were observed for ethnicity or parental education.

Table 1.

Descriptive frequencies for e-cigarette and cigarette initiation at baseline

Characteristic E-Cigarette Initiation (N=1197) Cigarette Initiation (N=1293)
No (n= 837) Yes (n= 327) Odds Ratios No (n= 1058) Yes (n= 208) Odds Ratios
N(%) N(%) (95% CI) N(%) N(%) (95% CI)
Cigarette use (baseline)
No 809 (73.5) 292 (26.5) Ref 1058 (83.6) 208 (16.4) --
Yes 28 (44.4) 35 (55.6) 3.46 (2.07, 5.79) -- --
E-cigarette use (baseline)
No 837 (71.9) 327 (28.1) -- 965 (87.3) 141 (12.8) Ref
Yes -- -- 92 (58.2) 66 (41.8) 4.91 (3.42, 7.05)
Hookah use (baseline)
No 783 (77.07) 233 (22.93) Ref 923 (87.7) 130 (12.4) Ref
Yes 54 (36.49) 94 (63.51) 5.85 (4.06, 8.43) 135 (63.4) 78 (36.6) 4.10 (2.94, 5.73)
Cigar use (baseline)
No 823 (73.0) 304 (27.0) Ref 1029 (85.0) 181 (15.0) Ref
Yes 14 (38.9) 22 (61.1) 4.25 (2.15, 8.42) 29 (51.8) 27 (48.2) 5.29 (3.06, 9.15)
Gender
Female 461 (73.1) 170 (26.9) Ref 576 (86.9) 87 (13.1) Ref
Male 376 (70.5) 157 (29.5) 1.13 (0.88, 1.46) 482 (79.9) 121 (20.1) 1.66 (1.23, 2.25)
Ethnicity
Non-Hispanic white 308 (70.32) 130 (29.68) Ref 399 (81.43) 91 (18.57) Ref
Hispanic white 398 (70.82) 164 (29.18) 0.98 (0.74, 1.28) 507 (84.08) 96 (15.92) 0.83 (0.61, 1.14)
Other 131 (79.88) 33 (20.12) 0.60 (0.39, 0.92) 152 (87.86) 21 (12.14) 0.61 (0.36, 1.01)
Parental Education
High School or less 201 (71.0) 82 (29.0) 0.99 (0.70, 1.41) 256 (84.5) 47 (15.5) 0.86 (0.57, 1.29)
Some college 241 (70.9) 99 (29.1) Ref 308 (82.4) 66 (17.6) Ref
College or above 310 (72.9) 115 (27.1) 0.90 (0.66, 1.24) 389 (83.3) 78 (16.7) 0.94 (0.65, 1.34)
Missing 85 (73.3) 31 (26.7) 0.89 (0.55, 1.42) 105 (86.1) 17 (13.9) 0.76 (0.42, 1.35)
Open in a new tab

Among never e-cigarette users at baseline (N=1197), 26.5% initiated e-cigarette use between baseline and follow-up (Table 1). Cigarette, hookah, and cigar use at baseline were strongly associated with e-cigarette initiation in unadjusted models (OR cigarette: 3.46, 95% CI: 2.07–5.79; OR hookah: 5.85, 95% CI: 4.06–8.43; OR cigar: 4.25, 95% CI: 2.15–8.42, respectively). Gender, ethnicity, and parental education were not associated with e-cigarette initiation.

No differences between those who were included in analyses or lost to follow-up were observed for e-cigarette use, hookah use, or cigar use (Supplemental Table 1). Those lost to follow up were more likely to be male, Hispanic White (vs. Non-Hispanic White), and had parents with less than a high school education (or data on parental education was missing). In addition, among the sample of e-cigarette never users at baseline, baseline cigarette use was more common among those lost to follow-up.

3.1. Cigarette Initiation SEM

For cigarette initiation, zero-order correlation coefficients ranged from 0.47–0.63 for susceptibility measures, 0.23–0.44 for marketing measures, 0.03–0.23 for social environment measures, and 0.31–0.42 for use of other tobacco products at baseline (Supplemental Table 2a).

In SEM models, e-cigarette use at baseline, marketing, and social environment factors were associated with cigarette initiation at follow-up in both the minimally adjusted (Ps<0.05) and maximally adjusted (Ps<0.05) models (Table 2). In addition, hookah use at baseline (P<0.001) and gender (p=0.01) were associated with cigarette initiation in the maximally adjusted model (P<0.001); ethnicity, parental education, and cigar use were not associated with cigarette initiation. Significant correlations were observed between cigarette susceptibility and marketing factors (r=0.13, P<0.001), and susceptibility and social environment factors (r=.60, P<0.001), but not marketing and social environment factors (r=0.09, P=0.08). The R2 for ever-use of cigarettes at follow-up in the minimally adjusted model was 0.209 (P<0.001), and in the maximally adjusted model was 0.259 (P<0.001). Both the minimally adjusted (Chi-square goodness of fit=178.92, d.f.=57, P<0.01; CFI=0.95; RMSEA=0.04) and the maximally adjusted (Chi-square goodness of fit=322.94, d.f.=121, P<0.01; CFI=0.99; RMSEA=0.04) models fit the data well.

Table 2.

Relationship between baseline susceptibility, marketing and social environment with subsequent cigarette Initiation at follow-up

Minimally Adjustedac Maximally Adjustedbc
Ever Use of E-Cigarettes at baseline 0.236 (P<0.001) 0.141 (P<0.001)
F1 (Susceptibility to cigarettes) 0.137 (P=0.045) 0.139 (P=0.049)
F2 (Marketing [cigarettes]) 0.131 (P=0.004) 0.129 (P=0.004)
F3 (Social Environment [cigarettes]) 0.213 (P=0.02) 0.199 (P=0.04)
Ever Use of Hookah at baseline 0.163 (P<0.001)
Ever Use of Cigars at baseline 0.060 (P=0.11)
Parent Education
 High School or less −0.015 (P=0.80)
 Some College Ref
 College or more 0.005 (P=0.93)
 Missing −0.042 (P=0.38)
Ethnicity
 Non-Hispanic White Ref
 Hispanic White −0.049 (P=0.36)
 Other −0.063 (P=0.21)
Male 0.102 (P=0.01)
Open in a new tab
a:

Effects of three latent factors and e-cigarette use at baseline on cigarette initiation at follow-up

b:

Effects of three latent factors, e-cigarette, cigar, and hookah use, and demographic variables at baseline on cigarette initiation at follow-up

c:

Parameter coefficients in probit link are standardized using MPlus STDY estimation method

3.2. E-cigarette Initiation SEM

For e-cigarette initiation, zero-order correlations coefficients ranged from 0.52–0.68 for susceptibility indicators, 0.23–0.49 for marketing measures, 0.19–0.30 for social environment measures, and 0.23–0.31 for use of other tobacco products at baseline (Supplemental Table 2b).

In SEM models, cigarette use at baseline and the three latent factors were all associated with e-cigarette initiation at follow-up in the minimally adjusted model (Table 3; P<0.008). After adjusting for sociodemographic variables and other tobacco product use at baseline, the three latent factors were still associated with e-cigarette initiation (Ps≤0.02), but the estimated effect of cigarette use was markedly attenuated and no longer significant (P=0.16). Hookah use at baseline was associated with e-cigarette initiation in the maximally adjusted model (P<0.001), but no associations of demographic characteristics or cigar use with e-cigarette initiation were observed. Significant correlations were observed between all three latent factors in the maximally adjusted model. The highest correlation was between e-cigarette susceptibility and social environment factors (r=0.60, P<0.001), followed by e-cigarette marketing and social environment factors (r=0.25, P<0.001), and e-cigarette susceptibility and marketing factors (r=0.16, P<0.001). The R2 for ever-use of e-cigarettes at follow-up in the minimally adjusted model was 0.194 (P<0.001), and in the maximally adjusted model was 0.284 (P<0.001). Both the minimally adjusted (Chi-square goodness of fit=111.43, d.f.=57, P<0.001; CFI=0.97; RMSEA=0.03) and the maximally adjusted (Chi-square goodness of fit=237.53, d.f.=121, P<0.001; CFI=0.99; RMSEA=0.03) models fit the data well.

Table 3.

Relationship between baseline susceptibility, marketing and social environment with subsequent e-cigarette Initiation at follow-up

Minimally Adjustedac Maximally Adjustedbc
Ever Use of Cigarettes at baseline 0.127 (P=0.001) 0.056 (P=0.16)
F1 (Susceptibility to e-cigarettes) 0.160 (P=0.008) 0.137 (P=0.022)
F2 (Marketing [e-cigarettes]) 0.136 (P=0.003) 0.121 (P=0.007)
F3 (Social Environment [e-cigarettes]) 0.236 (P=0.003) 0.246 (P=0.002)
Ever Use of Hookah at baseline 0.288 (P<0.001)
Ever Use of Cigars at baseline 0.020 (P=0.64)
Parent Education
  High School or less 0.012 (P=0.80)
  Some College Ref
  College or more 0.025 (P=0.59)
  Missing −0.009 (P=0.83)
Ethnicity
  Non-Hispanic White Ref
  Hispanic White 0.015 (P=0.75)
  Other −0.071 (P=0.09)
 Male 0.005 (P=0.89)
Open in a new tab
a:

Effects of three latent factors and cigarette use at baseline on e-cigarette initiation at follow-up

b:

Effects of three latent factors, cigarette, cigar, and hookah use, and demographic variables at baseline on e-cigarette initiation at follow-up

c:

Parameter coefficients in probit link are standardized using MPlus STDY estimation method

4. DISCUSSION

This prospective study assessed the independent effects of the tobacco social environment, exposure to tobacco marketing, susceptibility to tobacco product use, and use of other tobacco products with cigarette and e-cigarette initiation in adolescents using SEM to model all factors simultaneously. The results demonstrate that: (1) a pro-tobacco social environment, exposure to tobacco marketing, and susceptibility to tobacco product use each independently contribute to initiation of both cigarettes and e-cigarettes; (2) after accounting for the effects of these factors, e-cigarette use was significantly associated with cigarette initiation, but cigarette use was not associated with e-cigarette initiation; and (3) hookah use was strongly associated with subsequent initiation of both e-cigarette and cigarette use.

The use of SEM has several advantages relative to traditional multivariate techniques. First, while most multivariate techniques do not explicitly model measurement error, SEM statistically controls for measurement error for both independent and dependent variables by including error variance parameters in the model (Byrne, 2012; Novikova et al., 2013). Next, most multivariate techniques are designed to evaluate variables that are directly measured, which can be limiting if researchers are interested in testing underlying theoretical constructs, or latent factors. SEM allows for the estimation of these latent factors, via observed variables while accounting for measurement error. Importantly, SEM eliminates the issue of multicollinearity among covariates, which make it possible to examine effects of multiple correlated risk factors at the same time. Taken together, this allows SEM models to evaluate the effects of observed variables that are highly correlated (e.g., indicators for susceptibility to tobacco use), as well as factors that are related (e.g., susceptibility to use and a pro-tobacco social environment). Overall, an SEM approach may provide a better understanding of how risk factors, individually and collectively, influence tobacco product initiation, which can inform interventions and policy measures designed to reduce exposure to the factors that pose the greatest risk to youth tobacco initiation.

While all three factors were found to be associated with cigarette and e-cigarette initiation, the strongest association for initiation of both products was a pro-tobacco social environment. Interventions to help adolescents resist peer influences and efforts to denormalize e-cigarette and cigarette use in youth may help reverse the current high rates of use. Such programs have already been shown to be successful in reducing cigarette use among adolescents (Malone et al., 2012; Services., 2012). In addition, since cigarette and e-cigarette marketing exposure is linked with subsequent use of those products, efforts to further restrict pro-tobacco marketing, and marketing campaigns designed to deliver preventive messages across diverse media channels to which adolescents are easily exposed (Cruz et al., 2018; Hanewinkel et al., 2011; Henriksen et al., 2010; Villanti et al., 2016) may be effective strategies to reduce adolescent tobacco use. For example, health communication interventions that use fast-paced graphics, visuals, and music to provide accurate information about the risks of tobacco product use, like the “Wake Up” campaign in California, and the FDA’s “The Real Cost” campaign, are promising platforms that may help denormalize tobacco product use, counteract tobacco marketing targeted towards youth, and ultimately reduce tobacco product initiation among young adults (California Department of Public Health, 2017; Farrelly et al., 2017; Jamal et al., 2017).

The relationship between e-cigarette and combustible cigarette use is critical to evaluating the public health implications of e-cigarettes. E-cigarettes may lead to the reduction or cessation of combustible tobacco use among adult smokers, but could also lead to combustible cigarette initiation and the renormalization of smoking among youth and young adults (Fairchild et al., 2014). A number of studies report a substantial proportion of adolescent e-cigarette users have never used conventional cigarettes, and multiple reports from a series of prospective cohort studies have found that nonsmoking youth who use e-cigarettes are at increased risk of initiating combustible cigarette smoking (Barrington-Trimis et al., 2016; Best et al., 2017; Bold et al., 2018; Conner et al., 2017; Hammond et al., 2017; Leventhal et al., 2016; Leventhal et al., 2015; Loukas et al., 2018; Lozano et al., 2017; Miech et al., 2017; Morgenstern et al., 2018; Primack et al., 2015; Soneji et al., 2017; Spindle et al., 2017; Treur et al., 2018; Unger et al., 2016; Wills et al., 2016a; Wills et al., 2016b; Wills et al., 2016c). In the SEM analysis, e-cigarette use was associated with increased risk of combustible cigarette initiation, independent of the social environment, marketing, susceptibility, other tobacco use, and demographic characteristics. This suggests the association between e-cigarette use and subsequent smoking may result from more than a shared risk for e-cigarettes and cigarettes. On the other hand, in fully adjusted SEM models, we observed a markedly attenuated and statistically non-significant association between cigarette use and e-cigarette initiation. Taken together, these findings indicate e-cigarette use may encourage future cigarette use, but cigarette use does not necessarily result in future e-cigarette use. Thus, e-cigarettes may expose more youth to tobacco products, without providing much benefit via harm reduction, ultimately resulting in a net adverse public health impact in the young adult population. The overall impact of e-cigarettes on the public health of youth and young adult populations is also dependent upon longer term use patterns. If e-cigarettes are associated with subsequent initiation but not long term cigarette use, then the adverse impact of e-cigarettes is minimized; however, emerging evidence suggests that e-cigarette use is associated not only with cigarette initiation but with progression to more frequent cigarette use patterns and a similar trajectory toward regular cigarette use (Barrington-Trimis et al., 2018; Leventhal et al., 2016). Similarly, longer term transition patterns away from cigarettes will also affect the net public health impact of e-cigarettes, however, our findings that cigarette smokers do not have any greater likelihood of initiation of e-cigarettes suggests that they are also unlikely to have greater likelihood of continued and regular e-cigarette use.

Although recent research has primarily focused on the effect of e-cigarette use on cigarette initiation in youth, and vice versa, the role of hookah in both cigarette and e-cigarette initiation merits further investigation (Gilreath et al., 2016; Huh and Leventhal, 2016). Prospective studies have shown that hookah use was associated with cigarette initiation, current cigarette use, and higher intensity of smoking among young adults (Jaber et al., 2015; Soneji et al., 2015). In a previous cross-sectional analysis of CHS data, we observed that the prevalence of hookah use was higher than that of either e-cigarette or cigarette use in high school students (Gilreath et al., 2016). While use of all three tobacco products were correlated, we identified a group that had a pattern of both e-cigarette and hookah use that distinguished it from other patterns of polytobacco product use (Gilreath et al., 2016). In a 6-month prospective follow-up of a younger cohort enrolled in middle school, transitions to polytobacco product use were common, possibly with an intermediate transition from never tobacco use to e-cigarette and hookah use (Huh and Leventhal, 2016). In the current study, our findings of significant associations of previous e-cigarette and hookah use on the initiation of tobacco smoking were consistent with these previous reports. More research is needed to understand why e-cigarette and hookah use are strong risk factors for subsequent cigarette use in adolescents (Schneider and Diehl, 2016). Given that e-cigarettes and hookah are frequently sold and marketed in forms and flavors that may preferentially appeal to youth (Carpenter et al., 2005; Connolly, 2004; Vasiljevic et al., 2016), regulatory policies that successfully reduce youth exposure and access to these products might reduce both initiation of cigarettes and e-cigarettes, but also reduce the subsequent transition from e-cigarette to cigarette use.

This analysis is subject to some limitations. First, 25% of the sample were lost to follow up, and significant differences in retention were observed by gender, race and parental education (Ps<0.05, Supplemental Table 1). We cannot rule out the possible role of selection bias; however, the estimated effects were robust to adjustment for these characteristics. Second, the indicator variables for the latent variables in each SEM model were product specific. However, it is likely that susceptibility, marketing, and social environment factors are not product specific, but rather are influenced by multiple tobacco products. In addition, this study specifically examines risk factors for cigarette or e-cigarette initiation. Future studies should investigate factors associated with longer term and heavier use patterns as this data is made available. Moreover, future analyses could additionally examine bidirectional associations of the risk factors of this study with tobacco product use. The analyses adjust for many important factors, but additional risk factors for initiation, such as risk-taking and mental health, were not available. Overall, the variance accounted for by variables included in each model was relatively low; thus, other unmeasured variables may account for a greater proportion of cigarette or e-cigarette initiation. SEM is a useful tool for modeling multiple important risk factors in a single model, and our approach is amenable to inclusion of a wider range of established risk factors to gain a broader insight into how all potential risks influence cigarette and e-cigarette initiation. Finally, the current analyses did not explicitly assess whether young adults were using e-cigarettes to quit smoking cigarettes.

Conclusion

The analytical approach using SEM successfully identified independent effects of tobacco use, product-specific susceptibility, marketing exposure, and the social environment on initiation of cigarettes and e-cigarettes. Results support the hypothesis that e-cigarette use is associated with subsequent cigarette initiation in the presence of other strong risk factors for initiation, but provide little evidence of an association of cigarette use at baseline with initiation of e-cigarette use. Research to identify underlying mechanisms of these associations are warranted.

Supplementary Material

1

HIGHLIGHTS.

  • In SEM models, multiple independent risk factors – including e-cigarettes – were associated with cigarette initiation.

  • Multiple risk factors were also associated with e-cigarette initiation.

  • Notably, cigarette use was not associated with e-cigarette initiation in fully-adjusted SEM models.

Acknowledgments

Funding Source: Research reported in this publication was supported by grant numbers P50CA180905 and U54CA180905 from the National Cancer Institute at the National Institutes of Health (NIH) and the Food and Drug Administration (FDA) Center for Tobacco Products (CTP), and grant number K01DA042950 from the National Institute for Drug Abuse at NIH. The funders had no role in the design and conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.

Dr. Barrington-Trimis designed the study, collected data, and contributed to formulating the research question and interpretation of the results, provided oversight and mentorship to the first-author regarding the scope and design of the manuscript, critically reviewed the manuscript, and approved the manuscript as submitted.

Footnotes

Financial Disclosure: The authors have no financial relationships relevant to this article to disclose.

Conflict of Interest: The authors have no conflicts of interest relevant to this article to disclose. Robert Urman began a position at Amgen on April 15, 2019 and did not contribute to the paper after that date.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

REFERENCES

  1. Allison PD, 2003. Missing Data Techniques for Structural Equation Modeling. Journal of Abnormal Psychology 112, 545–557. [DOI] [PubMed] [Google Scholar]
  2. Barrington-Trimis JL, Berhane K, Unger JB, Cruz TB, Huh J, Leventhal AM, Urman R, Wang K, Howland S, Gilreath TD, Chou CP, Pentz MA, McConnell R, 2015. Psychosocial Factors Associated With Adolescent Electronic Cigarette and Cigarette Use. Pediatrics 136, 308–317. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Barrington-Trimis JL, Kong G, Leventhal AM, Liu F, Mayer M, Cruz TB, Krishnan-Sarin S, McConnell R, 2018. E-cigarette Use and Subsequent Smoking Frequency Among Adolescents. Pediatrics. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Barrington-Trimis JL, Urman R, Berhane K, Unger JB, Cruz TB, Pentz MA, Samet JM, Leventhal AM, McConnell R, 2016. E-cigarettes and Future Cigarette Use. Pediatrics. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bentler P, 1990. Comparative fit indexes in structural models. Psychological bulletin 107, 238–246. [DOI] [PubMed] [Google Scholar]
  6. Best C, Haseen F, Currie D, Ozakinci G, MacKintosh AM, Stead M, Eadie D, MacGregor A, Pearce J, Amos A, Frank J, Haw S, 2017. Relationship between trying an electronic cigarette and subsequent cigarette experimentation in Scottish adolescents: a cohort study. Tob Control. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bold KW, Kong G, Camenga DR, Simon P, Cavallo DA, Morean ME, Krishnan-Sarin S, 2017. Trajectories of e-cigarette and conventional cigarette use among youth. Pediatrics, e20171832. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bold KW, Kong G, Camenga DR, Simon P, Cavallo DA, Morean ME, Krishnan-Sarin S, 2018. Trajectories of E-Cigarette and Conventional Cigarette Use Among Youth. Pediatrics 141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Bold KW, Kong G, Cavallo DA, Camenga DR, Krishnan-Sarin S, 2016. E-cigarette susceptibility as a predictor of youth initiation of e-cigarettes. Nicotine Tob Res. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Browne M, Cudeck R, 1993. Alternative ways of assessing model fit In: Bollen K, Long J (Eds.), Testing Structural Equation Models. Sage; Beverly Hills, CA: pp. 136–162. [Google Scholar]
  11. Byrne BM, 2012. Structural Equation Modeling with Mplus: Basic Concepts, Applications, and Programming. Routledge Academic, New York. [Google Scholar]
  12. California Department of Public Health, 2017. California Debuts Ads to Counter E-cigarettes https://www.cdph.ca.gov/Programs/OPA/Pages/NR15-024.aspx.accessed on.
  13. Carpenter CM, Wayne GF, Pauly JL, Koh HK, Connolly GN, 2005. New cigarette brands with flavors that appeal to youth: tobacco marketing strategies. Health Aff (Millwood) 24, 1601–1610. [DOI] [PubMed] [Google Scholar]
  14. Conner M, Grogan S, Simms-Ellis R, Flett K, Sykes-Muskett B, Cowap L, Lawton R, Armitage CJ, Meads D, Torgerson C, West R, Siddiqi K, 2017. Do electronic cigarettes increase cigarette smoking in UK adolescents? Evidence from a 12-month prospective study. Tob Control. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Connolly GN, 2004. Sweet and spicy flavours: new brands for minorities and youth. Tob Control 13, 211–212. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Cruz TB, McConnell R, Low BW, Unger JB, Pentz MA, Urman R, Berhane K, Chou CP, Liu F, Barrington-Trimis J, 2018. Tobacco Marketing and Subsequent Use of Cigarettes, E-cigarettes and Hookah in Adolescents. Nicotine Tob Res. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Fairchild A, Bayer R, Colgrove J, 2014. The renormalization of smoking? E-Cigarettes and the tobacco “endgame”. The New England Journal of Medicine 370, 293–295. [DOI] [PubMed] [Google Scholar]
  18. Farrelly MC, Duke JC, Crankshaw EC, Eggers ME, Lee YO, Nonnemaker JM, Kim AE, Porter L, 2015. A Randomized Trial of the Effect of E-cigarette TV Advertisements on Intentions to Use E-cigarettes. Am J Prev Med 49, 686–693. [DOI] [PubMed] [Google Scholar]
  19. Farrelly MC, Duke JC, Nonnemaker J, MacMonegle A, Alexander TN, Zhao X, Delahanty JC, Rao P, Allen JA, 2017. Association Between The Real Cost Media Campaign and Smoking Initiation Among Youths — United States, 2014–2016. MMWR Morbidity and Mortality Weekly Report 66, 47–50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gilreath TD, Leventhal A, Barrington-Trimis JL, Unger JB, Cruz TB, Berhane K, Huh J, Urman R, Wang K, Howland S, Pentz MA, Chou CP, McConnell R, 2016. Patterns of Alternative Tobacco Product Use: Emergence of Hookah and E-cigarettes as Preferred Products Amongst Youth. J Adolesc Health 58, 181–185. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Hammond D, Reid JL, Cole AG, Leatherdale ST, 2017. Electronic cigarette use and smoking initiation among youth: a longitudinal cohort study. CMAJ 189, E1328–E1336. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hanewinkel R, Isensee B, Sargent JD, Morgenstern M, 2011. Cigarette advertising and teen smoking initiation. Pediatrics 127, e271–278. [DOI] [PubMed] [Google Scholar]
  23. Henriksen L, Schleicher NC, Feighery EC, Fortmann SP, 2010. A longitudinal study of exposure to retail cigarette advertising and smoking initiation. Pediatrics 126, 232–238. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hu L, Bentler P, 1998. Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological methods 3, 424–453. [Google Scholar]
  25. Huh J, Leventhal AM, 2016. Progression of Poly-tobacco Product Use Patterns in Adolescents. Am J Prev Med 51, 513–517. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Jaber R, Madhivanan P, Veledar E, Khader Y, Mzayek F, Maziak W, 2015. Waterpipe a gateway to cigarette smoking initiation among adolescents in Irbid, Jordan: a longitudinal study. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease 19, 481–487. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Jamal A, Gentzke A, Hu SS, Cullen KA, Apelberg BJ, Homa DM, King BA, 2017. Tobacco Use Among Middle and High School Students — United States, 2011–2016. MMWR Morbidity and Mortality Weekly Report 66, 597–693. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kong G, Barrington-Trimis J, Mayer M, McConnell R, Leventhal A, Krishnan-Sarin S, 2017. Cigar Use and Subsequent Cigarette Initiation. Society for Research on Nicotine and Tobacco, Florence, Italy. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Leventhal AM, Stone MD, Andrabi N, Barrington-Trimis J, Strong DR, Sussman S, Audrain-McGovern J, 2016. Association of e-Cigarette Vaping and Progression to Heavier Patterns of Cigarette Smoking. JAMA 316, 1918–1920. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Leventhal AM, Strong DR, Kirkpatrick MG, Unger JB, Sussman S, Riggs NR, Stone MD, Khoddam R, Samet JM, Audrain-McGovern J, 2015. Association of Electronic Cigarette Use With Initiation of Combustible Tobacco Product Smoking in Early Adolescence. JAMA 314, 700–707. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Loukas A, Marti CN, Cooper M, Pasch KE, Perry CL, 2018. Exclusive e-cigarette use predicts cigarette initiation among college students. Addict Behav 76, 343–347. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Lozano P, Barrientos-Gutierrez I, Arillo-Santillan E, Morello P, Mejia R, Sargent JD, Thrasher JF, 2017. A longitudinal study of electronic cigarette use and onset of conventional cigarette smoking and marijuana use among Mexican adolescents. Drug Alcohol Depend 180, 427–430. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Malone RE, Grundy Q, Bero LA, 2012. Tobacco industry denormalisation as a tobacco control intervention: a review. Tob Control 21, 162–170. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Mantey DS, Cooper MR, Clendennen SL, Pasch KE, Perry CL, 2016. E-Cigarette Marketing Exposure Is Associated With E-Cigarette Use Among US Youth. J Adolesc Health 58, 686–690. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. McConnell R, Berhane K, Yao L, Jerrett M, Lurmann F, Gilliland F, Künzli N, Gauderman J, Avol E, Thomas D, Peters J, 2006. Traffic, susceptibility, and childhood asthma. Environ Health Perspect 114, 766–772. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Miech R, Patrick ME, O’Malley PM, Johnston LD, 2017. E-cigarette use as a predictor of cigarette smoking: results from a 1-year follow-up of a national sample of 12th grade students. Tob Control. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Morgenstern M, Nies A, Goecke M, Hanewinkel R, 2018. E-Cigarettes and the Use of Conventional Cigarettes. Dtsch Arztebl Int 115, 243–248. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Muthén L, Muthén B, 1998–2015. Mplus User’s Guide. Seveth Edition. Los Angeles, CA: Muthén & Muthén. [Google Scholar]
  39. Novikova SI, Richman DM, Supekar K, Barnard-Brak L, Hall D, 2013. Chapter Three - NDAR: A Model Federal System for Secondary Analysis in Developmental Disabilities Research In: Urbano RC (Ed.), International Review of Research in Developmental Disabilities. Academic Press; pp. 123–153. [Google Scholar]
  40. Primack BA, Soneji S, Stoolmiller M, Fine MJ, Sargent JD, 2015. Progression to Traditional Cigarette Smoking After Electronic Cigarette Use Among US Adolescents and Young Adults. JAMA Pediatr, 1–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. SAS, 2013. Version 9.4 of the SAS System for Windows. SAS Institute Inc; Cary, NC. [Google Scholar]
  42. Schneider S, Diehl K, 2016. Vaping as a Catalyst for Smoking? An Initial Model on the Initiation of Electronic Cigarette Use and the Transition to Tobacco Smoking Among Adolescents. Nicotine & Tobacco Research 18, 647–653. [DOI] [PubMed] [Google Scholar]
  43. Services., U.S.D.o.H.a.H., 2012. Preventing Tobacco Use Among Youth and Young Adults: A Report of the Surgeon General. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Office on Smoking and Health, Atlanta. [PubMed] [Google Scholar]
  44. Soneji S, Barrington-Trimis JL, Wills TA, Leventhal AM, Unger JB, Gibson LA, Yang J, Primack BA, Andrews JA, Miech RA, Spindle TR, Dick DM, Eissenberg T, Hornik RC, Dang R, Sargent JD, 2017. Association Between Initial Use of e-Cigarettes and Subsequent Cigarette Smoking Among Adolescents and Young Adults: A Systematic Review and Meta-analysis. JAMA Pediatr. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Soneji S, Sargent JD, Tanski SE, Primack BA, 2015. Associations between initial water pipe tobacco smoking and snus use and subsequent cigarette smoking: results from a longitudinal study of US adolescents and young adults. JAMA Pediatr 169, 129–136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Spindle TR, Hiler MM, Cooke ME, Eissenberg T, Kendler KS, Dick DM, 2017. Electronic cigarette use and uptake of cigarette smoking: A longitudinal examination of U.S. college students. Addict Behav 67, 66–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Treur JL, Rozema AD, Mathijssen JJP, van Oers H, Vink JM, 2018. E-cigarette and waterpipe use in two adolescent cohorts: cross-sectional and longitudinal associations with conventional cigarette smoking. Eur J Epidemiol 33, 323–334. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES, 2014. The Health Consequences of Smoking - 50 Years of Progress. A report of the Surgeon General.
  49. Unger JB, Soto DW, Leventhal A, 2016. E-cigarette use and subsequent cigarette and marijuana use among Hispanic young adults. Drug Alcohol Depend 163, 261–264. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Urman R, McConnell R, Unger JB, Cruz TB, Samet JM, Berhane K, Barrington-Trimis JL, 2018. Electronic Cigarette and Cigarette Social Environments and Ever Use of Each Product: A Prospective Study of Young Adults in Southern California. Nicotine Tob Res. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. US Department of Health and Human Services, 2012. Preventing Tobacco Use Among Youth and Young Adults: A Report of the Surgeon General. Atlanta (GA): US Department of Health and Human Services, Centers for Disease Control and Prevention, Office on Smoking and Health. [Google Scholar]
  52. Vasiljevic M, Petrescu DC, Marteau TM, 2016. Impact of advertisements promoting candy-like flavoured e-cigarettes on appeal of tobacco smoking among children: an experimental study. Tob Control 25, e107–e112. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Villanti AC, Rath JM, Williams VF, Pearson JL, Richardson A, Abrams DB, Niaura RS, Vallone DM, 2016. Impact of Exposure to Electronic Cigarette Advertising on Susceptibility and Trial of Electronic Cigarettes and Cigarettes in US Young Adults: A Randomized Controlled Trial. Nicotine Tob Res 18, 1331–1339. [DOI] [PubMed] [Google Scholar]
  54. Wills TA, Gibbons FX, Sargent JD, Schweitzer RJ, 2016a. How Is the Effect of Adolescent E-Cigarette Use on Smoking Onset Mediated: A Longitudinal Analysis. Psychol Addict Behav. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Wills TA, Knight R, Sargent JD, Gibbons FX, Pagano I, Williams RJ, 2016b. Longitudinal study of e-cigarette use and onset of cigarette smoking among high school students in Hawaii. Tob Control. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Wills TA, Sargent JD, Gibbons FX, Pagano I, Schweitzer R, 2016c. E-cigarette use is differentially related to smoking onset among lower risk adolescents. Tob Control. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS1545511-supplement-1.docx (49.4KB, docx)

RESOURCES