Abstract
Establishment of psychometrically sound measures is critical to the development of effective interventions. The current study examined the psychometric properties, including factorial invariance, of a six item Temptations to Try Smoking Scale on a sample of middle school students. The sample of 6th grade students (N = 3527) was from 20 Rhode Island middle schools and was 52% male and 84% white. The Temptations to Try Smoking Scale consisted of two correlated subscales: Positive Social and Curiosity/Stress. Structural equation modeling was implemented to evaluate the factorial invariance across four different subgroups defined by gender (male/female), race (white/black), ethnicity (Hispanic/Non-Hispanic), and school size (<200/>200 6th graders). A model is factorially invariant when the measurement model is the same in each of the subgroups. Three levels of invariance were examined in sequential order: 1) Configural Invariance (unconstrained nonzero factor loadings); 2) Pattern Identity Invariance (equal factor loadings); and 3) Strong Factorial Invariance (equal factor loadings and measurement errors). Strong Factorial Invariance provided a good fit to the model across gender (CFI = .96), race (CFI = .96), ethnicity (CFI = .94), and school size (CFI = .97). Coefficient Alphas for the two subscales, Positive Social and Curiosity/Stress, were .87 and .86, respectively. These findings provide empirical support for the construct validity of the Temptations to Try Smoking Scale in middle school students.
Keywords: Factorial invariance, Temptations Scale, Transtheoretical Model, Smoking Prevention
Given the increase in smoking prevalence that occurs throughout middle school years, there is a pressing need to develop and evaluate interventions aimed at the prevention of smoking among adolescents. The Transtheoretical Model of Behavior Change (TTM) is a multidimensional model that has been used extensively to develop effective, population-based interventions for a wide range of behaviors (Prochaska & Velicer, 1997; Velicer et al., 2000). The TTM is based on the integration of four core constructs: stages of change, processes of change, decisional balance, and situational temptations. The situational temptation construct, adapted from Bandura’s self-efficacy theory (Bandura, 1977, 1982) and models of relapse prevention (Shiffman, 1986), reflects the intensity of urges to engage in a specific behavior when in difficult situations (Velicer, DiClemente, Rossi, & Prochaska, 1990). The original measure was developed using the sequential method of measure development (Jackson, 1970, 1971). (For a description of the development of the measure, items, and psychometric properties, see Velicer et al., 1990 and the website http://www.uri.edu/research/cprc.) A variation of the situational temptation measure, a six-item Temptation to Try Smoking Scale (TTSS), has been adapted for smoking prevention in adolescent populations and is currently being administered as part of a computer-tailored intervention based on the TTM.
To be an effective component of an intervention, the TTSS must be psychometrically sound. Factorial invariance is an important part of establishing psychometric properties of a measure. A measurement model is factorially invariant when the structural model is the same across different subgroups of a population. Different levels of factorial invariance are often tested using a sequential process, in which increasingly constrained models assess the equivalence of a measure across subgroups. The current study examined the psychometric properties, including factorial invariance, of the TTSS.
Method
Participants
Participants were 3527 sixth grade students from 20 Rhode Island middle schools involved in a TTM based intervention project. The average age of participants was 11.4 (SD = .68). The sample was 52% male and 84% white.
Temptations to Try Smoking Scale
The six-item Temptations to Try Smoking Scale consisted of two correlated subscales: Positive Social Situations and Curiosity about Smoking/Stress. The measurement model for the scale, including the six items, is presented in Figure 1. The scale employs a 5 point Likert-type response scale, where a response of “1” indicates “Not at all tempted” and a response of “5” indicates “Extremely tempted”. The scale was developed for middle school students using the sequential method of scale development (Jackson, 1971).
Figure 1.
Measurement Model for the Temptations to Try Smoking Scale
Analysis
Structural equation modeling (SEM) was implemented using EQS (Bentler, 2007) to evaluate factorial invariance across four different subgroups defined by gender (male/female), race (white/black), ethnicity (Hispanic/Non-Hispanic), and school size (<200/>200 6th graders). Non-responders were deleted. Table 1 presents sample sizes associated with each category for all four subgroups.
Table 1.
Sample Size by Category for Each Subgroup
| Subgroup | Category | N |
|---|---|---|
| Gender | Male | 1814 |
| Female | 1702 | |
| Racea | White | 2330 |
| Black | 95 | |
| Ethnicity | Hispanic | 473 |
| Non-Hispanic | 2974 | |
| School Sizeb | Small | 1234 |
| Large | 2293 |
does not include participants who selected more than one race or “other”
50% of schools were in the small category
For each subgroup, three levels of invariance (Configural, Pattern Identity, and Strong) were examined in sequential order. Configural Invariance, the first and weakest form of invariance assessed, is an unconstrained model for non-zero loadings and assumes the same number of factors and same items are associated with each factor across groups (Babbin et al., 2011; Harrington et al., 2011; Meredith, 1993; Ward, Velicer, Rossi, Fava, & Prochaska, 2004). The second level of invariance, Pattern Identity Invariance, requires equal factor loadings. The third and highest level, Strong Factorial Invariance, requires equal factor loadings and error variances.
Each level of invariance was assessed using the following fit indices: Comparative Fit Index (CFI), Normed Fit Index (NFI), Nonnormed Fit Index (NNFI) and Root Mean Square Error of Approximation (RMSEA). For CFI, NFI, and NNFI, a value ranging from 0 to 1 indicates how well the model fits the data, with larger values indicating a better fit. Values greater than .90 indicate good fit and values greater than .95 indicate very good fit (Kline, 2011; Bentler, 1992). For RMSEA, smaller values indicate better fit, with values less than .10 indicating good fit and values less than .05 indicating very good fit (Kline, 2011, Browne & Cudeck, 1993). In addition, the difference in CFI (ΔCFI) between a model and the next level of invariance was calculated to determine whether a null hypothesis of invariance should be rejected. A value of −0.01 or less indicates measurement invariance (Cheung & Rensvold, 2002).
Results
Gender
Sample size was adequate for comparing male (N = 1814) and female (N = 1702) subgroups. The highest level of invariance, Strong Factorial Invariance, produced a good model fit for gender (CFI = .961; NFI = .959; NNFI = .946; RMSE = .121; see Table 2).
Table 2.
Goodness-of–Fit Statistics for Three Invariance Models
| Model | NFI | NNFI | CFI | Δ CFI | RMSEA |
|---|---|---|---|---|---|
| Gender | |||||
| Configural Invariance | .965 | .936 | .966 | .131 | |
| Pattern Identity Invariance | .963 | .946 | .964 | −.002 | .121 |
| Strong Factorial Invariance | .959 | .946 | .961 | −.003 | .112 |
| Race | |||||
| Configural Invariance | .968 | .943 | .969 | .114 | |
| Pattern Identity Invariance | .966 | .953 | .969 | .000 | .104 |
| Strong Factorial Invariance | .958 | .946 | .961 | −.008 | .104 |
| Ethnicity | |||||
| Configural Invariance | .969 | .945 | .971 | .118 | |
| Pattern Identity Invariance | .968 | .953 | .969 | −.002 | .109 |
| Strong Factorial Invariance | .939 | .918 | .941 | −.028 | .140 |
| School | |||||
| Configural Invariance | .972 | .950 | .973 | .117 | |
| Pattern Identity Invariance | .972 | .959 | .973 | .000 | .106 |
| Strong Factorial Invariance | .970 | .961 | .972 | −.001 | .095 |
Race
Sample size was adequate for comparing white (N = 2330) and black (N = 95) subgroups. Strong Factorial Invariance produced a good model fit for race (CFI = .961; NFI: =.958; NNFI = .946; RMSEA: =.104; see Table 2).
Ethnicity
Sample size was adequate for comparing Hispanic (N = 473) and non-Hispanic (N = 2974) subgroups. Strong Factorial Invariance produced a good model fit for ethnicity (CFI = .941; NFI = .939; NNFI = .918; RMSEA = .140; see Table 2).
School size
Sample size was adequate for small schools (N = 1234) and large schools (N = 2293). Strong Factorial Invariance produced a good model fit for school size (CFI = .972; NFI = .970; NNFI = .961; RMSEA = .095; see Table 2).
Scale structure and reliabilities
Given that Strong Factorial Invariance held for each of the four cross-sample comparisons, the structure is reported only for the total sample (see Figure 1). The path coefficients ranged from .77 to .86 between the items and the first order factors indicating that all factor leading were very high. The correlation between the two first order factors was .93 indicating that there is substantial overlap. In the total sample, Coefficient Alpha for Positive Social and for Curiosity/Stress were .87 and .86, respectively.
Discussion
The six-item Temptations to Try Smoking Scale demonstrated a high level of factorial invariance across the subgroups examined in this study. While the RMSEA values for three of the strong factorial invariance models (gender, race, and ethnicity) were greater than .10, this was the only fit index examined that indicated a less than adequate fit for the models tested. The strong factorial invariance models for each of the subgroups examined returned fit indices of CFI, NFI, and NNFI around .95. The CFI, NFI, and NNFI fit indices for the four strong factorial invariance models were all above .90, which indicates acceptable fit of the model. The ΔCFI for three of the strong factorial invariance models (gender, race, and school size) was less than −0.01, indicating measurement invariance. These findings provide empirical support for the construct validity of the six-item Temptations to Try Smoking Scale for use with middle school students.
The Coefficient Alphas for the Positive Social and Curiosity/Stress subscales were good (.87 and .86, respectively). This finding provides support for the reliability of this short (three items per subscale) Temptations to Try Smoking Scale. When used as part of an intervention, this scale helps to keep participant burden low (i.e., few questions to respond to) while maintaining good reliability.
A major limitation of this study concerns the racial breakdown used for the invariance testing. Only white and black subgroups were used because sample sizes associated with other races were not sufficient to include in the analysis. Future research in this area would benefit from more diverse samples.
Highlights.
Measurement models should be equivalent across the different subgroups of a target population.
Temptations to Try Smoking Scale is a 6-item correlated model with 2 subscales: Positive Social Situations and Curiosity about Smoking/Stress.
Measure is part of a tailored prevention intervention for middle school students.
The strongest level of invariance, Strong Factorial Invariance, was a good fit for the model across all of the subgroups: gender, race, ethnicity, and school size.
Coefficient alpha was .87 for Positive Social Situations, and .86 for Curiosity about Smoking/Stress.
Acknowledgments
This paper was partially supported by Grant CA11919 from NCI and Grant DA020112 from NIDA.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- Babbin SF, Harrington M, Burditt C, Redding C, Paiva A, Meier K, Oatley K, McGee H, Velicer WF. Prevention of Alcohol Use in Middle School Students: Psychometric Assessment of the Decisional Balance Inventory. Addictive Behaviors. 2011;36:543–546. doi: 10.1016/j.addbeh.2011.01.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bandura A. Self-efficacy: Toward a unifying theory of behavior change. Psychological Review. 1977;84:191–215. doi: 10.1037//0033-295x.84.2.191. [DOI] [PubMed] [Google Scholar]
- Bandura A. Self-efficacy mechanisms in human agency. American Psychologist. 1982;37:122–147. [Google Scholar]
- Bentler PM. On the fit of models to covariances and methodology to the Bulletin. Psychological Bulletin. 1992;112:400–404. doi: 10.1037/0033-2909.112.3.400. [DOI] [PubMed] [Google Scholar]
- Bentler PM. EQS 6.1 structural equations program manual. Encino, CA: Multivariate Software; 2007. [Google Scholar]
- Browne MW, Cudeck R. Alternative ways of assessing model fit. In: Bollen KA, Long JS, editors. Testing structural equation models. Newbury Park, CA: Sage; 1993. pp. 445–455. [Google Scholar]
- Cheung GW, Rensvold RB. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling. 2002;9:233–255. [Google Scholar]
- Harrington M, Babbin SF, Redding C, Burditt C, Paiva A, Meier K, Oatley K, McGee HA, Velicer WF. Psychometric assessment of the temptations to try alcohol scale. Addictive Behaviors. 2011;36:431–433. doi: 10.1016/j.addbeh.2010.12.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jackson DN. A sequential system for personality scale development. In: Spielberger CD, editor. Current Topics in Clinical and Community Psychology. Vol. 2. New York: Academic Press; 1970. pp. 61–96. [Google Scholar]
- Jackson DN. The dynamics of structured personality tests: 1971. Psychological Review. 1971;78:229–248. [Google Scholar]
- Kline RB. Principles and Practice of Structural Equation Modeling. 3. The Guilford Press; NY, New York: 2011. [Google Scholar]
- Meredith W. Measurement invariance, factor analysis, and factorial invariance. Psychometrika. 1993;58:521–543. [Google Scholar]
- Prochaska JO, Velicer WF. The Transtheoretical Model of Behavior Change. American Journal of Health Promotion. 1997;12:38–48. doi: 10.4278/0890-1171-12.1.38. [DOI] [PubMed] [Google Scholar]
- Shiffman S. A cluster analytic classification of smoking relapse episodes. Addictive Behaviors. 1986;11:295–307. doi: 10.1016/0306-4603(86)90057-2. [DOI] [PubMed] [Google Scholar]
- Velicer WF, DiClemente C, Rossi JS, Prochaska JO. Relapse situations and self-efficacy: An integrative model. Addictive Behaviors. 1990;15:271–283. doi: 10.1016/0306-4603(90)90070-e. [DOI] [PubMed] [Google Scholar]
- Velicer WF, Prochaska JO, Fava JL, Rossi JS, Redding CA, Laforge RG, Robbins ML. Using the Transtheoretical Model for Population-based Approaches to Health Promotion and Disease Prevention. Homeostasis in Health and Disease. 2000;40:174–195. [Google Scholar]
- Ward R, Velicer WF, Rossi JS, Fava JL, Prochaska JO. Factorial Invariance and internal consistency for the decisional Balance inventory – short form. Addictive Behaviors. 2004;29:953–958. doi: 10.1016/j.addbeh.2004.02.042. [DOI] [PubMed] [Google Scholar]