Toward a More Systematic Assessment of Smoking: Development of a Smoking Module for PROMIS^®

Abstract

Introduction

The aim of the PROMIS^® Smoking Initiative is to develop, evaluate, and standardize item banks to assess cigarette smoking behavior and biopsychosocial constructs associated with smoking for both daily and non-daily smokers.

Methods

We used qualitative methods to develop the item pool (following the PROMIS^® approach: e.g., literature search, “binning and winnowing” of items, and focus groups and cognitive interviews to finalize wording and format), and quantitative methods (e.g., factor analysis) to develop the item banks.

Results

We considered a total of 1622 extant items, and 44 new items for inclusion in the smoking item banks. A final set of 277 items representing 11 conceptual domains was selected for field testing in a national sample of smokers. Using data from 3021 daily smokers in the field test, an iterative series of exploratory factor analyses and project team discussions resulted in six item banks: Positive Consequences of Smoking (40 items), Smoking Dependence/Craving (55 items), Health Consequences of Smoking (26 items), Psychosocial Consequences of Smoking (37 items), Coping Aspects of Smoking (30 items), and Social Factors of Smoking (23 items).

Conclusions

Inclusion of a smoking domain in the PROMIS^® framework will standardize measurement of key smoking constructs using state-of-the-art psychometric methods, and make them widely accessible to health care providers, smoking researchers and the large community of researchers using PROMIS^® who might not otherwise include an assessment of smoking in their design. Next steps include reducing the number of items in each domain, conducting confirmatory analyses, and duplicating the process for non-daily smokers.

1. INTRODUCTION

The theoretical constructs that are important to assess in smoking research and clinical practice are reasonably well defined; however, deciding how to assess each of these constructs is much less clear. Key constructs such as nicotine dependence, craving and nicotine withdrawal all can be assessed using myriad different scales and items. Instruments to assess other smoking-related constructs such as outcome expectancies, smoking motives, social influences, and nicotine effects are also abundant. The extensive diversity of smoking-related measures in the literature and lack of gold standard recommendations for their use present a challenge for researchers and clinicians (e.g., Niaura & Shadel, 2003; Shadel & Shiffman, 2005). For example, measure selection is often dictated by logistic factors that vary from study to study such as the time available for smokers to complete the measure, or by clinical heuristics for using assessments to guide treatment choice. One implication of this diversity is that the field lacks a comprehensive model for smoking assessment that: comprises a focus on core constructs; utilizes a well-defined set of reliable, validated, and non-overlapping items; is accessible and widely disseminated; and includes clear guidance for use. Such a model would serve as an important advancement to smoking research.

The issue of measurement choice is not new, nor is it unique to the field of tobacco control. Indeed, for assessment of patient reported outcomes (PROs),^I there is a wide range of generic and disease-specific instruments designed to measure constructs such as physical functioning, general health, and social functioning. Researchers studying PROs have been measuring these constructs with different questions based on different metrics, different time frames and different contexts, and as with tobacco assessment, choosing among available measures in similarly arbitrary or logistic-driven fashion. The main goals of PROMIS^® (Patient-Reported Outcomes Measurement Information System; http://www.nihpromis.org/default.aspx), which is part of the National Institutes of Health’s Roadmap initiative, are to standardize a set of assessment tools and to use modern measurement theory (i.e., Item Response Theory: see Edelen & Reeve, 2007 Jones & Thissen, 2007) and advances in computer technology to create item banks to measure PROs such that: the reliability for any subset of items within a given PRO bank can be calculated; scores based on different sets of items within the bank can be compared; items can be added and deleted as the understanding of each bank’s construct matures over time; and the use of tailored tests and computer based assessment enables minimization of respondent burden (Ader, 2007; Cella et al., 2007; Fries, Bruce, & Cella, 2005). PROMIS^® was developed, in part, to increase the availability and use of a common set of standardized assessment tools that long term would enhance the comparability of findings across studies examining patient-reported constructs, reduce respondent burden, and increase measurement precision. To date, item banks have been developed for the assessment of adult global health, physical function, symptoms of fatigue and pain, sleep/wake function (sleep disturbances, wake disturbances), and different facets of emotional and social health (emotional: depression, anxiety, anger; social: social function, social support) (Cella et al., 2007).

The goal of the PROMIS^® Smoking Initiative is to develop, evaluate, and standardize item banks to assess cigarette smoking behavior and biopsychosocial constructs associated with smoking, and incorporate these item banks into the PROMIS^® framework. The genesis of this effort sprung from the importance of considering smoking related constructs in assessing PROs and the fact that the PROMIS^® framework currently does not include smoking. However, the item banks produced by this initiative will be useful in a broad array of epidemiological research and clinical contexts. These item banks are intended to be relevant for both daily and non-daily adult smokers. Although we are ultimately interested in assessment among committed smokers as well as those in the process of quitting, this phase of the initiative focuses on assessment of smokers who are not planning to quit in the next 30 days.

In this article we describe a critical aspect of the PROMIS^® Smoking Initiative’s progress to date: how the 196 standardized items that comprise the preliminary PROMIS^® smoking item banks were selected from the thousands of smoking assessment items that exist across multiple conceptual domains, and how those items were combined to represent six theoretical domains central to the assessment of smoking behavior and the biopsychosocial constructs associated with smoking. Specifically, we describe the mixed method approach, an essential feature of PROMIS^® item bank development, adopted by the PROMIS^® Smoking Initiative that: 1) developed an item pool for field testing (qualitative phase); and 2) characterized the dimensional structure of the item pool in order to identify discrete assessment domains to be represented as item banks in the PROMIS^® framework (quantitative phase). Regular input into all aspects of the process was provided by an advisory board with experience developing PROMIS item banks, expertise in the psychometric methods required for item bank development, and clinical and research backgrounds relevant to smoking behavior that was assembled by the study team (see acknowledgements). The qualitative phase was conducted between December 2009 and February 2011; the quantitative phase began in September 2011 and is ongoing.

2. METHODS & RESULTS

2.1 Qualitative Phase

The qualitative phase included systematic literature review, binning and winnowing of items, qualitative item review, and final item revisions. These steps are described below; Figure 1 provides a schematic of the entire qualitative process.

Figure 1. Schematic representation of item pool development- qualitative phase.

This figure describes the process for sorting the 1622 smoking items into 11 bins. Boxes represent steps in the process, solid arrows represent the order of work, and dashed arrows and dashed boxes indicate the removal of items and bins at each step of this binning and winnowing process.

2.1.1 Systematic Literature Review

Following procedures adopted in the initial PROMIS^® effort, we developed the item pool by building on existing items as much as possible. Our initial search for items and scales was grounded in several contemporary reviews of the smoking assessment literature (Baer & Lichtenstein, 1988; Niaura & Shadel, 2003; Shadel & Shiffman, 2005; Shiffman, 1988). These reviews were used in two ways. First, these reviews provide lists of common (and some less commonly used) measures in the smoking research and cessation literature. All of the items, measures, and scales from these reviews were included in the item pool. Second, these reviews collectively provided a guiding conceptual framework that we used to highlight the key smoking domain components that are important for smoking research and treatment. Names for these conceptual domain components (e.g., dependence, craving, etc.) were used as keywords to perform a comprehensive search of the literature from the years 1970–2009 using PubMed, EBSCO, PsycINFO and Google Scholar databases. This search served to enrich the pool of potential smoking scales and items beyond those drawn from the assessment reviews cited previously. The Centers for Disease Control and Prevention Question Inventory on Tobacco Control (http://apps.nccd.cdc.gov/QIT/QuickSearch.aspx) and National Cancer Institute Tobacco Control Research branch (http://www.cancercontrol.cancer.gov/tcrb/) were also searched for non-overlapping content in these conceptual domains. Finally, we examined the “Methods” sections of all smoking-related articles that appeared in several high profile health and addictions journals (from January 2000 – January 2010: Journal of Abnormal Psychology; Journal of Consulting and Clinical Psychology; Health Psychology; Psychological Assessment; Assessment; Quality of Life Research; Psychology of Addictive Behaviors; Nicotine & Tobacco Research; and Addictive Behaviors) and collected any smoking items or measures that we had not yet obtained. As a final step, this item and measure list was reviewed by members of the advisory board to ensure comprehensiveness of the included items. The identified instruments (both tobacco-specific questionnaires and general health surveys) collectively resulted in an initial pool of 1622 items. We entered all items into a standardized item library that recorded the context, stem, response options, time frame, and instrument of origin for each item.

2.1.2 Creating a Smoking Item Database: Binning and Winnowing

Next, we followed the process of binning and winnowing that was developed and implemented by the initial PROMIS^® team (DeWalt et al., 2007). Two study team members independently sorted the 1622 smoking items into bins (“binning”) according to meaning and specific latent construct (without reference to the original source or construct indexed by those items). At the end of this process the two team members compared their resultant bins, discussed any discrepancies that occurred and, in cases where they could not resolve a discrepancy, a third and fourth team member were brought into the discussion until a resolution was found. This process resulted in a total of 1344 items being sorted into 33 bins (see left box in Figure 1)

To reduce of the number of items and bins (“winnowing”) the team reviewed the contents of each bin and removed: 1) items/bins that were not central to the maintenance of smoking, severity of habit, or barriers to quitting; 2) items within bins that contained semantic redundancy; and 3) items/bins that were not universally applicable to our target population of current adult smokers. All decisions about item winnowing were discussed by two study team members and reviewed by a third team member. Dashed arrows and dashed boxes in Figure 1 indicate the removal of items and bins at each step of this binning and winnowing process.

As a final step in the winnowing process we reorganized remaining bins and items by study team consensus. Specifically, we combined several bins with conceptually similar content and renamed some bins. The resultant set of 11 bins (259 items), which was reviewed and approved by the advisory board, is shown in the center box of Figure 1.

2.1.3 Qualitative Item Review

2.1.3.1 Item Revision

Next, we used a three-stage process to refine the items. In the first stage, we standardized item stems so that they were all worded in the present tense and had a first-person orientation; modified the time frames, as necessary, so that most items had either a past 30 day time frame or no time frame; and modified the response options according to PROMIS^® guidelines such that most items would be rated on one of two 5-point scales: 1=never to 5 = always; or 1 = not at all to 5 = very much. In the second stage, we focused on changing the wording of item stems to increase clarity, avoid slang, and break double-barreled items (i.e., items that assess more than one concept) into separate items. In the third stage, we asked a translation expert to review the items for translatability into Spanish.

2.1.3.2 Focus Groups

Four focus groups were conducted in Boston (n=12), Pittsburgh (n=9), and Los Angeles (n=10, 7). Participants were recruited through local vendors in each city. Individuals were eligible if they were 18 or older, had been smoking for at least a year, had smoked in the past 30 days and did not have plans to quit in the next 30 days. Respondents were chosen to represent variability in smoking behavior (non-daily and daily smokers), age, gender, ethnicity, marital status, employment, education and income. To ensure adequate representation of heavier smokers, the second group in Los Angeles consisted of individuals who smoked at least 10 cigarettes a day. Characteristics of the focus group participants are shown in Table 1.

TABLE 1.

Characteristics of Focus Group (N=38), Cognitive Interview (N=66), and Field Test (N=3021) Participants

Characteristic	Focus Groups (N=38)	Cognitive Interviews (N=66)	Field Test (N=3021)
	Mean, Median or Percentage	Mean, Median or Percentage	Mean, Median or Percentage
Female, %	46%	51%	54.85% (N=1657)
Race/Ethnicity, %
Non-Hispanic White	57%	47%	72.05%
African American	24%	18%	12.02%
Hispanic	11%	22%	11.68%
Asian	5%	3%	1.87%
Other	3%	7%	2.38% (N=72)
Age, mean (range: 19–65)	42.3 (SD =13.3)	40.9 (SD=11.6)	46.28 (SD=11.63)
Education, %
<High school graduate	5%	2%	3.19%
High school graduate	22%	11%	17.16%
Some college	27%	40%	37.50%
AA degree	22%	13%	12.55%
BA/BS degree	19%	20%	17.59%
Graduate degree	5%	15%	12.01% (N=363)
Employment
Full-time	62%	69%	53.33%
Part-time	11%	5%	12.02%
Unemployed/retired/student/homemaker	27%	13%	33.80%
Missing	0%	13%	.86% (N=26)
Income
< $20,000	19%	15%	22.35%
$20,000–$50,000	47%	36%	32.59%
$50,000–$100,000	22%	33%	33.07%
$100,000+	11%	16%	11.99% (N=362)
Smoking Patterns
Number of days smoked in past 30, median	25( range 0–30)	30(range 3– 30)	N/A
Number of cigarettes per day
Less than daily	32%	36%	.13% (N=4)
1–5	19%	22%	8.10%
6–10	24%	22%	22.46%
11–19	5%	4%	46.72%
20+	19%	16%	22.59%
Number of times quit for at least 24 hours
0	35%	45%	18.16%
1	24%	15%	12.53%
2–5	27%	24%	50.35%
≥6	14%	16%	18.96%
Quitting contemplation
Not thinking about quitting	11%	27%	39.69%
Thinking about quitting, but no plans	53%	53%	37.70%
Plans to quit in next 6 months	36%	20%	22.61% (N=683)

The focus groups’ main goal was to identify gaps in item coverage. The focus group participant discussions were moderated to solicit information on the validity of the smoking domains currently in our database and identify any domains that might be missing. The discussions were audio-recorded, professionally transcribed, and organized into broad themes. Two team members reviewed these themes to identify content that was not represented by existing items. Three team members generated an additional 44 candidate items to capture identified gaps in content coverage.

2.1.3.3. Cognitive Interviews

One-on-one cognitive interviews with smokers were conducted to identify potential problems in item wording and response options. Sixty-six participants were recruited in Los Angeles with the same inclusion criteria and respondent variability as in the focus groups. Characteristics of the cognitive interview participants are shown in Table 1. All items were directly probed for comprehension, confidence in response, and ease of recall. In addition, we tested time frame (e.g., 30 day vs. 7 day vs. no time frame) and scale anchors. The cognitive interviews identified several items as potentially problematic; these items were discussed by the study team and some were removed. Interviews also indicated that a 7-day time frame was too short to capture the experiences of non-daily smokers, but there was little difference in preference for a 30-day and no time frame. Thus, with the exception of a few special items that required a 12 month time frame, all items were presented with no time frame context.

2.1.4 Final Item Revisions

As a final step prior to field testing, the study team reviewed the large set of candidate items to reduce the item pool where possible (i.e., based on redundant content and potential problems in either the translation review or the cognitive interviews). The box in the bottom right corner of Figure 1 shows the final list of conceptual domains (i.e., bins) and the number of items in each, and provides an example item for each domain. At this stage, we contacted the authors of source instruments and obtained permission to use content from their instruments as described to generate the smoking item pool for field testing. The source instruments associated with the fielded items are available by request to the corresponding author.

2.2 Quantitative Phase

2.2.1 Field test

The 277 items were fielded between July and September 2011 via a randomized block design (Reeve et al., 2007) to a large national sample of smokers (N_(total)=5384; N_(daily)=4201; N_(non-daily)=1183). The block design was constructed to minimize respondent burden while maximizing the inter-item covariance coverage. All respondents supplied basic demographic information and completed thirteen smoking items assessing usage and quitting history; respondents additionally completed one of 26 overlapping forms containing an average of 147 smoking items (range=134–158). Finally, one of 8 randomly selected PROMIS short forms (selected to characterize sample and conduct known groups validation) was administered to each respondent. The sample was recruited by Harris Interactive through their online panel membership, and all assessments were completed via the internet. Eligibility criteria were identical to those used in the qualitative phase. Sample recruitment was targeted to reflect the demographic composition of US adult smokers.

The analyses reported here are based on data from a subset of N=3021 daily smokers. A random subset of the daily smokers (N=1180) has been set aside as a validation sample and will be used in subsequent analyses not reported here. Data from the non-daily smokers (N=1183) will also be evaluated separately. Sample characteristics are described in the rightmost column of Table 1.

2.2.2 Analyses

Our analytic approach aimed to identify distinct item banks assessing key smoking behavior domains. Figure 2 provides a schematic representation of our quantitative analyses. As can be seen in Figure 2, and is described further below, our interpretation of results and decision processes were guided by important qualitative methods throughout this analytic phase. We first conducted exploratory analyses using the domains from the qualitative phase as a basis for grouping the items. These analyses revealed substantial covariance among domains and among items across domains. Based on this evidence of overlap in conceptual content, we concluded that subsequent analyses should “start from scratch” in characterizing the factor structure of the item set.

Figure 2. Schematic representation of item bank development- quantitative phase.

This figure depicts the factor analytic approach and decision process used to identify the six smoking domains to be represented as item banks

To achieve this we began by conducting exploratory factor analysis (EFA) of all the items using IRTPRO’s (Cai, du Toit, and Thissen, 2011) EFA module with MH-RM estimation (Cai, 2010) and Quartimin rotation. The EFA was conducted to determine the total number of factors present and to set aside items that were not accounted for by any single factor. In deciding on the optimal number of factors to extract, we favored solutions in which the majority of items loaded ≥ 4 on one and only one factor. This served to minimize the number of items with substantial cross-factor loadings (i.e., ≥ 4 on a secondary factor), or without strong loadings on any one factor (i.e., <.4 on all factors). Several solutions that met these empirical criteria were further evaluated and we selected the solution that was deemed most meaningful in terms of substantive content and utility. Thus we selected a 19-factor solution that incorporated 226 of the items. Items that either had strong cross-loadings or did not load strongly on any factor were set aside for later consideration (see Figure 2).

Next, we sought to reduce the dimensionality of the 19 factor solution by exploring whether sets of factors from this solution could be combined together to form higher-order factors. The technique is based on the assumption that high inter-correlations among subsets of factors suggest only mild domain-level multidimensionality. The advantage of this approach is that it allows for unidimensional item banks with diverse content to be created by integrating these multiple factors with overlapping, but not identical themes.

To do this, we individually and then collectively evaluated the matrix of factor inter-correlations obtained from a confirmatory factor analysis model of the 19-factor solution. Though the average inter-factor correlation was r = 0.40, close evaluation of factor subsets indicated a small number of factor clusters (usually containing three or four factors each) with inter-correlations ranging from about r =.50 to r =.80. For example, three factors reflecting various positive aspects of smoking (smoking is energizing, smoking feels pleasant, and smoking makes me feel good) had inter-correlations of.87,.71, and.63 indicating that they contained highly related content. The research team identified several of these factor clusters and discussed their content in order to evaluate the desirability of a bank representing this content.

This process resulted in six factor cluster groups with substantively meaningful content: Positive Aspects of Smoking, Dependence/Craving, Health Consequences of Smoking, Psychosocial Consequences of Smoking, Coping Aspects of Smoking, and Social Factors of Smoking (see Figure 2). Next, we examined each groups’ composition and removed items with content that was redundant or departed from the theme of the group. Finally, items that had been set-aside earlier were qualitatively reevaluated and several were incorporated into one of the six groups, resulting in six preliminary PROMIS^® smoking banks consisting of 211 items. Based on these analyses, we consider the six banks to be final in terms of their respective domains and scope of content. However, additional analyses are necessary to refine each bank’s contents so that they are strongly unidimensional and items are invariant with respect to basic demographic groups (gender, age, ethnicity) and smoking patterns (past month quantity smoked, motivation to quit).

3. DISCUSSION

The goal of the PROMIS^® Smoking Initiative is to improve smoking assessment by: (1) using state-of-the-art methods to develop item banks representing smoking behaviors and biopsychosocial correlates associated with smoking; and (2) make these item banks readily available to health care providers, general health researchers and to those dedicated to research on smoking and cessation. In this paper, we described the extensive qualitative effort we undertook to develop an initial item pool for field testing, and the first phase of quantitative analyses used to identify six distinct smoking domains that will form the basis for the PROMIS^® smoking item banks. The process undertaken employed recognized procedures that were developed and established by PROMIS.

Limitations of this approach need to be considered. Although we started with extant items and scales, the majority of items in the final pool have been revised either through minor wording changes, different response categories, or different time frame contexts. Further, they are being administered individually via a web-based computer interface. These changes have likely impacted item function; however, they seem unlikely to have resulted in poorer functioning given the careful editing process employed. Additionally, we limited the scope of the fielded item pool to focus on adult daily and non-daily smokers who do not have any immediate plans to quit (i.e., not within the next 30 days). This definition led to the removal of items pertaining to concepts such as quitting, nicotine withdrawal, and others. Although these items were removed, it will be straightforward to expand the scope of the item banks in any of these directions once the initial set of item banks is fully developed and validated.

This research also has several strengths. Our qualitative phase employed a rigorous review process that conformed to the standards of PROMIS^® and obtained input from multiple sources (layperson, expert panel). Additionally, the inclusion of item review by a translation expert lays the groundwork for eventual field testing of a Spanish language version of the smoking item banks. Our quantitative item evaluation employed state-of-the-art psychometric techniques to optimize the composition of the final item banks and facilitate updating as needed over time. Another notable strength is the ongoing feedback throughout both phases from an advisory board with considerable collective expertise in PROMIS^® item bank development, psychometrics, and smoking-related issues (from both research and clinical perspectives).

Approaching assessment of smoking in this way has the potential to permit a substantial advance over and above extant approaches. Indeed, a major problem with smoking assessment is the diversity of measures that exist to assess various aspects of smoking behavior. This overabundance of measures complicates efforts to develop a more comprehensive assessment model and makes it challenging for researchers to choose the best measures for a given purpose. For example, there are at least eight assessments of nicotine dependence from which to choose, ranging from probably the earliest self-report instrument, the Fagerstrom Tolerance Questionnaire (FTQ; Fagerstrom, 1978), to newer assessments such as the Nicotine Dependence Syndrome Scale (NDSS; Shiffman et al., 2004) and the Wisconsin Inventory of Smoking Dependence Motives (WISDM; Piper et el., 2004). Nearly 150 items (not always distinct) are used to assess this core construct across instruments. The comprehensive mixed methods approach taken by the PROMIS^® Smoking Initiative considered the possible inclusion of all of these well-known instruments yet succeeded in reducing this item set by nearly 2/3 and in a way that systematically eliminated redundancy and retained a core set of the highest performing items that can be used to assess nicotine dependence and its core conceptual features. For example, despite this extensive reduction of total items, our field test included 37 items modified from the WISDM. Further, the item-banking approach will enable researchers to assess the dependence construct with great precision while minimizing respondent burden. Other advantages of item banking (e.g., scalability to existing dependence measures, facilitation of results across studies) mentioned earlier and elsewhere (e.g., Cella et al., 2007) also accrue.

Analyses of the field test data are ongoing. Our expectation is that further domain-level factor models will need to be estimated in order to reduce each of the six preliminary item banks to a core set of unidimensional items. Accordingly, immediate next steps are to reduce the final composition of the six item banks. Once this is completed, we will use data from the validation sample to confirm the structure of these banks. We also plan to evaluate this six group solution among the non-daily smokers. Although the basic structure is likely to adequately characterize the data from the non-daily smokers, we expect that several irrelevant items in the daily smoker analyses will emerge as important components of the six groups for the non-daily sample. Once these analyses are complete, the finalized smoking item banks will be incorporated into the larger PROMIS^® framework and will be easily accessible through Assessment Center (http://www.assessmentcenter.net), a widely used and user-friendly resource for creating, administering and scoring PROMIS^® domains and incorporating PROMIS^® measures into research study designs.

Highlights.

• We use a mixed methods approach to develop six item banks for smoking assessment.

• We followed PROMIS’s state-of-the-art procedures for item bank development.

• We developed six item banks representing distinct biopsychosocial smoking domains.

• Item banks will be easily accessible via the PROMIS assessment center.

• Wide utilization of the item banks will facilitate improved smoking assessment.