Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Jan 26.
Published in final edited form as: Addiction. 2019 Aug 18;114(11):1926–1940. doi: 10.1111/add.14736

Behavioral Economic Tobacco Demand in Relation to Cigarette Consumption and Nicotine Dependence: A Meta-Analysis of Cross-sectional Relationships

Alba González-Roz 1,2, Jacob Jackson 2, Cara Murphy 3, Damaris J Rohsenow 3, James MacKillop 2,3
PMCID: PMC7837316  NIHMSID: NIHMS1652928  PMID: 31313403

Abstract

Background and Aims:

A cigarette purchase task (CPT) aims to characterise individual variation in the reinforcing value of tobacco. This meta-analysis estimated the associations between cigarette demand, tobacco consumption, and nicotine dependence using this task.

Design:

A meta-analysis of cross-sectional studies identified by PubMed and PsycINFO databases was conducted. Fixed and random effects models were used. The study also examined the model used to derive elasticity of demand (exponential or exponentiated) as a potential moderator. Publication bias was assessed using fail-safe N, Begg-Mazumdar test, Eggeŕs test, Tweedie’s trim and fill approach, and meta-regression of publication year with effect size.

Setting:

Studies from any setting that reported coefficient correlations on the tested associations.

Participants:

Daily cigarette users (i.e., 5–38 cigarettes per day; N=7,649).

Measurements:

Cigarette consumption, nicotine dependence and five tobacco demand indicators: intensity (i.e., consumption at no cost), elasticity (i.e., sensitivity to rises in costs), Omax (maximum expenditure), Pmax (i.e., price at which consumption becomes elastic), and breakpoint (i.e., price at which consumption ceases).

Findings:

Twenty-three studies met inclusion criteria. All the CPT indices were significantly correlated with smoking behavior (r’s = .044–.572, ps = .012 – 10−8). Medium-to-large effect size associations were present for intensity, Omax, and elasticity, whereas small effects were obtained for breakpoint and Pmax. Evidence of a moderating effect of the different elasticity modelling approaches was not present. There was limited evidence of publication bias.

Conclusions:

All indices derived from the Cigarette Purchase Task (cigarette consumption, nicotine dependence, and five demand indices) were robustly associated with cigarette consumption and tobacco dependence. Of the demand indices, maximum expenditure, intensity and elasticity exhibited the largest magnitude associations.

Keywords: Tobacco, Nicotine, Smoking, Cigarette, Demand, Purchase Task, Reinforcing Value, Behavioral Economics, Meta-analysis

Introduction

A behavioral economic account of addiction is an extension of operant learning theory that emphasizes the importance of the reinforcing value of a drug and the availability of alternative reinforcers as prepotent determinants of addictive behavior (1, 2). The approach has similarities to other contemporary theoretical approaches but incorporates other biological, cognitive, and sociocultural factors via these common pathways. Historically, the reinforcing value of a drug has been examined using behavioral operant paradigms, such as progressive-ratio schedules of drug use contingent upon specific responses (e.g., button presses, plunger pulls) (e.g., 37). This approach generates behavioral data that can be examined using microeconomic demand curve analysis, generating several motivational indices. As shown in Figure 1, these include intensity (i.e., consumption at zero cost), Omax (i.e., maximum expenditure across prices), elasticity (i.e., sensitivity of demand to increases in costs), Pmax (i.e., price at which demand is maximized or becomes elastic) and breakpoint (i.e., the price that suppresses consumption to zero).

Figure 1.

Figure 1.

Prototypic cigarette demand and expenditure curves. Black circles reflect mean demand. Intensity describes consumption at no cost or very inexpensive prices and displays a positively decelerating trend as a function of unit price. Omax and Pmax indicates the maximum expenditure and price at which consumption is maximized, respectively. Elasticity (i.e., slope) is a proxy of the rapidly the intensity decreases as a function of unit price increases.

The value of demand curve analysis is that these different indices provide a comprehensive assessment of drug reinforcing value by quantifying, for example, unconstrained consumption, maximum output (expenditure), or price sensitivity (8). This is important because divergent findings are present across different CPT indices, suggesting that there is no single absolute index of reinforcement efficacy (912). Rather, it seems that a comprehensive perspective is needed because each of these parameters provides distinct motivational information. The two ends of the demand curve provide good examples of this. Intensity reveals an individual’s unconstrained appetite for the drug, how much they would consume if the response cost was entirely eliminated, whereas breakpoint reveals how far they would go to avoid entirely forgoing the drug. Between intensity and breakpoint, Omax uniquely addresses how much the individual is willing to give up in total, while elasticity and Pmax both speak to a person’s overall sensitivity to the cost of consumption.

More recently, ‘purchase tasks,’ or self-report measures that ask a participant to estimate the number of cigarettes they would consume at escalating financial costs (13), have been increasingly used in tobacco research. Specifically, participants identify their preferred consumption in the consent of an instructional set in which the cigarettes available are their preferred brand, they cannot stockpile cigarettes for another time, they do not have access to any other nicotine products, and they have the same income as they have in their life. Unlike operant behavioral laboratory measures, purchase tasks remove a number of experimental burdens (i.e., multiple assessment sessions, high participant compensation) and potential ethical problems among treatment-seeking individuals (14). In particular, the use of a Cigarette Purchase Task (CPT) has permitted the assessment of the reinforcing value of cigarettes in novel ways. For example, a standard CPT assessing preferences for a typical day (trait-level demand) could inform tobacco pricing and tax policy by using many more prices than would ever be feasible for an operant paradigm (15). As a result, there is considerable interest in using demand measures to examine individual variation in cigarette reinforcement (1619) and as predictors of treatment response (2022). In the latter case, both baseline and in-treatment demand changes have been shown to predict early treatment response over and above other smoking-related characteristics (such as cigarettes per day, nicotine dependence and number of negative CO samples) (2022), suggesting they provide complementary motivational information.

Similarly, in laboratory studies, a state-oriented CPT has been used to complement traditional measures of subjective craving (i.e., typically based on subjective desire estimations) by using more objective units (i.e., cigarettes consumed or money spent) (23, 24). These studies have revealed that subjective craving is variably correlated with motivation in behavioral economic units [(e.g., For example, nicotine withdrawal was found to increase most the focal demand indices (intensity, Omax, breakpoint, Pmax), but not the overall elasticity, while the presence of environmental smoking cues selectively attenuated elasticity)]. Thus, the task indices provide insights into how acute changes in tobacco motivation manifests in different ways. Purchase tasks have also been used in relation to varenicline, a smoking cessation pharmacotherapy (2528), to generate mechanistic insights into how the medication differentially affects the different indices.

Fundamentally, the CPT was developed to produce multiple indicators of reinforcing value that are valid and informative mechanisms in understanding the development and treatment of nicotine dependence. From a theoretical standpoint, the demand indices as novel putative determinants would be expected to be associated with the traditional health indicators, such as cigarettes per day or level of dependence. Furthermore, the CPT indices would be expected not to be associated with smoking-involvement to the point of collinearity, which would render them redundant, and they would be expected to demonstrate incremental contributions. If CPT indices fall short in these preceding areas, their utility would be limited. To a large extent however, these empirical relationships have been observed in the emerging literature. However, as the number of studies using the CPT have increased, the findings have become more heterogeneous in terms of the relationship between cigarette demand and the traditional indicators of tobacco involvement (e.g., level of consumption, nicotine dependence severity). Given the heterogeneity of findings and highly varied magnitudes of relationships observed between each demand index and smoking behavior, the goal of this study was two-fold: 1) to meta-analyze the findings on the cross-sectional studies assessing the relationship between CPT demand indices and both cigarettes per day and nicotine dependence severity; and 2) to evaluate the presence of ‘publication bias,’ also called small study bias, or the extent to which the published literature appears to over-represent small studies with significant findings, reflecting a higher likelihood of significant findings in underpowered studies to be published compared to nonsignificant findings. Fundamentally, given its relatively rapid adoption, these aims collectively sought to evaluate the concurrent validity of the CPT as a measure of tobacco reinforcing value.

Methods

Study selection and eligibility criteria

Studies were identified through literature search using the PubMed and PsycINFO databases as of September 14th, 2018. The specific Boolean search terms were: (cigarette OR cigarettes) AND (“demand” OR “purchase task” OR “reinforcing efficacy “or “reinforcing value” OR “reward value”). The main inclusion criterion was any peer-reviewed published cross-sectional study reporting relationships between indices of hypothetical cigarette demand, cigarette consumption (i.e., cigarettes per day) and nicotine dependence. As several different models can be used to derive elasticity (29, 30), only effect sizes for analyses that used the contemporary and commonly used exponential and exponentiated models (31, 32) were included for methodological consistency. Studies were excluded if the results were overlapping with previously published papers or if they were otherwise not relevant to the study question. On the theoretical basis of investigating demand as a general trait-like characteristic and because the health outcomes were trait-like tobacco indicators, laboratory studies that investigated state-level demand in relation to experimental manipulations were excluded. In total, six studies using a state-level CPT were excluded. Our primary reason for this is that under experimental manipulations (i.e., stress or negative mood inductions), demand indices are altered, thus leading to substantial deviations from the demand observed under non-experimental conditions (33). The meta-analysis was registered in the PROSPERO system for systematic reviews (ID: CRD42018109472) and conducted in accordance with the Preferred Reporting Items for Systematic Reviews (PRISMA statement) (34).

Data extraction

Two independent reviewers conducted the literature search and coded the studies independently. No disagreement in the number of included studies occurred between reviewers (100% concordance). Data were abstracted on the following variables: study country, participant characteristics (i.e., sample size, sex ratio, mean age, mean number of cigarettes per day, and nicotine dependence severity), structural characteristics of the CPT (i.e., prices, range of prices), measures of cigarette consumption and nicotine dependence, details on data processing (i.e., model used to derive the elasticity index and identification/management of non-systematic data), and outcomes (i.e., effect sizes of the assessed relationships). Data were solicited from authors of studies that did not report viable data to meet the inclusion criteria. In six cases, raw data were provided and were analyzed to permit inclusion in the meta-analysis.

Meta-analytic sample

A total of 931 records were identified through the literature search, of which 718 were unique. Initial screening of these articles revealed 61 relevant studies. Upon completion of 61 full-text reviews for eligibility, 22 papers with 23 studies were identified to meet inclusion criteria. A flow diagram conforming with Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (34) is provided in Figure 2.

Figure 2.

Figure 2.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Meta-analytic approach

Pearson’s r and Spearman’s ρ correlations from each study were converted to Fishers’ Z. To systematically compare the relationship of each CPT index (intensity, Omax, Pmax, elasticity and breakpoint) and the different tobacco indicators, we conducted a set of meta-regression analyses and reported: a) unique effect sizes for the correlations between each CPT index and cigarette consumption, and b) unique effect sizes for the correlations between each CPT index and nicotine dependence. The primary meta-analyses include data supplied by authors on our request, processed following the procedure specified in the original studies. Among the studies from which raw datasets were provided, we identified two that did not comprehensively address non-systematic data. For clarity, we reran a set of separate meta-analyses to inform if any variation resulted from using the Stein et al. (35) procedure (see Table S1 for details on the standard data processing applied).

Both fixed and random effects approaches were considered, but a random effects approach was selected as the primary method considering methodological heterogeneity across studies (e.g., different number of CPT items and prices). Fixed effects are reported for completeness and to characterize heterogeneity of effect size, defined using Cochran’s Q and I2. Cochran’s Q statistic reflects the sum of square differences among the individual weighted study effects and the overall mean; Q tests the significance of observed heterogeneity using a χ2 test. I2 reflects the percentage of study effect size variation that is explained by heterogeneity; I2 ≤ 25% suggests low heterogeneity, ~50% suggests moderate heterogeneity, and ≥75% suggests high heterogeneity across studies (36). Q statistic associated with the between groups difference in mixed effects analyses were reported for moderator analyses. Specifically, we examined systematic differences based on smoking behavior variable type (i.e., cigarette consumption vs. nicotine dependence) and model selection to derive the elasticity index (i.e., exponential vs exponentiated). Follow-up ‘jackknife’ analyses were conducted to evaluate the estimated effect size with each effect size excluded. To the extent that using any single measure of publication bias provides an incomplete perspective (37), we considered five measures: (a) the fail-safe N approach (i.e., number of missing studies that would render the observed effect sizes non-significant, with N values lower than 5k (number of included studies) +10 raising concerns) (38), (b) the two-tailed Begg-Mazumdar test (i.e., rank correlation between the standard effect size and their variances, with deviations from zero indicating the presence of publication bias), (c) the one-tailed Egger’s test (i.e., asymmetry of the funnel plot with intercept values close to zero indicating lesser publication bias), (d) The Tweedie’s trim and fill approach (i.e., computation of the effect sizes after imputation of missing studies) and, (e) meta-regression of the relationship between year of publication and effect size. Effect size coefficients were interpreted as per the Cohen (39) benchmarks: Coefficients ≤0.2 and in the range of 0.21 to 0.49 indicate a small effect; coefficients between 0.50 and 0.79 indicate a medium effect; and coefficients ≥0.8 indicate a large size effect. Analyses were conducted using SPSS 24 (SPSS Inc., Chicago IL, USA), GraphPad Prism 7.0 (La Jolla, California) and Comprehensive Meta-analysis 2.0.

Results

Study characteristics

Individual study characteristics are in Table 1 and illustrate considerable variation in study parameters. Sample sizes included ranged considerably, from N=11 to N=1,215 (total N = 7,649), with a median sample size of 128. The average age within studies ranged from 16.50 to 52.27 years old, with a median study age of 33.90. Sex ratios ranged from 18% to 62% female. Substantial differences were present in terms of the CPT parameters, with highly variable numbers of prices (range = 5–73 prices) and price amounts (maximum price range = US$1.00–US$1,120). The majority of studies (n = 17/23; 74%) were conducted in the United States, with the others being conducted in the United Kingdom, Spain, New Zealand, and Netherlands.

Table 1.

Study characteristics.

Study Country Sample Size Sex Ratio (F:M, %F) Mean Age Cigarette Purchase Task Tobacco Consumption Measure Nicotine Dependence Measure Elasticity model Management of non-systematic purchase data
Chase et al. 2013, Study 1(10) United Kingdom 241 123:118 49% 20.92 (1.54) 39 Prices: 2p – •5 ($0.03 – $7.50) C/D 6.49 (4.57) CDS-5 11.03 (4.44) Exponential Inspection of inadequacy of data for modelling elasticity (i.e., zero demand at the lowest price and fewer data points than parameters to fit)
Chase et al. 2013, Study 2(10) United Kingdom a100 a47:53 47% a20.72 (1.31) 39 Prices: 2p – •5 ($0.03 – $7.50) C/D 6.33 (4.44) CDS-5 11.73 (4.26) Exponential Inspection of inadequacy of data for modelling elasticity (i.e., zero demand at the lowest price and fewer data points than parameters to fit)
Dahne et al. 2017(16) USA 73 30:43, 42% 19.7 (1.15) 48 Prices: $0 – $9 CPSD 7.43 (4.72) mFTQ 3.90 (1.41) Exponential Inspection of inadequacy of data for modelling elasticity (i.e., R2 ≤30, missing data or not reported consumption)
Few et al. 2012(48) USA 11 2:9, 18% 36.5 (11.2) 73 Prices: $0 – $10 C/D 22.3 (7.5) FTND - Exponential No
Grace et al. 2015 (41) New Zealand 357 186:162, 53% b53.2 64 Prices: NZ$0 – NZ$5.00 ($0 – $3.75) C/D 14.76 (8.62) FTND 4.18 (2.20) Exponential No
Heckman et al. 2018(49) Netherlands 1,215 625:590 49% 38 (18) 10 Prices: €0-€30 ($0-$33) C/D 14 (7) HSI % low (89) % high (11) Exponential Stein et al. 2015(35) three criterion algorithm
MacKillop et al. 2008 (46) USA 33 14:19, 42% 19.30 (1.61) 19 Prices: $0 – $1,120 C/D 5.31 (7.36) FTND 1.90 (2.02) Linear Inspection of erratic responding and inadequacy of data for modelling elasticity (i.e., ≥1data point)
MacKillop et al. 2012(15) USA 1,056 412:644, 39% 31.59 (12.70) 73 Prices: $0 – $10 C/D 16.51 (10.95) FTND 4.17 (2.50) Exponential Inspection of missing data (i.e., ≥10 items) and low effort (i.e., ≥3 contradictions)
MacKillop et al. 2016 (21) USA 338 108:230, 32% 37.6 (10.0) 41 Prices: $0 – $35 C/D 37.6 (10) FTND 5.9 (1.9) Exponential No
MacKillop … Tidey, 2011(11) USA 24 10:14, 42% 43.58 (12.35) 19 Prices: $0 – $1,120 C/D 24.17 (6.57) FTND 6.92 (1.38) Exponential Inspection of erratic responding and inadequacy of data for modelling elasticity (i.e., ≥1data point)
McClure et al. 2013 (25) USA 47 24:23, 51% 44.0 (11.0) 18 Prices: $0 – $1,120 C/D 19.3 (5.5) FTND 5.55 (1.7) Exponential No
Murphy et al. 2011(9) USA 138 68:70, 49% 16.50 (14.08) 26 Prices: $0 – $1,120 C/D 5.97 (5.99) mFTQ 3.85 (1.88) Exponential Inspection of inadequacy of data for modelling elasticity (i.e., R2 ≥30)
O'Connor et al. 2016 (50) USA 1,114 613:501, 55% 18–34 (42%) 12 Prices: $0 – $11 C/D - FTND - Exponentiated Inspection of inadequacy of data for modelling elasticity (i.e., R2 ≤30)
Secades-Villa et al. 2016(20) Spain 159 99:60, 62% 44.8 (12.6) 19 Prices: €0 – €1000 ($0 – $1120) C/D 21.1 (8.5) FTND 5.5 (1.9) Exponential Inspection of >2 contradictions at escalating prices
Schlienz et al. 2014 (26) USA 52 32:20, 62% 48.3 (9.65) 19 Prices: $0 – $1120 C/D 17.55 (4.8) FTND 5.2 (2) Exponential Reversals were replaced by zero.
Smith et al. 2017(51) USA 839 358:481 43% 41.7 (13.2) 17 Prices: $0- $5 C/D 15.56 (7.60) FTND 5.14 (2.21) Exponentiated Inspection of increases from one price to the next higher price by > 10 cigarettes and > 100%. Same demand across all prices excluded. R2 ≤ 0.20 No reported consumption at all prices
Snider et al. 2017(43) USA 385 176:209 46% 33.6 (1.16) 5 Prices: $0-$1 C/D 17.36 (10.48) FTND 4.61 (1.88) Exponentiated Stein et al. 2015(35) three criterion algorithm Overconsumption >200 cigarettes
Strickland et al. 2016(52) USA c40 14:26 35% 44.1 (1.1) 19 Prices: $0 – $1000 C/D 10.4 (1.1) FTND 3.7 (0.4) Exponential/Exponentiated Stein et al. 2015(35) three criterion algorithm
Strickland et al. 2017(53) USA 46 21:25 46% d34 (IQR: 28–42) 16 Prices: $0 – $140 C/D e10 FTND e4 Exponentiated Stein et al. 2015 (35) three criterion algorithm
Van Hedger et al. 2018 (54) USA 35 17:18 49% 26.9 (0.6) 41 Prices: $0 – $35 C/D 8.2 (0.8) FTND 3.1 (0.3) Exponentiated No
Weidberg et al. 2018 (42) Spain 90 66:24 27% 52.27 (9.19) 19 Prices: €0 – €1000 ($0 – $1120) C/D 22.33 (7.88) FTND 6.41 (1.94) Exponential Inspection of increases from one price to the next higher price by > 10 cigarettes and >100%. Same demand across all prices (i.e., 0). R2 ≤ 0.20
Wilson et al. 2016(14) USA 15 6:9, 40% 38.42 (12.67) 15 Prices: $0.01 – $1120 C/D 23.58 (6.85) FTND 9.84 (2.01) Exponential No
Zhao et al. 2016(29) USA 1,214 666:551 55% 26.2 (7.7) 13 prices: $0–11$ C/D 11.23 (8.19) fFTND 1.34 (0.99) Two part mixed-effects model Exclusions based on same demand across all prices (i.e., 0) and first price = 0

Notes. p = UK pence; • = UK pound; $ = US dollar; NZ$ = New Zealand Dollar; € = Euro;

C/D = Cigarettes per Day; CDS-5 = The Cigarette Dependence Scale 5; CPSD = Cigarettes per smoking day; mFTQ = Modified Fagerström Tolerance Questionnaire; FTND = Fagerström Test of Nicotine Dependence; HSI = The Heaviness Smoking Index;

a

Data on age, sample size, and sex are provided as the original Chase et al. (10) paper. However, authors provided data for 128 and were then included in the meta-analyses.

b

Percentage of participants aged between 18–34.

c

The number of participants reflects the sample size included in Strickland et al.(52) paper. Note that for the meta-analysis purposes, data on 39 participants were included as it was the data provided by authors.

d

Median and interquartile range (IRQ) are provided.

e

Median is provided for data on cigarettes per day and nicotine dependence.

f

First item of the FTND was used.

Meta-analytic findings

The meta-analytic relationships between indices of cigarette demand, cigarette consumption and nicotine dependence are in Table 2, with forest plots in Figure 3. The results indicated that all the demand indices were statistically significantly associated with smoking-related measures, but with considerable variation in effect size (rs ranged from .044–.572). This indicates that higher number of cigarettes and nicotine dependence severity were significantly and positively related to higher volumetric demand consumption (i.e., intensity, Omax, Pmax and intensity), and negatively associated with cost-sensitivity to increases in costs; that is, lower elasticity. Medium to large effect size coefficients were present for intensity, Omax, and elasticity, whereas small coefficient sizes were present for breakpoint and Pmax.

Table 2.

Primary meta-analytic findings on the associations between tobacco demand, cigarette consumption, and nicotine dependence.

Demand Index k N rRE 95% Cs p rOSR rFE 95% Cis p I2 Q PQ
Cigarette Consumption

intensity 23 7,569 0.572 0.516–0.623 <10−8 0.559–0.588 0.599 0.584–0.613 <10−8 88.79 196.39 <10−3
Omax 23 7,569 0.376 0.315–0.434 <10−8 0.364–0.387 0.381 0.361–0.400 <10−8 84.42 141.20 <10−3
Pmax 20 6,164 0.044 0.010–0.078 .012 0.034–0.051 0.039 0.014–0.064 .002 25.50 25.50 .145
elasticity 21 6,237 −0.346 −0.416-−0.272 <10−8 −0.360–−0.332 −0.350 −0.371–−0.328 <10−8 86.98 153.70 <10−3
breakpoint 23 7,530 0.086 0.045–0.127 <10−4 0.078–0.096 0.082 0.060–0.105 <10−8 52.83 46.64 .002

Nicotine Dependence

intensity 23 7,597 0.460 0.410–0.508 <10−8 0.452–0.475 0.463 0.446–0.481 <10−8 79.97 109.86 <10−3
Omax 23 7,600 0.367 0.302–0.429 <10−8 0.343–0.379 0.358 0.338–0.378 <10−8 86.70 165.47 <10−3
Pmax 20 6,200 0.108 0.066–0.151 <10−6 0.092–0.117 0.095 0.070–0.120 <10−8 47.07 35.90 .011
elasticity 21 6,271 −0.326 −0.394–−0.255 <10−8 −0.349–−0.304 −0.336 −0.358–−0.314 <10−8 85.26 135.65 <10−3
breakpoint 23 7,558 0.139 0.090–0.187 <10−7 0.118–0.149 0.125 0.102–0.147 <10−8 67.40 67.49 <10−3

Note. k = number of effect sizes; N = sample size; rFE = Pearson’s r effect size statistic from the fixed effects model; rRE = Pearson’s r effect size statistic from the random effects model; rOSR; range of effect sizes obtained from one-study removed analysis; l2 = percentage of variation across studies due to heterogeneity; Q = Cochran’s Q test of homogeneity, PQ = p value corresponding to Cochran’s Q

Figure 3.

Figure 3.

Figure 3.

Figure 3.

Figure 3.

Figure 3.

Forest plots of the meta-analytic findings for Intensity (Panel A), Omax (Panel B), Pmax (Panel C), elasticity (Panel D) and breakpoint (Panel E).

The jackknife analyses did not reveal substantive changes in estimated effect sizes (Table 2). Findings from the fixed effects approach were generally similar in magnitude and revealed substantial heterogeneity across studies for all the CPT indices (I2 = 25.50 – 88.79) (Table 2). Meta-analytic findings including the standardized data processing did not significantly differ from the primary findings (see Table S2).

Mixed random effects models to examine systematic differences in obtained effect sizes based on smoking behavior variable type (i.e., cigarette consumption vs. nicotine dependence) were non-significant for breakpoint (Q = 2.683; p = .101), Omax (Q = 0.039; p = .843), elasticity (Q = 0.151; p = .698), but significant for Pmax (Q = 5.361; p = .021) and intensity (Q = 8.827; p = .003). This reflected significantly larger effect size associations between intensity and cigarette consumption relative to nicotine dependence, and, conversely, larger associations between Pmax and nicotine dependence relative to cigarette consumption (see Table 2).

Using either the exponential or the exponentiated models to derive elasticity of demand did not significantly impact the relationship between elasticity and the two smoking-related measures: cigarette consumption Q = 1.613, p = .204; nicotine dependence Q = 1.665, p = .197.

Publication Bias

Evidence of publication bias was generally not present (see Table 3). For cigarettes/day, Rosenthaĺs failsafe N indicated that a median of 2,969 (range 39–4,703) studies would be needed to render the examined associations not significant (p-value > .05). Evidence of publication bias for all the CPT indices was not present for the Begg-Mazumdar test (all Kendall’s τ ≤ 0.12, all p values > .05) and Eggeŕs tests (all intercepts ≤ 0.55, all p values > .05). Though Duval and Tweediés trim and fill analyses suggested two unpublished studies for elasticity, imputation of these studies did not indicate any significant impact on the estimated effect size (observed ES = 0.35; adjusted ES = 0.35).

Table 3.

Publication bias assessment.

Demand Index Fail-safe N Begg-Mazumdar test Egger’s regression analysis The Tweedie’s trim and fill approach Meta-regression of the relationship between year and effect size

N Kendall’ τ P Intercept 95% CI p N trimmed ES 95% CI Slope 95% CI p
Cigarette Consumption

Intensity 3,095 .007 .957 −.931 [−3.141, 1.277] .390 0 .572 [.572, .516] .016 [−.014, .047] .304
Omax 4,703 .071 .634 .019 [−1.887, 1.926] .983 0 .376 [.315, .433] −.029 [−.057, −.001] .041
Pmax 39 .121 .455 .553 [−.352, 1.459] .215 0 .043 [.009, .078] −.021 [−.037, −.006] .005
Elasticity 2,969 −.014 .927 −.041 [−2.271, 2.189] .969 2 −.357 [−.425,−.286] .018 [−.021, .058] .362
Breakpoint 199 −.007 .957 .198 [−.892, 1.289] .708 0 .086 [.045, .126] −.023 [−.040, −.006] .006

Nicotine Dependence

Intensity 7,324 −.094 .526 −.075 [−1.757, 1.605] .926 3 .444 [.394, .491] −.001 [−.028, .028] .944
Omax 4,262 .126 .398 .293 [−1.766, 2.353] .769 0 .367 [.301, .429] −.0300 [−.058, −.002] .035
Pmax 227 .173 .284 .551 [−.537, 1.639] .301 0 .108 [.065, .150] −.024 [−.043, −.005] .011
Elasticity 2,621 .033 .832 .287 [−1.805, 2.380] .776 7 −.388 [−.456, −.316] .025 [−.011, .061] .177
Breakpoint 499 .007 .957 .348 [−.958, 1.656] .584 0 .139 [.090, .187] −.022 [−.043, −.0008] .041

For nicotine dependence, the results were similar: Rosenthaĺs failsafe Mdn = 2,621 (range = 227–7,324); non-significant Begg-Mazumdar tests (all Kendall’s τ ≤ 0.17, all p values > .05); and non-significant Eggeŕs tests (all intercepts ≤ .55, all p values > .05). However, Duval and Tweediés trim and fill procedure suggested 7 and 3 unpublished effect sizes for elasticity and intensity, respectively. Imputation of potentially unpublished studies attenuated the intensity estimate (observed ES = 0.460; adjusted ES = 0.444) and increased the elasticity estimate (observed ES = −0.326; adjusted ES = −0.388). No unpublished effects for Omax, Pmax, breakpoint, or intensity were found.

With regard to the meta-regression analyses, the results suggested that more recent studies reported smaller correlations between the tobacco-related variables and breakpoint, Omax, and Pmax (Table 3), although not intensity or elasticity.

Discussion

All of the CPT indices were found to be significantly related to the smoking indicators in anticipated directions (negatively for elasticity, positively for the other indicators). However, there was considerable variation in the observed effect sizes, with Omax, intensity and elasticity exhibiting medium effect size associations that were notably larger relative to Pmax and breakpoint. The publication bias indices did not suggest meaningful numbers of unpublished studies and adjustments resulted in modest changes in effect size estimates.

The different magnitudes of association between the demand indices and the smoking variables converges with the proposal that relative reinforcing value is not a homogeneous construct (8). The fact that Omax, intensity and elasticity showed the largest correlations with measures of addiction aligns with clinical research showing their predictive validity on cessation outcomes over and above smoking-related variables (20, 2122). The fact that we found these three indices are more relevant when it comes to smoking involvement in general further suggests which are the key aspects of drug valuation leading to dependence.

Accordingly, several real-world applications stem from this result. Using these three indices within clinical contexts offers the possibility to characterize different patient profiles by for example clustering patients into different levels of demand. A recent attempt to do so has been performed by Nighbor et al. (40) through correlations of median splits of demand indices (i.e., low Omax vs. high Omax) and the number of quit attempts during pregnancy. Also, given their relevance in accounting for smoking persistence, the results suggest that prospective behavioral and pharmacological treatments could be screened using Omax, intensity, and elasticity (in addition to more traditional indicators). Because demand indices are sensitive to clinical and regulatory interventions (41, 42, 28), nicotine reinforcement could be regarded as a measure of their effectiveness or as a surrogate endpoint for new interventions. Omax, Pmax and breakpoint essentially capture economic aspects of demand, which are valuable to inform on which parameters may be manipulated when behavioral economic interventions, such as Contingency Management or nicotine replacement therapies are delivered. Accordingly, high measures on these indices could inform on the necessity to increase the magnitude of reward needed in CM or appropriate medication dose. More broadly, intensity and elasticity can inform on treatment intensity (i.e., longer treatment), given they capture aspects related to persistence in the behavior (i.e., resistance to change captured by elasticity and overall level of consumption captured by intensity). Of course, in each of these domains, it will be critical to demonstrate the incremental utility of the demand indices beyond conventional indicators such as cigarettes per day and nicotine dependence.

In the same way that the findings highlight certain CPT indices as being of greatest relevance, the converse is that other indices emerged as being less relevant. Instigated by the increasing trend to use low-price density tasks (29, 43, 44), a recent study has proposed that breakpoint alone might serve as an accurate measure of demand (45). However, the current findings do not converge with the Athamneh et al. study (45), in light of the current evidence that this index was only modestly correlated with the tobacco indicators, especially compared to intensity or Omax. Thus, using breakpoint as a sole proxy of tobacco use motivation is not warranted. Similarly, Pmax exhibited the smallest magnitude associations with the tobacco indicators. As previous studies have found that Pmax is highly correlated, or even collinear, with breakpoint, which exhibited larger effect sizes in the current study, and have also found that Pmax is less reliable than other demand indices (9, 21, 46), its utility as an individual index is an open question.

The significant negative association between effect size and publication year for breakpoint, Omax and Pmax, suggests smaller effect sizes in more recent studies. One possible explanation for this finding is the increasing inclusion of smokers with comorbid psychopathology, such as other substance use or affective disorders, adding more variability in smoking-related variables. Unlike early studies that recruited more homogeneous university-based samples, more recent studies have relied on community and clinical samples where levels of comorbidities are typically higher (16, 1921). Studies of smokers with comorbid psychopathology showed that intensity and elasticity serve as markers of nicotine addiction (17) and treatment response more strongly than the remaining indices do (20, 22).

These findings should be interpreted in the context of their strengths and limitations. Strengths include a comprehensive search strategy, pre-registration of the protocol, a systematic evaluation of both cigarette consumption and nicotine dependence, and a thorough consideration of publication bias. However, limitations include the relative ‘youth’ of this literature, meaning that the aggregated sample (i.e., 7,649 total participants) was moderately large, but not tens or hundreds of thousands of individuals, which would provide more definitive findings. Furthermore, in a larger overall literature, more nuanced questions, like systematic differences in demand based on psychiatric comorbidities or other moderating variables, could be directly addressed. More generally, it is important to acknowledge that the current findings address the ‘main effect’ relationships between cigarette demand and aspects of smoking, but do not address the unique or incremental associations of the indices. Determining which combinations of indices are specifically and uniquely related to smoking behavior remains a future priority.

At a broader level, the current report pertains to cross-sectional findings and cannot elucidate the causal role of demand in the etiology or treatment of tobacco use disorder. It was notable that the associations with nicotine dependence were not systematically larger than the associations with cigarettes per day, so the etiological significance of tobacco demand remains an open question. Theoretically, the development of high reinforcing value of cigarettes is a key motivating factor in the etiology of tobacco addiction, but an alternative is that higher demand is actually part of the manifestation of tobacco addiction, meaning it is a consequence rather than a cause. However, it is increasingly recognized that the distinction between level of substance use itself and diagnosis of substance use disorder may not be a trenchant one (47). In this case, tobacco use disorder is fundamentally a disorder of excessive cigarette consumption, so the absence of different effect sizes between the demand indices and cigarettes per day and nicotine dependence may not be informative about etiology. Beyond speculation, given the robust evidence for cross-sectional associations between cigarette demand and tobacco involvement there is clearly a high need for longitudinal investigations that can address its etiological relevance.

Supplementary Material

supplementary materials

Acknowledgements

This research was supported by the National Agency of Research of the Spanish Ministry of Science, Innovation and Universities (BES-2016-076663), grants from the National Institutes of Health (R01 DA024652, R01 AA024930, P30 DA027827) and the Peter Boris Chair in Addictions Research. The authors are very grateful to the scientists who provided information or data from previous works for inclusion.

Footnotes

Interests to Declare: AGR, JJ, CM, and DJR have no potential conflicts of interest to declare; JJ has partial or full ownership in HiNT Inc., MedStack, Inc., and Jacob Jackson Capital Corp., JM is a principal in BEAM Diagnostics, Inc.

References

  • 1.Bickel WK, Johnson MW, Koffarnus MN, MacKillop J, Murphy JG. The behavioral economics of substance use disorders: reinforcement pathologies and their repair. Annu Rev Clin Psychol. 2014;10:641–77 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.MacKillop J The behavioral economics and neuroeconomics of alcohol use disorders. Alcohol Clin Exp Res. 2016;40:672–85 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Stoops WW. Reinforcing effects of stimulants in humans: sensitivity of progressive ratio schedules. Exp Clin Psychopharmacol. 2008;16:503–512 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Willner P, Hardman S, Eaton G. Subjective and behavioural evaluation of cigarette cravings. Psychopharmacology. 1995;118:171–77 [DOI] [PubMed] [Google Scholar]
  • 5.Johnson MW, Bickel WK. The behavioral economics of cigarette smoking: The concurrent presence of a substitute and an independent reinforcer. Behav Pharmacol. 2003;14;137–44 [DOI] [PubMed] [Google Scholar]
  • 6.Johnson MW, Bickel WK. Replacing relative reinforcing efficacy with behavioral economic demand curves. J Exp Anal Behav. 2006;85;73–93 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Johnson MW, Bickel WK, Kirshenbaum AP. Substitutes for tobacco smoking: a behavioral economic analysis of nicotine gum, denicotinized cigarettes, and nicotine-containing cigarettes. Drug Alcohol Depend. 2004;74;253–64 [DOI] [PubMed] [Google Scholar]
  • 8.Bickel WK, Jarmalowicz DP, Mueller ET, Gatchalian KM. The behavioral economics and neuroeconomics of reinforcer pathologies: implications for etiology and treatment of addiction. Curr Psychiatry Rep. 2011;13:406–415 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Murphy JG, MacKillop J, Tidey JW, Brazil LA, Colby SM. Validity of a demand curve measure of nicotine reinforcement with adolescent smokers. Drug Alcohol Depend. 2011;113:207–14 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Chase HW, Mackillop J, Hogarth L. Isolating behavioural economic indices of demand in relation to nicotine dependence. Psychopharmacology. 2013;226:371–80 [DOI] [PubMed] [Google Scholar]
  • 11.MacKillop J, Tidey JW. Cigarette demand and delayed reward discounting in nicotine-dependent individuals with schizophrenia and controls: an initial study. Psychopharmacology. 2011;216:91–9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Bickel WK, Marsch LA, Carroll ME. Deconstructing relative reinforcing efficacy and situating the measures of pharmacological reinforcement with behavioral economics: a theoretical proposal. Psychopharmacology. 2000;153:44–56 [DOI] [PubMed] [Google Scholar]
  • 13.Koffarnus MN, Kaplan BA. Clinical models of decision making in addiction. Pharmacol Biochem Behav. 2018;164:71–83 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Wilson AG, Franck CT, Koffarnus MN, Bickel WK. Behavioral economics of cigarette purchase tasks: within-subject comparison of real, potentially real, and hypothetical cigarettes. Nicotine Tob Res. 2016;18:524–30 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.MacKillop J, Few LR, Murphy JG, Wier LM, Acker J, Murphy C, et al. Chaloupka F. High-resolution behavioral economic analysis of cigarette demand to inform tax policy. Addiction. 2012;107:2191–2200 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Dahne J, Murphy JG, MacPherson L. Depressive symptoms and cigarette demand as a function of induced stress. Nicotine Tob Res. 2017;19:49–58 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Farris SG, Aston ER, Zvolensky MJ, Abrantes AM, Metrik J. Psychopathology and tobacco demand. Drug Alcohol Depend. 2017;177:59–66 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Farris SG, Aston ER, Abrantes AM, Zvolensky MJ. Tobacco demand, delay discounting, and smoking topography among smokers with and without psychopathology. Drug Alcohol Depend. 2017;179:247–53 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Secades-Villa R, Weidberg S, González-Roz A, Reed DD, Fernández-Hermida JR. Cigarette demand among smokers with elevated depressive symptoms: an experimental comparison with low depressive symptoms. Psychopharmacology. 2018;235:719–28 [DOI] [PubMed] [Google Scholar]
  • 20.Secades-Villa R, Pericot-Valverde I, Weidberg S. Relative reinforcing efficacy of cigarettes as a predictor of smoking abstinence among treatment-seeking smokers. Psychopharmacology. 2016;233:3103–12 [DOI] [PubMed] [Google Scholar]
  • 21.Mackillop J, Murphy CM, Martin RA, Stojek M, Tidey JW, Colby SM, et al. Predictive validity of a cigarette purchase task in a randomized controlled trial of contingent vouchers for smoking in individuals with substance use disorders. Nicotine Tob Res. 2016;18:531–7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.González-Roz A, Secades-Villa R, Weidberg S, García-Pérez Á, Reed DD. Latent structure of the CPT among treatment-seeking smokers with depression and its predictive validity on smoking abstinence. Nicotine Tob Res. 2018 [DOI] [PubMed] [Google Scholar]
  • 23.MacKillop J, Brown CL, Stojek MK, Murphy CM, Sweet L, Niaura RS. Behavioral economic analysis of withdrawal- and cue-elicited craving for tobacco: an initial investigation. Nicotine Tob Res. 2012;14:1426–34 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Hitsman B, MacKillop J, Lingford-Hughes A, Williams TM, Ahmad F, Adams S, et al. Munafò MR. Effects of acute tyrosine/phenylalanine depletion on the selective processing of smoking-related cues and the relative value of cigarettes in smokers. Psychopharmacology. 2008;196:611–21 [DOI] [PubMed] [Google Scholar]
  • 25.McClure EA, Vandrey RG, Johnson MW, Stitzer ML. Effects of varenicline on abstinence and smoking reward following a programmed lapse. Nicotine Tob Res. 2013;15:139–48 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Schlienz NJ, Hawk LW Jr., Tiffany ST, O’Connor RJ, Mahoney MC. The impact of pre-cessation varenicline on behavioral economic indices of smoking reinforcement. Addict Behav. 2014;39:1484–90 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Green R, Ray LA. Effects of varenicline on subjective craving and relative reinforcing value of cigarettes. Drug Alcohol Depend. 2018;188:53–59 [DOI] [PubMed] [Google Scholar]
  • 28.Murphy CM, MacKillop K, Martin RA, Tidey JW, Colby SM, Rohsenow DJ. Effects of varenicline versus transdermal nicotine replacement therapy on cigarette demand on quit day in individuals with substance use disorders. Psychopharmacology. 2017;234:2443–2452 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Zhao T, Luo X, Chu H, Le CT, Epstein LH, Thomas JL. A two‐part mixed effects model for cigarette purchase task data. J Exp Anal Behav. 2016;106:242–53 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Liao WJ, Luo XH, Le CT, Chu HT, Epstein LH, Yu J, et al. Thomas JL. Analysis of cigarette purchase task instrument data with a left-censored mixed effects model. Exp Clin Psychopharmacol. 2013;21:124–132 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Koffarnus MN, Franck CT, Stein JS, Bickel WK. A modified exponential behavioral economic demand model to better describe consumption data. Exp Clin Psychopharmacol. 2015;23:504–12 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Hursh SR, Silberberg A. Economic demand and essential value. Psychol Rev. 2008;115:186–98 [DOI] [PubMed] [Google Scholar]
  • 33.Aston ER, Cassidy RN. Behavioral economic demand assessments in the addictions. Curr Opin Psychol. 2019;30:42–47 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg Case Rep. 2010;8:336–41 [DOI] [PubMed] [Google Scholar]
  • 35.Stein JS, Koffarnus MN, Snider SE, Quisenberry AJ, Bickel WK. Identification and management of nonsystematic purchase task data: toward best practice. Exp Clin Psychopharmacol. 2015;23:377–86 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Rothstein HR, Sutton AJ, Borenstein M. Publication bias in meta-analysis Prevention, assessment and adjustements. Chichester, UK: John Wiley & Sons, Ltd; 2005 [Google Scholar]
  • 38.Rosenthal R The “File Drawer Problem” and tolerance for null results. Psychol Bull. 1979:86:638–641. [Google Scholar]
  • 39.Cohen J Statistical Power Analysis for the Behavioral Sciences. Hilldale, NJ: Erlbaum; 1988. [Google Scholar]
  • 40.Nighbor TD, Zvorsky I, Kurti AN, Skelly JM, Bickel WK, Reed DD, et al. Higgins ST. Examining interrelationships between the cigarette purchase task and delay discounting among pregnant women. J Exp Anal Behav. 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Grace RC, Kivell BM, Laugesen M. Predicting decreases in smoking with a cigarette purchase task: evidence from an excise tax rise in New Zealand. Tob Control. 2015;24:582–7 [DOI] [PubMed] [Google Scholar]
  • 42.Weidberg S, Vallejo-Seco G, González-Roz A, García-Pérez Á, Secades-Villa R. In-treatment cigarette demand among treatment-seeking smokers with depressive symptoms. Addict Behav. 2018;82:35–43 [DOI] [PubMed] [Google Scholar]
  • 43.Snider SE, Cummings KM, Bickel WK. Behavioral economic substitution between conventional cigarettes and e-cigarettes differs as a function of the frequency of e-cigarette use. Drug Alcohol Depend. 2017;177:14–22 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Koffarnus MN, Wilson AG, Bickel WK. Effects of experimental income on demand for potentially real cigarettes. Nicotine Tob Res. 2015;17:292–298 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Athammeh LN, Stein JS, Amlung M, Bickel WK. Validation of a brief behavioral economic assessment of demand among cigarette smokers. Exp Clin Psychopharmacol. 2019;1:96–102 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.MacKillop J, Murphy JG, Ray LA, Eisenberg DT, Lisman SA, Lum JK, et al. Further validation of a cigarette purchase task for assessing the relative reinforcing efficacy of nicotine in college smokers. Exp Clin Psychopharmacol. 2008;16:57–65 [DOI] [PubMed] [Google Scholar]
  • 47.Nutt DJ, Gual A, Anderson P, Rehm J. Why less is always more in the treatment of alcohol use disorders. JAMA Psychiatry. 2019. [DOI] [PubMed] [Google Scholar]
  • 48.Few LR, Acker J, Murphy C, MacKillop J. Temporal stability of a cigarette purchase task. Nicotine Tob Res. 2012;14:761–5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Heckman BW, Cummings KM, Nahas GJ, Willemsen MC, O’Connor RJ, Borland R, et al. Carpenter MJ. Behavioral economic purchase tasks to estimate demand for novel nicotine/tobacco products and prospectively predict future use: evidence from the Netherlands. Nicotine Tob Res. 2018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.O’Connor RJ, Heckman BW, Adkison SE, Rees VW, Hatsukami DK, Bickel WK. Persistence and amplitude of cigarette demand in relation to quit intentions and attempts. Psychopharmacology. 2016;233:2365–71 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Smith TT, Cassidy RN, Tidey JW, Luo X, Le CT, Hatsukami DK, et al. Impact of smoking reduced nicotine content cigarettes on sensitivity to cigarette price: further results from a multi-site clinical trial. Addiction. 2017;112:349–59 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Strickland JC, Lile JA, Rush CR, Stoops WW. Comparing exponential and exponentiated models of drug demand in cocaine users. Exp Clin Psychopharmacol. 2016;24:447–55, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Strickland JC, Stoops WW. Stimulus selectivity of drug purchase tasks: a preliminary study evaluating alcohol and cigarette demand. Exp Clin Psychopharmacol. 2017;25:198–207 [DOI] [PubMed] [Google Scholar]
  • 54.Van Hedger K, Kushner MJ, Lee R, de Wit H. Oxytocin reduces cigarette consumption in daily smokers. Nicotine Tob Res. 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Tabachnick BG, Fidell LS. Using multivariate statistics. Boston: Allyn and Bacon; 2000. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplementary materials

RESOURCES