Identification and Management of Nonsystematic Cross-Commodity Data: Toward Best Practice

Abstract

Data systematicity has been an important area of consideration for behavioral economic demand. Stein et al. (2015) introduced criteria and an accompanying algorithm to aid researchers in identifying data series that may be considered “nonsystematic”. That is, data that may not follow empirically based assumptions such as an overall decrease in consumption as cost of a commodity increases and consistency in decreases in consumption. However, those criteria and algorithm are only directly applicable to own-price demand, or demand for a commodity that is increasing in price. Cross-price demand, or demand for a second commodity that changes as a function of some other commodity, does not have a similar set of criteria or algorithm for assessing cross-commodity demand systematicity. Cross-price or cross-commodity demand is useful in understanding how changes in one substance or commodity may change the consumption of another substance or commodity. Thus, we extend the Stein et al. criteria and algorithm to classify if a cross-commodity can be considered a substitute, complement, or independent, and then assess its systematicity based on its classification. We demonstrate this algorithm on three different cross-commodity demand data sets and describe important considerations regarding data exclusions to prevent biasing results from own-price and cross-price demand.

Behavioral economic demand (hereafter demand), the study of commodity consumption under different constraints, has been fruitful in multiple areas of study, such as consumer decision-making and substance use (Hursh & Roma, 2013, 2016; Strickland et al., 2020; Zvorsky et al., 2019). Most commonly, changes in purchasing/consumption of some commodity are assessed as a function of increase in price or some other constraint for that commodity. This is known as own-price demand (Hursh, 1984; Hursh & Roma, 2016). For humans, a common way to assess this is with hypothetical purchase tasks which typically consist of a series of hypothetical questions that ask how much of a commodity someone would purchase and consume at various costs per unit of that commodity (Roma et al., 2016, 2017). Generally, it is expected that consumption or purchasing of a commodity will decrease as price increases (Hursh, 1984; Hursh & Silberberg, 2008). However, participant factors, like inattentiveness or misunderstanding of task instructions, may also lead to patterns that are not generally expected. Features of task design (e.g., random price order or ascending price order) may also affect response patterns. These considerations led to the development of “systematicity” criteria (Stein et al., 2015) for determining if the responses from demand tasks follow general trends expected of demand data.

The three systematicity criteria proposed by Stein et al (2015) for demand data are (1) trend, (2) bounce, and (3) reversals from zero. The trend criterion was developed based on the assumption that consumption is typically assumed to decrease from the least constraint/cost to the highest constraint/cost. The bounce criterion was developed based on the assumption that after consumption for a commodity has decreased, consumption ought not increase as the constraint increases further. Finally, reversals from zero is the assumption that once a consumption of a commodity reaches zero, consumption ought not resume as the constraint/cost increases above the point constraint/cost suppressed all consumption. We want to stress that in cases where some criterion is violated, such as consumption not decreasing by a sufficient level as determined by an algorithm, it does not necessarily mean that data were the result of lack of participant attentiveness or actual violation of a presupposed law of demand. For example, lack of trend may be indicative of a constraint that does not affect an individual’s consumption or using a range of constraints too limited to affect consumption. For example, if prices in a purchase task range between $0 to $10 per drink, there might be a larger proportion of trend violations compared to a range of prices between $0 to $100, but this would be due to the task rather than any violations of “laws of demand”. While the Stein et al. (2015) criteria account for decreases in a proportional manner based on the range of prices, the range of prices may not elicit the behavior of interest. That is, nonsystematic data may be an artifact of the methods. These originally proposed criteria, much like the Johnson and Bickel (2008) criteria inspired for delay discounting data, are not intended to be the sole justification for removing data from a dataset that do not conform to hypotheses or make analyses difficult. Instead, these criteria are meant to be a tool for researchers to quantify the degree to which their methods result in unexpected data patterns and which participants were not under experimental control of the task and task instructions. Important to note, this is the same philosophy we adopt in creating the cross-commodity criteria discussed later in this paper.

Cross-Price Demand

Now that own-price demand and own-price systematicity have been described, we introduce the concept of cross-price demand. Cross-price demand is the change in consumption of some commodity as a function of changes to constraints/cost of some other commodity. The contextualized reinforcer pathology approach partially characterizes addiction as the preference for drug rewards relative to drug and non-drug alternatives (Acuff et al., 2023), and cross-price demand can provide an important framework to understand substance misuse. That is, as the cost of Commodity A increases, cross-price demand assesses any change in consumption for a Commodity B that does not change in price. For example, an increase in the cost of cigarettes and a corresponding decrease in cigarette consumption may result in an increase, decrease, or no change in cannabis consumption (while cannabis cost remains the same). Figure 1 has examples of different patterns of changes in the consumption of cannabis (Commodity B) to increases in cost of cigarettes (Commodity A). If cannabis consumption increases as a function of increased cigarette cost, cannabis would be considered a substitute for cigarettes (Figure 1: Cannabis as a Substitute). In other terms, the individual is replacing decreased cigarette consumption by switching to cannabis consumption. This may be relevant if one were interested in identifying if consumption of one commodity would replace another if it were placed under high constraint (e.g., criminalized, taxed). Conversely, if consumption of cannabis decreases as a function of increased cigarette price and a corresponding decrease in cigarette consumption, cannabis would be considered a complement to cigarettes (Figure 1: Cannabis as a Complement). A complementary commodity is one where its consumption decreases alongside decreases in consumption of some other commodity. Complementary relationships are useful for determining how intervening on one substance may have a beneficial decrease in another without additional intervention. Finally, cannabis would be considered an independent in the case that cannabis consumption does not change as a function of changes to cigarette constraints (Figure 1: Cannabis as an Independent). It is important to state that a commodity being identified as a substitute, complement, or independent is not inherent in the commodity, but instead based on the relationship between the individual/organism and their environment. Not all individuals will have the same directional changes to Commodity B and for this it would be relevant to consider questions like “for whom does cannabis act as a substitute for cigarettes?” rather than an absolute of “is cannabis a substitute for cigarettes?”.

Figure 1. Demonstrations of Own-Price and Cross-Price Demand.

Note. Idealized demand curves for own-price cigarette demand (Commodity A; top left panel) and cannabis cross-price demand (Commodity B; top right, bottom left, and bottom right panels). Y-axes are the number of cigarettes or grams of cannabis purchased. X-axes are the cost of cigarettes. Solid line/filled circles represent the number of cigarettes purchased. Dashed line/empty circles represent the number of grams of cannabis purchased. Idealized own-price demand is modeled with the Rzeszutek et al. (2025) equation: $Q_{Own}=Q_0·e^{(-\alpha ·Q_0·C)}$. Idealized cross-price demand is based on a non-logarithmic version of the (Hursh & Roma, 2013, 2016) cross-price demand model: $Q_{\mathit{\text{Cross}}}=Q_{\mathit{\text{alone}}}+{I·e}^{(-\beta ·C)}$. See Rzeszutek et al. (2025) and Hursh and Roma (2016) for more details on these respective models and their parameters.

There has been an increased interest in examining drug consumption in the presence of alternatives (Weinsztok et al., 2023). However, as highlighted by Stein et al. (2015) and Bono et al (2024), there are no criteria that currently exist to address if cross-commodity consumption data are “systematic”. A recent review of cross-commodity purchase tasks by (Weinsztok et al., 2023) found that of 23 substance-related cross-commodity purchase tasks that were published after Stein et al. (2015), about half reported criteria for data exclusion in analysis of own-price demand data (i.e., the price-manipulated commodity), and most used the Stein et al. criteria when excluding data. However, to our knowledge, no similar criteria were applied to cross-commodity consumption data other than consumption values that were considered statistical outliers. Therefore, an algorithm to evaluate demand systematicity in cross-commodity arrangements is needed. For consistency within the manuscript, commodity A will always refer to the own-price demand/adjusting-price commodity whereas commodity B will always refer to the cross-price demand/fixed-price commodity.

Determining Cross-Commodity Systematicity

Here, we offer a logical extension of the algorithm described in Stein et al. (2015) for own-price demand systematicity that is appropriate for cross-commodity consumption. We apply this to some published and unpublished cross-commodity data to demonstrate how it might be implemented. To decrease the barrier to entry for using this algorithm, we have also incorporated these analysis tools into the R package beezdemand (Kaplan, Gilroy, et al., 2019) and web-based tool shinybeez (Kaplan & Reed, 2025) as well as provide the code and data used in these analyses. Determining cross-commodity systematicity with this algorithm is a multi-step process, discussed below. It is necessary to note that in the following steps and criteria, determining cross-commodity systematicity (i.e., commodity B) is not dependent on the systematicity or responding of the own-price commodity (i.e. commodity A). Because of this, responding for commodity B may be identified as systematic, whereas responding for commodity A may not be.

Step 1: Determine if cross-commodity consumption is a complement, substitute, or independent.

Cross-commodity data (i.e., responding for commodity B) can logically take the shape of a complement, substitute, or independent, and therefore it can uniquely increase or decrease as the cost of commodity A increases. Therefore, cross-commodity classification (i.e., complement, substitute, independent) is necessary prior to the application any systematicity criteria as the classification of data series determines how the systematicity criteria will be applied. The first step is determine which relationship is in place by calculating the overall directional change in consumption with the $\Delta Q$ criterion from Stein et al. (2015). This criterion essentially determines if there is a change in consumption, Q, from the lowest price/cost (i.e., Q₁ at C₁) and the highest price/cost (i.e., Q_n at C_n) normalized to the range of prices (see Equation 1) and compares that difference against some decision criteria based on consumption at the lowest price (e.g., .025 * Q₁).

$$\Delta Q=\frac{{log}_{10}\left(Q_1\right)-{log}_{10}\left(Q_n\right)}{{log}_{10}\left(C_n\right)-{log}_{10}\left(C_1\right)}$$ (1)

Much like in the Stein et al. (2015) criteria, if $\Delta Q$ is a positive value greater than the selected threshold, then the cross-commodity consumption is flagged as a complement. That is, consumption at the lowest price is higher than consumption at the highest price. This would indicate in a gross sense that cross-commodity consumption decreased as a function of cost of the adjusting price item. Likewise, we can flag the cross-commodity as a substitute if $\Delta Q$ in Equation 1 is a negative value that exceeds the negative threshold (e.g., −0.025). That is, cross-commodity consumption increased as a function of increasing price of the adjusting price commodity. If cross-commodity consumption does not exceed the positive or negative threshold to be determined as a complement or substitute respectively, then the data series is identified as an independent. To prevent problems with consumption values of 0, a small constant of 0.01 is added to each consumption value, mirroring how the Stein et al. (2015) algorithm is implemented in beezdemand. Note that there is no analog of the Trend criterion from the Stein et al. algorithm because cross-commodity data can logically trend in any direction, contradicting the original criterion assumption that consumption typically decreases from the lowest to highest priced commodity.

Step 2: Determine systematicity based on cross-commodity category

Once the cross-commodity data series (i.e., commodity B) has been identified as either a complement, substitute, or independent based on gross trend, it can be assessed for systematicity criteria specific to its classification. The two categories of systematicity criteria we propose for cross-commodity demand are the bounce criteria and the reversals from zero/returns to zero criteria.

Bounce:

After commodity B (i.e., cross-commodity) consumption has been flagged as either a complement, substitute, or independent, a version of the bounce criterion from Stein et al. (2015) is applied to the data series. If the cross-commodity consumption was flagged as a complement, the bounce criterion from Stein et al. is identical to that of own-price consumption. For complements, this is done by determining if a response increases by 25% of Q₁ from the previous point. Note that this calculation is done using untransformed values. Then, to determine if a data series is “systematic” or “price-consistent responding” the number of total number of bounces identified in this way are compared to the number of bounces over the number of possible price increases (i.e., moving from left to right), yielding a percentage. The top left panel in Figure 2–Own Price/Complement Systematicity has an example of how the decision boundary is determined and which points are determined as violating the Bounce criterion. For the Stein et al. criteria, a .10 proportion of opportunities exceeded is recommended, but this can be increased or decreased based on researcher need. For example, if there are 11 prices used in a purchase task, this means there are 10 opportunities for a bounce to occur. If there are 10 opportunities, then there need to be at least two points that are determined as bouncy for the data series to be flagged for bounce using the .10 proportion threshold. Using this value also means that when using fewer than 10 price points, this .10 threshold will result in any bounces flagging a data series.

Figure 2. Demonstrations of the Cross-Commodity Systematicity Algorithm.

Note. In all panels, the y-axes are the grams of cannabis purchased (Commodity B) and the x-axes are the cost of cigarettes (Commodity A). Points that are “systematic” as determined by the algorithm are white circles, “non-systematic” points are indicated with grey fill (bounce up), black fill (bounce down), triangles up (reversal from zero), or triangles down (return to zero). Points can be more than one non-systematic category. Decision boundaries are indicated by solid vertical lines with flat tops/bottoms. Data points with a value above, below, or both are the point used to determine the range of the boundary, and dependent on if the data path was determined as complement, substitute, or independent, the direction of the boundary. While either the first or last point of the data series can be used to determine the boundary, all bounce violations are determined from left to right.

For substitutes, this is the same process but is based on a decrease in responding as price increases, using a value of 25% of Q_n (consumption at the highest price) for each subsequent purchase as the cost of the adjusting-price item increases. Consumption at the highest price is used as a basis for determining bounce size as this is generally the point of highest consumption for substitutes. The top right panel in Figure 2–Substitute Systematicity has an example of how this algorithm is applied for data series identified as a substitute.

For independents, the 25% threshold is calculated as 25% of the maximum of consumption at Q₀ and Q_n. Using this value, both bounces up and bounces down are calculated using the logic for complements and substitutes, respectively. To determine if a data series flagged as an independent is not “systematic”, the minimum value of bounce violations using the complement and substitute procedure is set as the final bounce value. This is because an independent may trend toward a complement or substitute, and the criteria shouldn’t be biased against such data.¹ By taking the minimum of the complement and substitute criteria, only bounces that cannot be accounted for by a slight trend in data should be flagged. For example, in the case of bottom left panel of Figure 2–Independent Systematicity (1), the data series flagged as independent has three instances of bounce down and two instances of bounce up. The two instances of bounce up would be used as the final count of bounce violations to determine if this dataset met the 10% systematicity threshold. There is also an example of an independent data series with variability not meeting the bounce threshold in either direction which can be found in the bottom right panel of Figure 2–Independent Systematicity (2). In all cases, complements, substitutes, or independents, the bounce criterion is always applied from left to right. The rationale for this is that the definition for all own-price and cross-price consumption is based on the assumption: as the cost of the own-price commodity increases the consumption of the cross-price commodity is affected.

Reversals from Zero and Returns to Zero:

We have also incorporated the reversals from zero criteria much like Stein et al. (2015), where a data point is flagged as having “reversing from zero” when a response follows a given number of zeros. For data identified as a complement, the implementation of this criterion is identical to the Stein et al. reversals from zero criteria, where the default is a reversal being flagged after a value greater than zero is present after two consecutive zero responses. To make a similar consideration for substitutes, we introduce the criteria “returns to zero”, where if a data series has been identified as a substitute (i.e., trends up as price increases), then any responses of zero following two non-zero responses is flagged as a return to zero. In the case of independents, both reversals from and returns to zero are calculated and the minimum value of the two is used to determine systematicity. Examples of reversals from and returns to zero can be found in Figure 2.

Step 3: Report Own-Price and Cross-Price Systematicity

When applying behavioral economic demand systematicity criteria to a data series, rates of systematicity should be reported in manuscripts. This allows for meta-analyses to determine if there are systematic deviations from systematicity in different groups, tasks, or commodities. This allows for transparency within behavioral economic demand data and can help to improve task development. See Tables 1–3 as examples.

Table 1.

Low Nicotine - Kaplan et al. (2018) Systematicity Data

Own-Price/Commodity A Systematicity Violations
Commodity (n)	Trend/$\Delta Q$			B-Up				Reversals
Own 100% (67)	1 (1.5%)			3 (4.5%)				2 (3.0%)
Own 2% (57)	3 (5.3%)			0 (0%)				3 (5.3%)
Own 98% (59)	1 (1.7%)			1 (1.7%)				0 (0%)
Cross-Price/Commodity B Systematicity Metrics
	Cross-Commodity Category			Systematicity Violations
Commodity (n)	$\Delta Q_{\mathit{\text{Down}}}$	$\Delta Q_{Up}$	$\Delta Q_{\mathit{\text{None}}}$	B-Up*	B-Down*	B-Ind*	B-Any	Reversals	Returns
Cross 100% (67)	9 (13.4%)	24 (35.8%)	34 (50.7%)	0 (0%)	1 (4.2%)	1 (2.9%)	2 (3%)	6 (9%)	1 (1.5%)
Cross 2% (57)	6 (10.5%)	15 (26.3%)	36 (63.2%)	1 (16.7%)	2 (13.3%)	2 (5.6%)	5 (8.8%)	3 (5.3%)	0 (0%)
Cross 98% (59)	10 (16.9%)	26 (44.1%)	23 (39%)	0 (0%)	2 (7.7%)	3 (13%)	5 (8.5%)	4 (6.8%)	1 (1.7%)

Note: Own-price systematicity violations based on the Stein et al. (2015) criteria and cross-price systematicity on the proposed criteria. Count and percentages by commodity across own-price/adjusting commodity and cross-price/fixed commodity by condition. Trend/$\Delta Q$: Stein et al. trend criteria violations. $\Delta Q_{\mathit{\text{Down}}}$: Trend identified down (complement) $\Delta Q_{Up}$: Trend identified up (substitute). $\Delta Q_{\mathit{\text{None}}}$: Trend identified as neither up nor down (independent). B-Up: Bounce up violations. B-Down: Bounce down violations. B-Ind: Bounces up and down violations for independents. B-Any: Overall bounces irrespective of trend. Reversals: Reversals from zero. Returns: Returns to zero. For cross-commodity bounce violations, percentages are based on the number of trends identified for the relevant bounce criteria and indicated with an *.

Table 3.

In progress Experimental Tobacco Marketplace Data

Own-Price/Commodity A Systematicity Violations
Commodity (n)	Trend/$\Delta Q$			B-Up				Reversals
Own Cig. (47)	3 (6.4%)			5 (10.6%)				0 (0%)
Cross-Price/Commodity B Systematicity Metrics
	Cross-Commodity Category			Systematicity Violations
Commodity (n)	$\Delta Q_{\mathit{\text{Down}}}$	$\Delta Q_{Up}$	$\Delta Q_{\mathit{\text{None}}}$	B-Up*	B-Down*	B-Ind*	B-Any	Reversals	Returns
Cross Cig. (47)	4 (8.5%)	6 (12.8%)	37 (78.7%)	3 (75%)	0 (0%)	2 (5.4%)	5 (10.6%)	1 (2.1%)	1 (2.1%)
Cross E-Cig. (47)	6 (12.8%)	15 (31.9%)	26 (55.3%)	3 (50%)	3 (20%)	2 (7.7%)	8 (17%)	2 (4.3%)	1 (2.1%)

Note: Own-price systematicity violations based on the Stein et al. (2015) criteria and cross-price systematicity on the proposed criteria. Count and percentages by commodity across own-price/adjusting commodity and cross-price/fixed commodity by condition. Trend/$\Delta Q$: Stein et al. trend criteria violations. $\Delta Q_{\mathit{\text{Down}}}$: Trend identified down (complement) $\Delta Q_{Up}$: Trend identified up (substitute). $\Delta Q_{\mathit{\text{None}}}$: Trend identified as neither up nor down (independent). B-Up: Bounce up violations. B-Down: Bounce down violations. B-Ind: Bounces up and down violations for independents. B-Any: Overall bounces irrespective of trend. Reversals: Reversals from zero. Returns: Returns to zero. For cross-commodity bounce violations, percentages are based on the number of trends identified for the relevant bounce criteria and indicated with an *.

Step 4: Sensitivity Analyses and Data Exclusions

This step is extremely important for own-price and cross-price demand. A variety of different criteria have been used to identify non-systematic data series (e.g., Bruner & Johnson, 2014; Kaplan et al., 2018; Stein et al., 2015; Weinsztok et al., 2023) and other illogical data patterns (e.g., extreme consumption values) which may then result in either removal of data series or replacement of specific consumption values (e.g., Winsorizing). While the systematicity criteria presented here and in Stein et al. (2015) are intended to be descriptive in nature, if any data are excluded from an analysis based on any criteria, we recommend that at a minimum the criteria for exclusion be clearly articulated and described, and that a sensitivity analysis be conducted for the data with exclusions and the data without exclusions. To conduct a sensitivity analysis in this case, one would run the same analysis on a dataset before and after data were excluded to determine whether the removal of data affected the conclusions of the study. Whether the conclusions were affected should then be reported in the manuscript. The reason why this is such an important step is that depending on the nature of the exclusions, results could be adversely affected and conclusions could be fundamentally different. However, if these sensitivity analyses are not conducted, it is impossible to know how interpretations may be affected. If a sensitivity analysis cannot be conducted, reasons should be explicit as to why. For example, if a proposed non-linear analysis results in non-convergence issues due to the irregularity in the data, this should also be clearly stated. Also, if possible, alternate endpoints other than demand used in the study or participant characteristics (e.g., use disorder severity, age, sex, etc.) should be compared before and after exclusion to detect whether the exclusion biased the sample in other ways.

Example Applications of Cross-Commodity Systematicity Criteria

To demonstrate these proposed cross-commodity systematicity criterion, we present some cross-commodity demand data from three different datasets. The first dataset is from already published data (Kaplan, Pope, et al., 2019). These consist of between-subject cross-commodity consumption for cigarettes and alternative cigarettes over three different narrative scenarios from participants recruited from Amazon Mechanical Turk (MTurk) in 2017. This dataset had 16 price points for the adjusting price cigarettes. The second dataset is unpublished data of within-subject cross-commodity demand between cigarettes and cannabis consisting of participants recruited from MTurk in 2023. This dataset had 11 prices for adjusting price cannabis, and 14 prices for adjusting price cigarettes. The third dataset is from an ongoing study which incorporates the Experimental Tobacco Marketplace (ETM) consisting of participants recruited from the Appalachian region in Kentucky (Clinical Trials ID: NCT06234722). This dataset has 6 prices for adjusting price preferred-flavor cigarettes. While more products were assessed in the ETM, we only looked at cigarette smokers and their own-price demand for preferred-flavor cigarettes, cross-commodity purchasing of non-preferred cigarettes, and cross-commodity purchasing of e-cigarettes. In all three datasets being used for demonstrative purposes, prices were presented in ascending order. For the low-nicotine study, all purchasing opportunities were presented on the same page, whereas for the cannabis/cigarette and ongoing ETM data, purchasing opportunities were presented on separate pages.

Data Analysis

For own-price demand data series, the Stein et al. (2015) criteria for trend (i.e., Trend/$\Delta Q$), bounces up, and reversals from zeros were applied. For cross-price data series, Steps 1–3 described above were applied. No additional screening that occurred in Kaplan et al. (2019) were used to exclude data prior to this analysis. Code and data used for the analysis are available in the supplemental documents. Analyses were conducted in R 4.4.2 (R Core Team, 2024), and packages data.table (Dowle & Srinivasan, 2020) and beezdemand (Kaplan, Gilroy, et al., 2019) were used for data organization and basing cross-commodity systematicity respectively.

We also conducted an analysis of different bounce thresholds for cross-commodity categorization across different $\Delta Q$ rates. The purpose of this was to demonstrate how changing these decision thresholds affect the relative proportion of data paths were identified and how nonsystematic data changed as a function of this. The $\Delta Q$ category thresholds assessed were .001, .025, .075, .150, .250, .375, and .500. The bounce thresholds assessed were .050, .100, .200, .300, and .400. We also conducted the same analysis with own-price demand from these datasets, which is further described and reported in the supplemental document.

Results

Tables 1, 2, and 3 contain the counts and percentages of own-price and cross-price demand for the Kaplan et al. (2019) dataset, cigarette and cannabis dataset, and the ETM dataset respectively. For bounce criteria, because the number of price points differed, this also results in the threshold that was used (.10 proportion of opportunities) to be more sensitive. That is, for the ETM dataset, there are only five opportunities for a bounce to occur, and thus a single bounce would be above the threshold. In contrast, for the Kaplan et al. (2019) study, there were 15 opportunities for bounce to occur, which required two bounces to meet/exceed the .10 proportion for being identified as having failed the bounce criteria. The cigarette and cannabis dataset also required at least two points being identified as a bounce to be flagged. For own-price demand most participants from Kaplan et al. (2019) and the ongoing ETM study had data series that would pass the trend criteria (> 90% passed) and similar numbers for the bounce and reversals criteria. By contrast, the cigarette and cannabis own-price demand only had ~60% of participants who displayed a decreasing trend for either commodity. Bounce for both cigarette and cannabis from this dataset also was substantially higher than the other two, with ~40% of participants data series being flagged for having exceeded the bounce threshold.

Table 2.

Unpublished Cannabis and Cigarette Systematicity Data

Own-Price/Commodity A Systematicity Violations
Commodity (n)	Trend/$\Delta Q$			B-Up				Reversals
Own Can. (99)	42 (42.4%)			40 (40.4%)				1 (1%)
Own Cig. (99)	37 (37.4%)			34 (34.3%)				0 (0%)
Cross-Price/Commodity B Systematicity Metrics
	Cross-Commodity Category			Systematicity Violations
Commodity (n)	$\Delta Q_{\mathit{\text{Down}}}$	$\Delta Q_{Up}$	$\Delta Q_{\mathit{\text{None}}}$	B-Up*	B-Down*	B-Ind*	B-Any	Reversals	Returns
Cross Cig. (99)	40 (40.4%)	20 (20.2%)	39 (39.4%)	7 (17.5%)	6 (30%)	9 (23.1%)	22 (22.2%)	1 (1%)	0 (0%)
Cross Can. (99)	39 (39.4%)	19 (19.2%)	41 (41.4%)	13 (33.3%)	11 (57.9%)	8 (19.5%)	32 (32.3%)	0 (0%)	1 (1%)

Note: Own-price systematicity violations based on the Stein et al. (2015) criteria and cross-price systematicity on the proposed criteria. Count and percentages by commodity across own-price/adjusting commodity and cross-price/fixed commodity by condition. Trend/$\Delta Q$: Stein et al. trend criteria violations. $\Delta Q_{\mathit{\text{Down}}}$: Trend identified down (complement) $\Delta Q_{Up}$: Trend identified up (substitute). $\Delta Q_{\mathit{\text{None}}}$: Trend identified as neither up nor down (independent). B-Up: Bounce up violations. B-Down: Bounce down violations. B-Ind: Bounces up and down violations for independents. B-Any: Overall bounces irrespective of trend. Reversals: Reversals from zero. Returns: Returns to zero. For cross-commodity bounce violations, percentages are based on the number of trends identified for the relevant bounce criteria and indicated with an *.

Cross-commodity trends were, from lowest to highest, complements, substitutes, and independents as determined by the $\Delta Q$ assignment (see “default” value proportions in Figure 3). Generally, the highest number of bounce violations occurred for complements, while the substitutes and independents had lower number of bounce violations. For the cigarette and cannabis dataset, most data series were identified as complements or independents ~40% for both fixed price cigarettes and cannabis, with roughly 20% of participants’ data series being identified as substitutes. Bounce violations were also generally higher in the cigarette and cannabis dataset for cross-commodity demand as well.

Figure 3. Cross Commodity Threshold Analyses.

Note. Proportions of cross-commodity identifications/categorizations based on different ΔQ thresholds and bounce thresholds for all three datasets. Y-axes, proportions, x-axes, ΔQ thresholds. For the top left panel data paths represent the proportions of cross-commodity category for all three datasets. For all other panels data paths represent the proportions of data series that were identified to have a bounce based on different bounce thresholds. Thick squares which highlight data points represent the “default” criteria of a ΔQ of .025 and a bounce threshold of .1.

Results of the threshold analysis for own-price and cross-price systematicity for each study can be found in the supplemental document. Cross-commodity categorization and bounce violations using “default” values of $\Delta Q$ (0.025) and bounce (.10) are highlighted with thick black squares in this figure. As the $\Delta Q$ threshold for cross-commodity assignment increased, the relative proportion of data series identified as independent increased while substitutes and complements decreased. Results of the threshold analysis for each study can be found in the supplemental document.

Discussion

The purpose of this study was to propose and demonstrate the use of cross-commodity systematicity criteria, expanding upon the Stein et al. (2015) own-price demand criteria. When possible, we attempted to create rules that are consistent with the Stein et al. criteria, only adding rules and stipulations when necessary to accommodate the different patterns that cross-commodity demand data can coherently take. As cross-commodity data can reasonably show an increasing trend, a decreasing trend, or no trend, there is no ‘trend’ criterion in this version. The ‘bounce’ and ‘reversals from zero’ criteria were then adapted to accommodate different patterns of cross-commodity data.

The goal of these criteria is to provide a standardized way to assess systematicity of cross-commodity demand data. These criteria could be useful to characterize datasets, revealing which participants were under experimental control of the task. Additionally, these criteria could be used as a dependent measure when developing new demand tasks or refining experimental procedures to assess which set of procedures results in the most systematic data. For example, one could investigate whether the implementation of an instructional demand task session with a researcher-led question-and-answer period improves the systematicity of demand data compared to standard written instructions (for examples on instructions on other data quality metrics, see Freitas-Lemos et al., 2022).

An additional use of these criteria that should be pursued with caution is to screen data for elimination from further data analysis. Common analyses for demand data typically involve either 1) fitting all data to a single mixed-effects model, or 2) extracting demand parameters from individual subjects’ demand curves and then analyzing further (i.e., a two-stage analysis). The presence of nonsystematic data can pose problems for either method. In mixed-effects models, nonsystematic data can result in the model failing to converge, yielding no result. In a two-stage analysis, it can be impossible to estimate demand parameters from nonsystematic data that do not conform to the assumptions of demand. However, removal of data that do not conform with expected results can bias the results toward the researcher’s hypothesis and violate the assumptions of statistical testing. If data are removed from a dataset in this way, a prudent step is to perform a sensitivity analysis to determine if the removal affected the conclusions of the study, followed by transparent reporting of the sensitivity analysis in any manuscript or presentation. When possible though, a more statistically sound method could be to report the prevalence of systematicity, but then include all data in the final data analysis with a method that allows for the incorporation of nonsystematic data. Mixed-effects models can allow for the incorporation of all demand data in a single analysis, even if some data do not conform to the expected data patterns or are nonsystematic. Such methods preserve the assumptions of statistical tests that rely on estimating the variance present in the whole dataset, and do not artificially reduce variance by removing those subjects with the highest deviation from group means. For a detailed explanation of the use of mixed-effects modeling for demand data, see Kaplan et al. (2021).

The criteria described here include two systematicity criteria for cross-commodity demand data, but the presence of these criteria does not imply that other criteria may not exist and be relevant in specific cases. One additional criterion that we (e.g., Rzeszutek et al., 2022, 2023) and others (e.g., Dwyer et al., 2022) have used in the past as an indicator of participants not being under experimental control of the task is the presence of extreme consumption values that are not feasible given the task constraints. For example, a common set of cigarette purchase task instructions asks participants to indicate how many cigarettes they would purchase and consume in a 24-hour period at various prices (Reed et al., 2020). If a participant follows these instructions, it would be likely be physically impossible and therefore extremely unlikely to consume more than ~10 packs of cigarettes (200 cigarettes) in one 24-hour period. However, some participants who are not under experimental control of these instructions may indicate that they would consume more than 200 cigarettes, sometimes fantastical amounts exceeding 1000 cigarettes. The presence of extreme values in a dataset such as these can have a large impact on statistical results and the estimation of group means. Therefore, like in other areas, some researchers have used this as a basis for removal of data or ‘Winsorizing’ data to values no greater than 3.29 standard deviations from the mean.

While the criteria and analyses conducted for this manuscript cannot be used to identify “best practice” for conducting cross-commodity studies to decrease nonsystematic data, we can assume that certain presentations may decrease nonsystematic responding. For example, randomized sequences appear to increase nonsystematic responding with hypothetical purchase tasks (Salzer et al., 2021; Tomlinson et al., 2023). We cannot definitively assume that factors that affect own-price demand systematicity also affect cross-commodity systematicity, but it is likely that best practices from own-price demand may also “improve” cross-commodity demand. Further examination of how cross-commodity demand is affected by methodological implementations is warranted.

There exist limitations of this proposed algorithm. First, the cutoff points we used as recommendations here for defining substitutes, complements, and the number of bounces that indicate a deviation from systematicity are arbitrary. Because of this, we have conducted threshold analyses to compare how different $\Delta Q$ categorizations and bounce criteria may affect cross-commodity systematicity decisions. While the current manuscript focuses on directly applying the Stein et al. (2015) criteria and thresholds to cross-commodity demand, we want to emphasize that the general standards here may not be universally applicable.² There could be no universal cutoff that applies in all cases, and researchers may decide that different cutoff values are more appropriate for their experimental conditions. However, the presence of a standard across studies can improve reproducibility and interpretability of studies across time and environments. Second, the criteria proposed here may not fully quantify the degree of systematicity in all cases. Edge cases will exist where the algorithm results in flagging a greater or fewer number of datapoints than what visually appear to be nonsystematic. For example, a possible edge case exists if both the first and last datapoint in a series is at zero consumption. Because the algorithm relies on the greater of these two points to determine the threshold for determining a ‘bounce’, both of these values being zero results in any nonzero change in consumption from one point to the next being classified as a ‘bounce’ (Figure 4). Also, depending on the units of measurement being used (e.g., packs of cigarettes vs individual cigarettes), this could also result in higher rates of bounce violations.

Figure 4. Possible Edge Case Affecting Cross-Commodity Identification.

Note. Y-axis is the grams of cannabis purchased (Commodity B) and the x-axis is the cost of cigarettes (Commodity A). Points that are “systematic” as determined by the algorithm are white circles, “non-systematic” points are indicated with grey fill (bounce up), black fill (bounce down), triangles up (reversal from zero), or triangles down (return to zero). Points can be more than one non-systematic category. Decision boundaries are indicated by solid vertical lines with flat tops/bottoms. Data points with a value above, below, or both are the point used to determine the range of the boundary, and dependent on if the data path was determined as complement, substitute, or independent, the direction of the boundary.

In conclusion, the criteria presented here may be used to quantify the degree of systematicity in cross-commodity demand data to help researchers assess experimental conditions to improve experimental control and to flag specific data series that violate the assumptions of demand. We recommend that these criteria be used judiciously in accordance with statistical norms to characterize their data and, if necessary and only after appropriate statistical measures such as sensitivity analysis have been conducted, flag data for removal or separate analyses.

Public Health Significance:

Cross-commodity demand is a useful method to assess how changes in consumption of one commodity affect consumption of another. However, there are currently no guidelines to assess nonsystematic cross-commodity demand data. We introduce methods and recommendations for assessing nonsystematicity for cross-commodity demand, which can help to identify procedural differences that affect cross-commodity demand systematicity.