Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Oct 4.
Published in final edited form as: Exp Clin Psychopharmacol. 2021 Apr 29;30(5):725–739. doi: 10.1037/pha0000459

Reinforcer Pathology of Internet-Related Behaviors Among College Students: Data From Six Countries

Samuel F Acuff 1, Angelina Pilatti 2, Megan Collins 3, Leanne Hides 3, Nutankumar S Thingujam 4, Wen Jia Chai 5, Wai Meng Yap 6, Ruichong Shuai 7, Lee Hogarth 7, Adrian J Bravo 8, James G Murphy 1
PMCID: PMC8553798  NIHMSID: NIHMS1705724  PMID: 33914568

Abstract

Research has demonstrated that repeated engagement in low-effort behaviors that are associated with immediate reward, such as Internet use, can result in a pathological reinforcement process in which the behavior is increasingly selected over other activities due, in part, to a low availability of alternative activities and to a strong preference for immediate rather than delayed rewards (delay discounting). However, this reinforcer pathology model has not been generalized to other Internet-related behaviors, such as online gaming or smartphone use. Given the widespread availability of these technologies, it is also important to examine whether reinforcer pathology of Internet-related behaviors is culturally universal or culture-specific. The current study examines relations between behavioral economic constructs (Internet demand, delay discounting, and alternative reinforcement) and Internet-related addictive behaviors (harmful Internet use, smartphone use, online gaming, and Internet sexual behavior) in a cross-sectional sample of college students (N = 1,406) from six different countries (Argentina, Australia, India, Malaysia, the United Kingdom, and the United States). Using structural equation modeling, Internet demand was associated with harmful Internet use, smartphone use, and online gaming; delay discounting was associated with harmful smartphone use; and alternative reinforcement was associated with harmful Internet and smartphone use. The models were partially invariant across countries. However, mean levels of behavioral economic variables differed across countries, country-level gross domestic product, person-level income, and sex at birth. Results support behavioral economic theory and highlight the importance of considering both individual and country-level sociocultural contextual factors in models for understanding harmful engagement with Internet-related behaviors.

Keywords: internet use, behavioral economics, delay discounting, alternative reinforcement, demand

Internet access in high-income countries has become almost ubiquitous, with some countries, including the United States, nearly achieving saturation (Perrin & Duggan, 2015). Although Internet access is more limited in low-income countries, access has increased at a similar rate with approximately 20% of the population gaining access every 10 years (International Telecommunication Union, 2017). The Internet provides unprecedented access to education, communication, and entertainment for the individual and has also been identified as an indicator of economic growth (Manyika & Roxburgh, 2011; The World Bank, 2016), and, as such, the United Nations has declared Internet access as a basic human right (La Rue, 2011).

Despite this, widespread Internet access has also been followed by both pop-cultural and academic reports suggesting that technology usage leads to different forms of psychological, emotional, and functional impairment (Twenge, 2019), leading to calls for recognition of official psychiatric designations for various Internet-related behaviors, such as general Internet use (Young et al., 1999), online gaming (Saunders et al., 2017), Internet sexual behavior (Delmonico & Miller, 2003), and smartphone use (Kuss et al., 2018). Apart from online gaming, Internet-related behavioral “addictions” are not generally accepted as psychiatric disorders. In part, this is due to inconsistent definitions, some of which mirror diagnostic criteria for substance use disorder from the Diagnostic and Statistical Manual—5 (American Psychiatric Association [APA], 2013) in attempts to legitimize the construct without actually fitting the phenomena, and the inaccurate classification of problematic behavior to the route of administration or medium, rather than the primary, specific behavior itself (i.e., defining any Internet-related behavior as problematic rather than specifying the specific behavior, such as online shopping or gambling; Baggio et al., 2018). Further, many fear that legitimizing the classification of behavioral addictions will pathologize common and often functional behaviors and reduce the legitimacy of substance addictions and other existing psychiatric conditions. Nonetheless, a significant number of individuals do report that Internet-related behaviors can interfere with important aspects of their lives and can result in social, emotional, and financial harm. Internet-related behaviors share many “first principles” with other substances that demonstrate addictive potential, as these behaviors provide immediate reinforcement, are generally inexpensive and highly accessible (Perrin & Duggan, 2015), and share many of the same comorbidities with substance use disorders (Lin et al., 2018). Thus, regardless of the legitimacy of harmful engagement with Internet-related behaviors as a psychiatric disorder, serious investigation of mechanisms of harmful engagement may help elucidate greater understanding of addiction more generally and may help provide a framework for treatment targets for those who do report harmful engagement.

Behavioral economics may provide a consilient framework for understanding harmful engagement in Internet-related behaviors. Behavioral economics is a molar contextual theory of addictive behavior that combines principles of operant learning theory and macroeconomics to explain choice behavior (Ainslie & Herrnstein, 1981; Rachlin, 1997). Humans make decisions in a way that maximizes utility (i.e., value), and, therefore, decisions are cost/benefit analyses intended to maximize benefits and minimize costs (Herrnstein et al., 1993; Rachlin et al., 1981). Choices are made in the context of all available reinforcers, meaning that the value of any one activity depends on the availability of alternatives (Herrnstein, 1974). Further, costs and benefits for competing reinforcers may be delivered on varying reinforcement schedules, such that some reinforcers have immediate benefits and delayed costs while others have delayed benefits but immediate costs. The choice that maximizes utility, then, is temporally dynamic and dependent upon the frame of reference (i.e., local vs. global; Ainslie & Monterosso, 2003; Vuchinich & Heather, 2003) and may not reflect utility maximization over a more extended temporal window (Herrnstein, 1990). From this perspective, a pattern of addictive behavior can be understood as a reinforcer pathology defined by a strong preference for the immediately available, addictive commodity compared to the alternative, delayed reinforcers (Bickel et al., 2014). This overvaluation is most likely when the addictive commodity is characterized by high levels of immediate reward and delayed costs, among individuals with a greater preference for immediate rewards compared to delayed rewards (i.e., local frame of reference), and in a context devoid of alternatives (Acuff et al., 2019; Ahmed, 2018; Herrnstein, 1974; Hogarth & Field, 2020).

In behavioral economic literature, this overvaluation is referred to as behavioral economic demand. Behavioral economic demand, and Internet demand specifically, is an index of relative reinforcing value, or the reinforcing value of the Internet (here operationalized as purchasing behavior) as a function of increasing constraints. Demand is assessed with purchase tasks emulating self-administration progressive ratio tasks (Petry & Bickel, 1998) which have been used in preclinical human research for studying reinforcer efficacy. Purchase tasks provide an easy and ethical alternative to these self-administration paradigms and, as a result, researchers have developed hypothetical purchase tasks for a range of substances and commodities. In a typical task, participants read a scenario and report hypothetical consumption of the reinforcer at escalating prices (Murphy & MacKillop, 2006). The resulting data generates indices reflecting various aspects of motivation, including intensity (i.e., consumption with no constraints), Omax (i.e., maximum expenditure), and elasticity (i.e., rate of change in consumption as a function of price). Meaningful between-person variation has been detected in these indices, such that greater levels of demand are associated with more problematic substance use (Zvorsky et al., 2019). Delay discounting captures the devaluation of a reward as a function of time to receipt (Mazur & Herrnstein, 1988). Delay discounting is often measured with a series of systematically modified choices between smaller, sooner and larger, later rewards that allow researchers to identify the point at which participants shift from choosing the larger, later reward to the smaller, sooner reward. Individual differences in this discounting function may be a transdiagnostic risk factor for a range of psychiatric conditions (Amlung et al., 2019) and in particular for addictive behaviors (Amlung et al., 2016; MacKillop et al., 2011). Alternative reinforcement is often measured with reinforcement schedules asking participants to report the frequency and enjoyment of various activities while they were not using the commodity or substance (Acuff et al., 2019; Correia et al., 1998). Greater levels of alternative reinforcement are protective against addictive behaviors (Acuff et al., 2019; Higgins et al., 2004; Vuchinich & Tucker, 1988). Associations with these indices of reinforcer pathology (demand, delay discounting, and alternative reinforcement) have been demonstrated across substances like alcohol (Acuff, Soltis, et al., 2018; Lemley et al., 2016), cigarette use (Bickel et al., 1999; MacKillop et al., 2012), and opioid use (Jacobs & Bickel, 1999; Meshesha et al., 2017), and have more recently been useful in understanding behavioral addictions such as gambling (Weinstock et al., 2016), ultraviolet tanning (Becirevic et al., 2017; Reed et al., 2016), and compulsive eating (Amlung et al., 2016; Snider et al., 2020), suggesting that these behaviors are liable to become overvalued and preferred relative to long-term, alternative reinforcers.

Harmful engagement in Internet-related behaviors may also reflect reinforcer pathology processes. Internet-related behaviors often provide low-effort and relatively immediate access to reinforcement from important life domains (i.e., social, leisure, and sexual) that otherwise require greater effort to obtain. Further, websites and video games are designed intentionally to increase the likelihood of continued use by providing rapidly changing novel stimuli to maintain attention while minimizing effort costs (Andersson, 2018). This shift in the cost/benefit ratio could result in an overvaluation of the Internet-related behavior relative to other higher response cost operant behaviors (e.g., in-person socialization, physical exercise, and work) and an increase in the molar pattern of time or other resources allocated to Internet behavior. Indeed, research has demonstrated that behavioral economic Internet demand accurately quantifies the valuation of Internet use and is concurrently associated with indices of harmful Internet use among college students and young adults (Acuff, MacKillop, et al., 2018; Broadbent & Dakki, 2015). Further, delay discounting has demonstrated significant associations with harmful Internet use (Acuff, MacKillop, et al., 2018; Saville et al., 2010), online gaming (Buono et al., 2017; Tian et al., 2018), smartphone use (Tang et al., 2017), and Internet sexual behavior (Negash et al., 2016). One study also found that indices of Internet demand interacted with delay discounting, such that participants with both high demand and high delay discounting reported greater harmful Internet use compared to those with only high demand or high delay discounting, or low levels of both (Acuff, MacKillop, et al., 2018). Further, lower access to alternative rewards is associated with greater levels of harmful Internet use (Acuff, MacKillop, et al., 2018), and one study showed that gaming addiction was negatively related to real-world social support, but positively related to in-game social support (Tham et al., 2020).

Behavioral economics highlights the importance of contextual constraints on choice behavior, in that the norms, choice architecture, and availability of alternatives in any given environment may result in differential engagement with a given behavior. The traditional operationalization of “context” is often the immediate choice environment and adjacently available alternative reinforcers. Although these “contexts” are important, this definition is limited, and different factors affect the choice context as the defined scope of the context changes. This may be true even as the scope reaches national and cultural levels, as policy and economic conditions at these levels may influence the availability and cost of reinforcers. Indeed, the Internet has spread rapidly across the globe and is now used across socioeconomic categories, ages, and national borders. Individuals in different countries may have vastly different constraints on technology engagement, and behavioral economic theory highlights the contextual dependence of choices to engage with these behaviors. The monetary cost of Internet access varies widely across countries (West, 2015), which may differentially impact demand as prices increase. Levels of Internet availability differ by gross domestic product (GDP), with only 16%, on average, reporting Internet use in low-income countries compared to 85% in high-income countries (International Telecommunication Union, 2017). Exploring these relations in a cross-cultural sample will help elucidate the influence of higher-level contextual variables on behavior.

Current Study

Although behavioral economic models have been widely applied to explain addictive behaviors, only a handful of studies have explored behavioral economics of engagement in Internet-related behaviors, with limited generalizability. The current study explored the relationship between behavioral economic predictors (Internet demand, delay discounting, and alternative reinforcement) and Internet-related behaviors with associated reports of harmful use (harmful Internet use, smartphone use, online gaming, and Internet sexual behavior) in a sample of college students from six different countries (Argentina, Australia, India, Malaysia, the United Kingdom, and the United States). We chose college students because they are more likely to access and regularly use the Internet across different countries regardless of GDP. The first aim was to understand mean differences in Internet demand, delay discounting, alternative reinforcement, and Internet-related behaviors across person- (e.g., sex at birth, whether the individual pays for their own Internet, and income) and country-level factors (e.g., country and GDP per capita categories). Our second aim was to examine relations between behavioral economic variables and harmful engagement in Internet-related behaviors and to examine invariance of these relations across country, GDP, and sex. We hypothesized that higher Internet demand and delay discounting, and lower alternative reinforcement, would be associated with greater harmful engagement in Internet-related behaviors; however, we did not have any a priori predictions about invariance testing across these three variables.

This study extends previous research by (a) including multiple Internet-related behaviors; and (b) exploring the relations using data from participants in six different countries. These extensions are critical for several reasons. First, although reinforcer pathology of harmful Internet use has been previously described, it is unclear whether the same relations exist for other Internet-related behaviors. Second, attempts to understand fundamental psychological processes have typically been executed in Western, educated, industrialized, rich, and democratic (WEIRD) societies, and assumptions about the generalizability of our findings to other populations may not be true (Henrich et al., 2010). To fully understand the psychological mechanisms underlying addictive behaviors we need to examine these behaviors in more diverse samples. Third, behavioral economic theory suggests that constraints (i.e., availability and cost) might exist at a macro level, which would represent an understudied between-group difference that highlights a more expansive context in which decisions are made.

Method

Participants and Procedures

Participants were college students at universities from six different countries (Argentina, Australia, India, Malaysia, the United States, and the United Kingdom) who completed a survey about mental health and Internet using behaviors. In total, 1,505 college students completed the survey (defined as at least 80% completion). All data were collected either in person or through university subject pools. Participants were required to successfully answer at least four out of five attention check items for data to be used in analyses (Hauser & Schwarz, 2016). Thus, the resulting sample included 1,406 college students from Argentina (n = 300, 76% female), Australia (n = 254, 68% female), India (n = 153, 62.7% female),1 Malaysia (n = 323, 77.8% female), the United States (n = 217, 73.6% female), and the United Kingdom (n = 173, 82% female). The institutional review boards (or institutional equivalent) at each individual university approved the research. Demographics are reported in Table 1. In the full sample, the average age was 20.92 (SD = 3.71), 73.9% identified as female, 44.7% paid for their own Internet, and the median family income was $50,000–$75,000 per year. These variables differed across countries. Notably, the Argentinian, Malaysian, and Indian samples were significantly older and reported lower median income levels compared to the other groups. Further, those in Argentina and India were more likely to report paying for their own Internet, while participants from the United Kingdom were less likely to report paying for their own Internet.

Table 1.

Descriptive Statistics for Primary Study Variables, Split by Country

U.S. (n = 217) Argentina (n = 300) Australia (n = 254) Malaysia (n = 323) U.K. (n = 173) India (n = 153)
Variables M (SD), % M (SD), % M (SD), % M (SD), % M (SD), % M (SD), % χ2 (df), p value F-value (df), p value
Sex (% female) 73.6% 76% 68% 77.8% 82% 62.7% 23.49 (5), < .001
Age 20.09 23.25 19.55 22.16 18.88 21.46 56.3 (5, 1,156), < .001
Median income $50,000–$75,000/year $25,000–$50,000/year $75,000–$100,000/year $25,000–$50,000/year $75,000–$100,000/year Less than $25,000/year 352.67 (25), < .001
% that pay for their own Internet 30% 93.7% 30.5% 27% 10.5% 67.3% 483.62 (5), < .001
Intensity 15.95 (8.82) 14.28 (8.52) 14.09 (8.86) 16.19 (8.49) 15.47 (8.42) 14.7 (8.80) 2.76 (5, 1,394), .02
Breakpoint 19.65 (25.59) 18.61 (27.66) 21.09 (26.29) 15.25 (26.34) 21.00 (27.42) 18.37 (29.51) 1.73 (5, 1,394), .12
Omax 41.69 (82.14) 32.4 (65.75) 27.18 (45.37) 24.08 (51.92) 25.55 (33.92) 32.66 (61.02) 2.77 (5, 1,394), .02
Elasticity .0134 (.0182) .0202 (.0226) .0183 (.0205) .0199 (.0209) .0163 (.0165) .0246 (.0262) 5.68 (5, 1,346), < .001
Delay discounting .5524 (.2624) .5571 (.1976) .4526 (.2481) .5361 (.3074) .4411 (.1998) .6769 (.2903) 19.61 (5, 1,383), < .001
Activity engagement 4.52 (1.76) 3.66 (1.84) 4.13 (1.71) 3.50 (1.46) 3.82 (1.31) 4.73 (1.83) 18.80 (1,405), < .001
PIUQ Total 24.75 (9.44) 29.05 (8.5) 30.73 (8.59) 33.23 (8.51) 29.76 (7.93) 31.33 (7.78) 26.58 (5, 1,380), < .001
SAS Total 22.97 (9.23) 26.7 (9.3) 30.35 (9.71) 32.76 (9.33) 27.93 (8.2) 30.4 (8.72) —
GA Total 7.07 (3.52) 7.32 (3.64) 8.21 (4.2) 8.33 (4.06) 7.11 (3.72) 8.97 (4.42) 7.73 (5, 1,383), < .001
ISB Total 0.58 (1.15) 0.52 (0.94) 0.65 (1.08) 1.09 (1.23) 0.33 (0.85) 1.02 (1.12) 17.23 (5, 1,386), < .001
Open in a new tab

Note. M = mean; SD = Standard deviation; df = degrees of freedom; PIUQ = Problematic Internet Use Questionnaire; SAS = Smartphone Addiction Scale; GA = Gaming Addiction Scale; ISB = Compulsive Internet Sexual Behavior Scale.

Measures

Measures were administered in English for participants in the United States, the United Kingdom, India, Malaysia, and Australia. Measures were translated into Spanish for data collection in Argentina. Two psychologists who were proficient in English and Spanish and knowledgeable of the scale’s rationale and development translated the original English version of the measures to Spanish. The two versions were compared with and subjected to discussion until consensus was reached, particularly emphasizing linguistic and cultural aspects. After that, cognitive interviews were carried out with ten college students. The aim was to obtain information about how well each item was understood and how appropriate it was for the local context. Based on these interviews final adjustments were made. The two psychologists that conducted the translation and the college students that participated in the cognitive interviews belonged to the same Argentinian geographic and ethnic community as the present sample.

Demographics

Participants reported their sex at birth (0 = Male, 1 = Female), age, whether they pay for Internet (0 = No, 1 = Yes), and annual parental income. In order to equate monetary values across countries we used Purchasing Power Parity (PPP), an economic index that ascertains the equivalent purchasing power of different world currencies. PPP determines the amount of currency in one country with the same purchasing power of another amount in a different currency. In order to make a comparison across the six countries, we identified the PPP equivalent of 1 U.S. Dollar (USD) at the time of study launch (2018) across the remaining five countries (Argentina = 13.56 pesos; Australia = 1.44 AUD; Malaysia = 1.46 Ringgit; the United Kingdom = 0.7 Pounds; India = 18.04 Rupees).

Harmful Internet-Related Behavior

Internet Use.

Harmful Internet use was operationalized with The Problematic Internet Use Questionnaire (PIUQ; Demetrovics et al., 2008). In the PIUQ, participants report the frequency with which each of the 18 statements applies to them (1 = Never to 5 = Always). The current study used a modified 12-item version of the PIUQ that was found to be scalar measurement invariant across all the countries used in the current study (see Supplemental materials). The internal consistency in the current study was good for both the full sample (α = .89) and for individual countries (α range = .85–.92).

Smartphone Use.

Smartphone Use was measured with 10 items from the Smartphone Addiction Scale (Kwon et al., 2013). Participants reported how much they agreed with the 10 statements on a 6-point scale (1 = Strongly Disagree to 6 = Strongly Agree). The current study used a modified eight-item version of the Smartphone Addiction Scale that was found to be metric measurement invariant across all the countries used in the current study (see Supplemental materials). The internal consistency in the current study was good for both the full sample (α = .85) and for individual countries (α range = .80–.86).

Online Gaming.

Gaming addiction was assessed with seven items from the gaming addiction scale (Lemmens et al., 2009). Participants reported how often they had experienced seven statements related to gaming addiction. Items were rated on a 5-point scale (1 = Never to 5 = Very Often). The seven-item measure has been used in a previous large-scale study examining harmful Internet-related behaviors among young adults (Baggio et al., 2018). The current study used a modified five-item version of the Gaming Addiction Scale that is scalar measurement invariant across all the countries used in the current study (see Supplemental materials). The internal consistency in the current study was good for both the full sample (α = .89) and for individual countries (α range = .85–.92).

Internet Sexual Behavior.

We used six items from the Internet Sexual Behavior Scale (Delmonico & Miller, 2003) to examine harmful Internet sexual behavior. Participants dichotomously rated whether each statement has been true for their experience. The items were summed to create a total score. (Baggio et al., 2018). The measure demonstrated scalar measurement invariance across all the countries used in the current study (see Supplemental materials). The six-item version of this measure has been used in a previous large-scale study examining harmful Internet-related behavior among young adults (Baggio et al., 2018). Cronbach’s alpha, calculated using tetrachoric correlations, was good in the current sample (α = .85).

Internet Demand.

Internet demand was measured with an Internet Purchase Task (IPT; Acuff, MacKillop, et al., 2018; Broadbent & Dakki, 2015). Participants saw the following text before making responses:

Imagine you are in a situation where you have to pay hourly to access the Internet. In order to use the Internet, you must purchase hourly time; you are unable to use a cellular data plan or any other means to access the Internet. However, these purchases are for entertainment and social purposes only (i.e., to browse the Internet, shop, play games, watch movies, and use social media). You would not need to pay for the Internet for work or educational purposes. For each of the prices below, please indicate how many hours of Internet access you would buy at each price over a 24-hour period. (For purposes other than work or school.)

After the participants read the instructions, they reported how many hours of Internet they would purchase in a period of 24 hr at 25 escalating prices (U.S. Prices: free, $0.50, $1.00, $2.00, $3.00, $4.00, $5.00, $6.00, $7.00, $8.00, $9.00, $10.00, $15.00, $20.00, $25.00, $30.00, $35.00, $40.00, $45.00, $50.00, $60.00, $70.00, $80.00, $90.00, $100.00). For the remaining five countries, prices were multiplied by the relevant PPP index in order to obtain comparable Internet demand across countries. These data generate observed indices of Internet demand, three of which were used in the current study: Intensity (consumption with no constraint, or when Internet is free), breakpoint (price at which consumption reaches zero), and Omax (maximum expenditure. We also derived elasticity (the rate of change in consumption as a function of price) using the exponentiated equation (Koffarnus et al., 2015). In the current study, k (2.08) was calculated by subtracting the log10-transformed average consumption at the highest price from the log10-transformed average consumption at the lowest price using the raw data from Internet purchase task (Koffarnus et al., 2015). Data were screened for inconsistency using the Stein macro (Stein et al., 2015) based on the following criteria: (a) trend (detection limit for ΔQ < 0.025); (b) bounce (detection limit for B = 0.10); (c) reversal from zero (detection limit number for reversals = one reversal). The Stein macro identified 53 inconsistent cases (3.7%); 47 of these cases (3.3% of the full sample) reported a single value across all prices or reported Internet consumption only at one price; these participants were removed from elasticity calculation but were included in observed analyses. The Stein macro identified six participants (.4% of the full sample) in which the data were inconsistent, and these cases were removed from all analyses. Elasticity data were calculated using the Demand Curve Analyzer software (Gilroy et al., 2018), which is freely available online at https://github.com/miyamot0/DemandCalculatorQT.2

Delay Discounting.

Delay discounting was assessed with the eight-item Delayed Reward Discounting Task (Gray et al., 2014). Participants make choices between smaller sooner, or larger later monetary rewards in each item. In order to equate reward values across countries, prices were multiplied by the relevant PPP index. We calculated a ratio of immediate choice out of total choices to quantify discounting rates. Items were presented randomly (i.e., in no specific monetary or temporal order). In order to understand differences in constraints for the delay discounting task, we ranked each item based on its immediate value and its delay length; these rankings were added together to effectively order the level of combined constraints (i.e., time and monetary reward value) from delay and immediate reward value on choices.

Activity Engagement.

Engagement in alternative activities was measured with a reinforcement survey approach (Acuff et al., 2019; MacPhillamy & Lewinsohn, 1982). Participants reported how frequently they engaged in (0 = 0 times in the past month to 4 = More than once per day) and how much they enjoyed (0 = unpleasant or neutral to 4 = extremely pleasant) each of 17 activities (e.g., “eating a meal with friends,” “attending a sporting event”) over the past month. The frequency and enjoyment rating of each activity were multiplied together to create a cross product. The activity engagement index was the average cross-product across all activities.

Data Analysis

Data Preparation

Outliers, defined as values four or greater standard deviations away from the mean were identified and corrected to one value above the highest nonoutlying value (Tabachnick & Fidell, 2013). Measurement scalar invariance for the PIUQ, GAS, and CISBS was established prior to use in the current study (see Supplemental materials); only metric invariance was met for the SAS. For this reason, we did not examine mean differences for this variable, but the metric invariance allowed for its inclusion in the SEM models.

Descriptive Data and Country-Level Differences

First, we descriptively examined the effects of (a) price constraints on reported consumption on the Internet purchase task and (b) delay and reward magnitude on choices on the delay discounting task. We examined differences in Internet demand, delay discounting, and activity engagement as a function of country, GDP per capita (country-level), sex at birth, and income. We also examined item-level differences in demand (Supplemental materials) and delay discounting as a function of the country. GDP per capita categories were determined by placing the countries with the three highest GDPs in one category and the GDP of the three lower countries in one category. We used ANOVAs to examine differences in each price in the purchase task, demand indices, the delay discounting ratio, and activity engagement across countries; Chi-square tests were used to examine item-level differences in delay discounting.

Structural Regressions

Next, we examined the relations between Internet demand, delay discounting, and activity engagement with Internet-related behaviors using a structural regression analysis in Mplus v8 (Muthén & Muthén, 2017). Missing data were handled using Full Information Maximum Likelihood, and Robust Maximum Likelihood was used as the estimator in the model. All tested models were overidentified. Data analyses followed the two-step approach (Kline, 2016) in which (a) the measurement model of the Internet demand latent variable was specified in the context of the overall model; and (b) the structural aspect of the full model is specified. The Internet demand latent variable included three indicators: Elasticity, Omax, and Intensity. These three demand indices demonstrate the most consistent relations with addictive behaviors and have been used in previous latent models examining alcohol demand (Acuff, MacKillop, et al., 2018; Acuff, Soltis, et al., 2018; Zvorsky et al., 2019). In the hypothesized model, the PIUQ total score, the SAS total score, the GAS total score, and the CISBS total score were all regressed onto Internet demand (latent), the delay discounting ratio, and the activity engagement score. We modeled covariances between all technologically mediated addictions, and between all three behavioral economic predictors. Finally, sex at birth, income, whether the individual paid for their own Internet were added as covariates.

We report five model fit indices suggested by Hu & Bentler (1999) and Kline (2016) to evaluate the overall model fit. These include the model χ2 (nonsignificant values indicative of good model fit), Tucker–Lewis Index (TLI; acceptable fit > 0.95), the Comparative Fit Index (CFI; acceptable fit > 0.90), Standardized Root Mean Square Residual (SRMR; acceptable fit < .08), and Root Mean Square Error of Approximation (RMSEA; acceptable fit < .08).

Structural Invariance Testing and Person-Level Interaction Analyses

To test the impact of country-level variables on the overall model, we ran multigroup analyses to test model invariance across countries and average GDP per capita groups. We first tested a model with all pathways unconstrained across countries and examined the model fit indices. Next, we constrained all paths to be equal and examined the change in fit indices from the unconstrained to the constrained model. To be considered model invariant, model fit indices should be relatively similar to the unconstrained model (ΔCFI should be ≤0.010, Cheung and Rensvold, 2002; ΔRMSEA ought to be ≤0.015, Chen, 2007).

Finally, we examined the influence of person-level variables on the overall model. For each moderator analysis, interaction pathways were added to each of the pathways between Internet demand, delay discounting, and activity engagement with the technologically mediated addictive behaviors. Tested moderators included sex at birth, income, and whether the individual paid for their own Internet access.3

Results

Descriptive Data and Country-Level Differences in Internet Demand and Delay Discounting

The exponentiated equation provided an excellent fit for the participant level (full sample mean R2 = .94, median R2 =.96; country-level mean R2 = .90–.94, country-level median R2 = .95–.97) and aggregate (full sample R2 = .97; country-level R2 = .95–.99) demand data. Price level data split by country are presented in Supplemental Table 1; Demand curves split by country can be found in Figure 1. Across all countries, hypothetical Internet consumption decreased as the price increased. However, countries demonstrated notable price level differences. When Internet was freely available (Intensity), Argentina had significantly lower demand than Malaysia; all other countries were in between and were not significantly different. Once a price was introduced (i.e., $0.50), all countries reduced demand; however, reported Internet consumption for participants in the United States was significantly higher than all other countries. This effect remained until Internet price reached $6.00, at which point demand levels across countries continued to level out until mean consumption values were equivalent, starting at $60.00 and remaining so throughout the remainder of the task.

Figure 1.

Figure 1

Open in a new tab

Hypothetical Internet Consumption (Hours of Internet Purchased for a 24-hr Period) at Each Price in the Hypothetical Internet Purchase Task, Split by Country. X-Axis Is Log-Transformed. Across Countries, Consumption Decreased as Price Increased

Demand demonstrated country-level differences at the index level (Table 1). The United States had a significantly higher Omax value than all the other countries, and Argentina, Malaysia, and India reported significantly greater elasticity compared to the United States, Australia, and the United Kingdom. Participants from Australia and the United Kingdom demonstrated the lowest discounting of delayed rewards, while participants from India demonstrated the highest. Item level data for the delay discounting measure are presented in Table 2. Across countries, as constraints decreased, participants became more likely to select the delayed reward. However, countries still differed in mean discounting levels. Compared to participants from other countries, participants from Malaysia and India were slightly less likely to select the immediate reward in the item with the greatest constraints; however, as the delayed reward became more valuable (i.e., immediate reward value and/or the delay decreased), these participants were still more likely to select the immediate reward compared to participants from other countries.

Table 2.

Likelihood of Selecting the Immediate Reward on Each Item, Ordered by Degree of Effect of Reward Magnitude and Delay Constraints, of the Delay Discounting Task, Split by Country

U.S. (n = 217) Argentina (n = 300) Australia (n = 254) Malaysia (n = 323) U.K. (n = 173) India (n = 153)
Price % % % % % % χ2, p value
$99 Now vs. $100/1 Year 91% 96% 90% 84% 93% 83% 36.87, <.001
$80 Now vs. $100/6 Months 86% 93% 76% 78% 83% 82% 36.87, <.001
$50 Now vs. $100/1 Year 77% 81% 69% 70% 73% 85% 23.58, <.001
$80 Now vs. $100/1 Month 72% 74% 58% 61% 59% 76% 35.50, <.001
$70 Now vs. $100/1 Month 53% 52% 37% 49% 29% 72% 75.57, <.001
$40 Now vs. $100/1 Month 25% 24% 15% 36% 10% 54% 110.28, <.001
$10 Now vs. $100/6 Months 27% 18% 14% 32% 5% 48% 114.15, <.001
$30 Now vs. $100/2 Weeks 11% 7% 4% 19% 1% 45% 190.05, <.001
Open in a new tab

Note. Degrees of freedom for all analyses are identical (df = 5).

Table 3 reports differences in key study variables across GDP, sex at birth, and person-level income. Low GDP countries reported significantly lower breakpoint values and activity engagement and higher elasticity and delay discounting compared to high-GDP countries. Males reported significantly higher intensity compared to females. Those who pay for their own Internet reported significantly lower intensity, and higher delay discounting, compared to those who do not pay for their own Internet. Finally, a graded relationship existed between parent income and breakpoint, elasticity, and delay discounting, such that those with lower income levels reported lower breakpoint values and higher elasticity and delay discounting compared to higher income levels.

Table 3.

Descriptive Statistics for Primary Study Variables, Split by Country GDP and Person-Level Characteristics

GDP per capita Sex at birth Pay for own Internet
Low GDP (n = 768) High GDP (n = 638) Male (n = 367) Female (n = 1,037) Pays (n = 624) Does not Pay (n = 776)
Variables M (SD) M (SD) M (SD) M (SD) M (SD) M (SD)
Intensity 15.15 (8.60) 15.07 (8.75) 16.14 (8.75)* 14.78 (8.61) 14.46 (8.68)* 15.65 (8.63)
Breakpoint 17.18 (27.51)* 20.59 (26.34) 20.53 (27.86) 18.10 (26.73) 19.05 (27.91) 18.47 (26.31)
Omax 29.02 (59.52) 31.51 (58.26) 34.69 (63.91) 28.58 (57.09) 32.40 (64.93) 28.34 (53.62)
Elasticity .0209 (.0227)* .0162 (.0188) .0170 (.0196) .0194 (.0216) .0200 (.0220) .0177 (.0203)
Delay discounting .57 (.27)* .48 (.25) .54 (.27) .53 (.26) .56 (.25)* .51 (.27)
Activity engagement 3.81 (1.75)* 4.17 (1.65) 3.98 (1.81) 3.97 (1.69) 3.91 (1.78) 4.02 (1.66)
Parent income level
Less than $25,000 (n = 311) $25,000–$100,000 (n = 744) Greater than $100,000 (n = 317) F-value (df), p value
Intensity 15.07 (8.58) 15.03 (8.69) 15.19 (8.66) .04 (2, 1,369), .96
Breakpoint 14.44 (24.00) 18.93 (27.41) 22.95 (28.85) 7.80 (2, 1,369), < .001
Omax 23.66 (51.61) 31.81 (61.72) 33.06 (59.68) 2.55 (2, 1,369), .08
Elasticity .0232 (.0251) .0181 (.0195) .0163 (.0205) 9.30 (2, 1,323), < .001
Delay discounting .57 (.29) .53 (.25) .48 (.26) 9.61 (2, 1,358), < .001
Activity engagement 3.91 (1.87) 3.96 (1.63) 4.03 (1.76) .42 (1,377), .66
Open in a new tab

Note. GDP categories were determined by grouping the lowest three and highest three GDP per capita countries together. India (GDP per capita = $2,104), Malaysia (GDP per capita = $11,072) and Argentina (GDP per capita = $11,658) were placed in the low-GDP group, while the United Kingdom (GDP per capita = $42,580), the United States (GDP per capita = $54,542), and Australia (GPD per capita = $56,420) were placed in the high-GDP group.

* =

significant difference between groups.

Reinforcer Pathology Structural Regression

Table 4 shows fit index statistics for all tested structural models. The initial measurement model for the hypothesized model demonstrated excellent fit, and values for all indices were within acceptable limits. Elasticity (unstandardized beta [B] = −.46, S.E. = .08, p < .001), Omax (B = 9.26, S.E. = 1.38, p < .001), and intensity (constrained to 1) all loaded well on the Internet demand latent factor (standardized estimates for latent factor loadings are presented in Figure 3).

Table 4.

Model Fit and Structural Invariance Testing Results

Overall Fit Index Model Comparative Fit
Model χ2 df CFI TLI RMSEA (90% CI) SRMR Model Comparison Δχ2 Δdf ΔCFI ΔRMSEA
Full sample
M1 Measurement Model 54.48 18 .978 .924 .038 (.027 & .05) .024 — — — — —
S1 Final Model 54.48 18 .978 .924 .038 (.027 & .05) .024 M1 vs. S1 0 0 0 0
Structural invariance across countries
SC1 unconstrained 204.24 128 .956 .870 .051 (.037 & .064) .039 — — — — —
SC2 Fully constrained 480.43 338 .918 .908 .043 (.034 & .051) .072 SC1 vs. SC2 276.19 210 .038 .08
SC3 Fully constrained, Freed: 18 453.42 333 .931 .921 .04 (.03 & .049) .07 SC1 vs. SC3 249.18 205 .025 .11
SC4 Fully constrained, Freed: 18, 23 436.47 328 .938 .928 .038 (.028 & .047) .069 SC1 vs. SC4 232.23 200 .018 .13
SC5 Fully constrained, Freed: 18, 23, 39 418.00 323 .945 .936 .036 (.025 & .045) .068 SC1 vs. SC5 213.76 195 .011 .15
SC6 Fully constrained, Freed: 18, 23, 39, 6 408.64 318 .948 .938 .035 (.024 & .045) .068 SC1 vs. SC6 204.4 190 .008 .16
Structural invariance across GDP per capita level
SG1 unconstrained 87.00 40 .972 .912 .041 (.029 & .053) .028 — — — — —
SG2 Fully constrained 166.82 82 .950 .923 .039 (.030 & .047) .043 SG1 vs. SG2 79.82 42 .022 .002
SG3 Fully constrained, Freed: 18 156.29 81 .955 .931 .037 (.028 & .045) .041 SG1 vs. SG3 69.29 41 .017 .004
SG4 Fully constrained, Freed: 18, 40 149.19 80 .959 .936 .035 (.027 & .044) .041 SG1 vs. SG4 62.19 40 .013 .006
SG5 Fully constrained, Freed: 18, 40, 42 143.36 79 .962 .939 .034 (.025 & .043) .040 SG1 vs. SG5 56.36 39 .01 .007
Open in a new tab

Note. df = degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CFI = Comparative Fit Index; SRMR = Standardized Root Mean Squared Residual; AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion; SC = Structure Invariance for Country; SG = Structural Invariance for GDP.

Figure 3.

Figure 3

Open in a new tab

The Hypothesized Model Demonstrating Relations Between Both Delay Discounting and Internet Demand (Latent) With Harmful Internet Use, Harmful Smartphone Use, Online Gaming, and Internet Sexual Behavior. Estimates in Italics Are Standardized, While the Estimates in Standard Font Are Unstandardized. The Model Also Controlled for Sex at Birth, Person-Level Income, and Whether the Individual Paid for Their Own Internet (Not Modeled Above)

Note. * p < .05. ** p < .01. *** p < .001

The structural model demonstrated identical fit to the measurement model because the models contained the same number of modeled pathways. The model suggested that higher Internet demand was associated with higher scores on the problematic Internet use questionnaire (standardized beta [β] = .23, p < .001), the smartphone addiction scale (β = .23, p < .001), the gaming addiction scale (β = .14, p < .001), but not on the Internet sexual behavior scale (β = .03, p = .35). A visual demonstration of Internet demand curves by severity scores for each Internet-related behavior can be found in Figure 2. Higher delay discounting ratio was associated with higher scores on the smartphone addiction scale (β = .07, p = .01), but was not associated with scores on the problematic Internet use questionnaire (β = .03, p = .33), the gaming addiction scale (β = .02, p = .36), or sexual behavior scale (β = .02, p = .53). Lower activity engagement was associated with higher scores on the Problematic Internet Use Questionnaire (β = −.09, p = .005), the smartphone addiction scale (β = −.09, p = .003), and the gaming addiction scale (β = −.07, p = .02), but not the Internet sexual behavior scale (β = −.04, p = .12). Higher scores on the Problematic Internet Use Questionnaire were significantly associated with higher scores on the smartphone addiction scale (β = .66, p < .001), the gaming addiction scale (β = .27, p < .001), and the Internet sexual behavior scale (β = .24, p < .001). Higher scores on the smartphone addiction scale were significantly associated with higher scores on the gaming addiction scale (β = .26, p < .001) and the Internet sexual behavior scale (β = .23, p < .001). Finally, higher scores on the gaming addiction scale were significantly associated with higher scores on the Internet sexual behavior scale (β = .11, p < .001). Exogenous behavioral economic variables were not associated with one another.4

Figure 2.

Figure 2

Open in a new tab

Demand Curves for Average Consumption on the Hypothetical Internet Purchase Task, Separated Into Groups of Those Who Scored in the Bottom and Top Thirds on the (a) Problematic Internet Use Questionnaire (PIUQ), (b) Smartphone Addiction Scale (SAS), (c) Online Gaming Scale (GAS), and (d) Internet Sexual Behavior (CISB) Scale. The x-Axis Is Log-Transformed. For a, b, and c, Those Who Score in the Bottom Third on the Corresponding Scale Reported Less Internet Consumption at Lower Prices. As Price (x-Axis) Increased, Consumption Decreased at a Slower Rate for Those Who Score in the Top Third of the Corresponding Scale Compared With Those Who Score in the Bottom Third. There Were No Differences in Responding on the CISB Scale Between Those Who Scored in the Lower and Top Third of the Measure

Structural Invariance of the Hypothesized Reinforcer Pathology Model

Fit for the structural invariance testing of the model can be found in Table 4. We first examined structural invariance across countries. The unconstrained model, which allowed all pathways to freely vary across countries, demonstrated adequate model fit. The results for the fully constrained model suggested that the model was variant across countries (ΔCFI = .038). To identify an invariant model, we systematically identified the paths with the greatest contribution to reducing model fit within the fully constrained model and allowed these paths to be freely estimated: Activity engagement on gaming addiction (SC3; constraint number 18); activity engagement on Internet sexual behavior (SC4; constraint number 23); Paying for Internet on smartphone addiction (SC5; constraint number 39); and income on Problematic Internet Use (SC6; constraint number 6). Freeing these paths resulted in a partially invariant model. Across countries, activity engagement generally demonstrated a negative effect on gaming addiction with the exception of Malaysia, although the association was nonsignificant (β = .11; B = .31, S.E. = .17, p = .06). Across countries, the association between activity engagement and Internet sexual behavior generally hovered around zero (β ranges: −.24–.048). The greatest effect, in the United Kingdom was significant (β = −.24, B = −.15, S.E. = .06 p = .01); this association was not significant for any other country. Across countries, the association between paying for your own Internet and smartphone addiction generally hovered around zero (β ranges: −.12–.19). The greatest effects in India (β = .19, B = −3.29, S.E. = 1.17, p = .005) and the United Kingdom (β = −= .00B = 2.69, S.E. = 1.17, p = .02), were significant; this association was not significant for any other country. Across countries, the association between income and problematic Internet use generally hovered around zero (β ranges: −.14–.09). The greatest effect, in India, was significant (β = −.14, B = −1.02, S.E. = .50, p = .04); this association was not significant for any other country.

We also examined structural invariance across GDP per capita. The unconstrained model demonstrated adequate model fit. The results for the fully constrained model suggested that the model was variant across GDP levels (ΔCFI = .022). Using the same approach to achieve model invariance, we identified three pathways (i.e., constraint number 18: activity engagement on gaming addiction; constraint number 40: paying for Internet on gaming addiction; and constraint number 42: paying for Internet on Internet sexual behavior) that contributed to model variance. For countries with low GDPs, activity engagement was not significantly associated with gaming addiction (β = .01, B = .03, S.E. = .08, p = .74), whereas this association was significant for countries with higher GDPs (β = −.15, B = −.33, S.E. = .08, p < .001). For countries with low GDPs, paying for your own Internet was significantly associated with gaming addiction (β = .10, B = .84, S.E. = .28, p = .003), whereas this association was nonsignificant for countries with higher GDPs (β = −.04, B = −.30, S.E. = .32, p = .35). Finally, for countries with low GDPs, paying for your own Internet was significantly associated with Internet sexual behavior (β = .09, B = .20, S.E. = .08, p = .014), whereas this association was nonsignificant for countries with higher GDPs (β = −.04, B = −.10, S.E. = .10, p = .32).

Person-Level Moderation Analyses of the Hypothesized Model

We next examined whether delay discounting, Internet demand, and activity engagement interact with sex at birth, income, and whether the individual pays for their own Internet on each Internet-related behavior in a comprehensive model. None of the moderators significantly interacted with Internet demand, delay discounting, or activity engagement to predict any of the technologically mediated addictions.

Discussion

The current study extends behavioral economic models of potentially harmful Internet-related behavior in a cross-cultural sample of college students. Across countries, reported consumption on the hypothetical purchase task conformed to the demand curve model that has accurately described the association between price and consumption across other potentially addictive behaviors (Murphy & MacKillop, 2006). Hypothetical Internet consumption reduced as the price increased across all six countries. Interestingly, the reduction in Internet purchasing behavior that coincides with the introduction of prices in the current study is steeper than what is typically observed with other substances (Acuff & Murphy, 2017; Few et al., 2012). This may demonstrate a fundamental difference between the Internet and substances. The Internet can be used passively (streaming television) and for many activities, and there is no pharmacological satiation that occurs, resulting in greater initial amplitudes across all participants. Further, the reduction observed in this international study is greater than that of previous studies that examined Internet demand in the U.S. (Acuff, MacKillop, et al., 2018; Broadbent & Dakki, 2015). The Internet purchase tasks used in previous studies provided a limited number of hours to purchase (e.g., 5 hr total), which restricted the range of the possible amplitudes that could be observed. Thus, it is possible that the steep reduction in purchasing behavior that occurs at early prices was not found in previous studies because of the limited range which participants could reduce.

Although based on nonrepresentative samples, the present results suggest the possibility of differences in behavioral economic variables across countries. It is likely that the country variable is a proxy for economic, cultural, or political factors. The current study examined the economic factor country-level GDP and found that those in low-GDP countries demonstrated lower demand, higher discounting, and lower alternative reinforcement compared to those in high-GDP countries, an effect that was replicated across differences in income at the person-level for demand and delay discounting. Although these results are preliminary, they suggest that molar economic factors can impact the individual choice context. Lower GDPs may represent conditions of scarcity, which have a well-documented effect on delay discounting and demand (Snider et al., 2020; Sze et al., 2017). Although the Internet spurs economic growth (Manyika & Roxburgh, 2011), these results may reflect that the Internet is still considered secondary to more essential needs among those in the lowest income brackets, may be seen as more of a luxury in some countries, and also differs widely in price across countries (Monthly Internet prices: Argentina, $13.69; India, $12.90; U.S., $50.00; U.K., $35.71; Malaysia, $31.75; Australia, $48.35; Howdle, 2020). Previous research has linked low socioeconomic status with greater delay discounting (Green et al., 1996) and alternative reinforcement (Leventhal et al., 2015). These results replicate and extend these findings and suggest that conditions of economic scarcity on a country level may generally shift the individual toward a more local frame of reference that maximizes utility in the short rather than the long term, and may result in diminished availability of or resources to engage in alternatives. Other factors, such as acceptability and normative use of these Internet-related behaviors across cultures, may also have an impact on the choice context but were not captured in the current study. Future research should use representative epidemiological samples to more adequately characterize the country-level factors influencing individual choice behavior.

Consistent with research on other addictive commodities (Acuff et al., 2019; Correia et al., 1998), greater activity engagement was associated with less harmful smartphone and Internet use. Many of the activities measured by the activity engagement measure required attentional allocation away from technology (i.e., “going to a park or being in nature”; “performing a hobby”) and thus serve as substitutes for the behaviors in question. However, some of the activities require money and may be confounded by socioeconomic status (i.e., “attending a cultural event”; “attending a sporting event”), which may serve to increase error in predicting harmful Internet-related behavior, as Internet-related activities often require hardware and financial investment (e.g., smartphones, computers, and gaming consoles). Indeed, participants in countries with higher overall GDP did report greater activity engagement, and structural invariance analyses highlighted a robust, significant association of alternative reinforcement on online gaming in high-GDP countries compared to no effect in low-GDP countries. Future research should attempt to parse apart the complex relationship between socioeconomic status, alternative reinforcement, and harmful engagement with Internet-related behaviors.

Consistent with previous research (Acuff, MacKillop, et al., 2018; Broadbent & Dakki, 2015), Internet demand was significantly and positively associated with harmful Internet use, smartphone use, and gaming. The results suggest that these three phenomena are conceptually related and that Internet demand may serve as a universally sensitive marker of severity. Internet sexual behavior, however, was not related to indices of the Internet purchase task. The Internet purchase task quantifies the hypothetical reinforcing value of Internet access, and the lack of correlation between Internet sexual behavior and Internet demand suggests that this behavior may be more accurately captured by indices of sexual reinforcer pathology rather than Internet reinforcer pathology (Mulhauser et al., 2018). The results suggest that the motivational unit is likely the sexual content itself, rather than the Internet access, that drives behavior.

Consistent with previous research (Tang et al., 2017), delay discounting was related to harmful smartphone use, but not harmful Internet use, online gaming, or Internet sexual behavior. Smartphones are often the most easily accessible form of technology in that they are typical with the person and do not require any space, startup time, or other hardware. It is possible that the delay discounting measure is most sensitive to the instantaneous nature of a choice to use a smartphone. Unlike previous studies, our study found no relation between delay discounting and harmful Internet use and online gaming (Buono et al., 2017; Saville et al., 2010; Tian et al., 2018). These studies primarily examined mean differences between groups of video game users and comparisons, but did not control for income. In the present study, income was related to delay discounting, which may explain the differences in results across studies.

Strengths, Limitations, and Future Directions

The current study used a large overall sample with representatives from various cultures. Further, conclusions were drawn from robust data analytic approaches accounting for variance across countries, using multiple behavioral economic measurement approaches. However, the study also had several significant limitations. First, we examined these relations among a predominantly female sample of college students. Second, the purchase task used for the current study measured demand for general Internet use rather than for specific online activities. Third, the analyses used preexisting problem measures that do not classify problems relative to all possible problems from addictive behaviors, making it hard to determine whether the behavioral economic indices are measuring “harm” comparable to other addictions. Fourth, the study did not include important indices of cultural differences, such as measures of individualism versus collectivism. Future studies should use larger samples from a greater number of countries and include more country-level variables to determine overarching deterministic factors for Internet demand. Fifth, while the equating of monetary expenditure across countries is a strength, this also means that digit effects may vary across countries based on the conversion rates and introduces the problem of the left digit effect. Although this effect is relatively small, it should be considered in future analyses (Salzer et al., 2019). Sixth, some colleges and countries provide free Internet which may have influenced individual’s abilities to complete the Internet Purchase Task effectively. Finally, the study was cross-sectional, and neither temporal precedence nor causation cannot be established.

Conclusion

Our results suggest that engagement with some categories of Internet-related behaviors may result in reinforcer pathology in which the immediately reinforcing behavior may be overvalued relative to alternatives. Although these relations were largely consistent across countries, there were important country-level differences in the mean levels of behavioral economic indices. There was also meaningful variance in relationships between behavioral economic indices and measures of Internet-related behaviors, such as a large association between alternative activity engagement and harmful engagement in Internet-related behaviors for high-GDP countries, compared to a null effect in low-GDP countries. The results highlight potentially important country and social–cultural influences on the reinforcing efficacy of Internet use and points to macro factors in the choice context with impact on decision-making.

Supplementary Material

Supplemental Materials

Public Significance Statement.

Engagement in some Internet-related behaviors may result in an overvaluation of these behaviors relative to alternatives, although this does not imply harm equivalent to substance addictions. These relations were largely consistent across countries, although there were mean differences in behavioral economic indices. The results highlight potentially important country and social–cultural influences on the reinforcing efficacy of Internet use and points to macro factors in the choice context with an impact on decision-making.

Acknowledgments

This work was supported by the National Institute of Health Grant F31 AA027140 (PI: Samuel F. Acuff). The funding source had no role other than financial support. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Alcohol Abuse and Alcoholism or the National Institutes of Health.

Footnotes

1

The attention check items were not included in the survey for data collected in India; thus, all participants who completed the survey were included in final analyses.

2

Some participants report a single consumption level across all prices (e.g., 1 hr of Internet no matter the price) or one price followed by zeros. Although these are consistent and systematic responding and may represent real choice options for some participants, the elasticity formula requires a certain degree of variability in order to for the formula to provide a value. Thus, the data was retained for non-elasticity demand indices, but dropped when calculating elasticity. This is common practice in studies examining demand (Acuff et al., 2020; Dahne et al., 2017).

3

We initially considered multilevel analysis. Although the complexity of multilevel analysis prevents a simple rule for optimal group sample sizes, a general recommendation is at least 30 groups (Hox, 2010). Thus, we were unable to complete multilevel modeling and instead opted for the invariance testing approach, which has been used in a number of cross-cultural analyses (Bravo et al., 2018; Mezquita et al., 2018).

4

We also examined interactions between behavioral economic variables of delay discounting and Internet demand in predicting Internet-related addictive behaviors. These analyses were nonsignificant.

The authors declare no conflict of interest.

References

  1. Acuff SF, Dennhardt AA, Correia CJ, & Murphy JG (2019). Measurement of substance-free reinforcement in addiction: A systematic review. Clinical Psychology Review, 70, 79–90. 10.1016/j.cpr.2019.04.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Acuff SF, MacKillop J, & Murphy JG (2018). Applying behavioral economic theory to problematic internet use: An initial investigation. Psychology of Addictive Behaviors, 32(7), 846–857. 10.1037/adb0000404 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Acuff SF, & Murphy JG (2017). Further examination of the temporal stability of alcohol demand. Behavioural Processes, 141(Part 1), 33–41. 10.1016/j.beproc.2017.03.020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Acuff SF, Soltis KE, Dennhardt AA, Berlin KS, & Murphy JG (2018). Evaluating behavioral economic models of heavy drinking among college students. Alcoholism: Clinical and Experimental Research, 42(7), 1304–1314. 10.1111/acer.13774 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Acuff SF, Soltis KE, & Murphy JG (2020). Using demand curves to quantify the reinforcing value of social and solitary drinking. Alcoholism: Clinical and Experimental Research. Advance online publication. 10.1111/acer.14382 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Ahmed SH (2018). Trying to make sense of rodents’ drug choice behavior. Progress in Neuro-Psychopharmacology and Biological Psychiatry, 87-(September 2017), 3–10. 10.1016/j.pnpbp.2017.09.027 [DOI] [PubMed] [Google Scholar]
  7. Ainslie G, & Herrnstein RJ (1981). Preference reversal and delayed reinforcement. Animal Learning & Behavior, 9(4), 476–482. 10.3758/BF03209777 [DOI] [Google Scholar]
  8. Ainslie G, & Monterosso JR (2003). Building blocks of self-control: Increased tolerance for delay with bundled rewards. Journal of the Experimental Analysis of Behavior, 79(1), 37–48. 10.1901/jeab.2003.79-37 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. American Psychiatric Association. (2013). DSM-5. In Diagnostic and statistical manual of mental disorders. American Psychiatric Publishing. [Google Scholar]
  10. Amlung MT, Marsden E, Holshausen K, Morris V, Patel H, Vedelago L, Naish KR, Reed DD, & McCabe RE (2019). Delay discounting as a transdiagnostic process in psychiatric disorders: A Meta-analysis. JAMA Psychiatry. Advance online publication. 10.1001/jamapsychiatry.2019.2102 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Amlung MT, Petker T, Jackson JNS, Balodis I, & MacKillop J (2016). Steep discounting of delayed monetary and food rewards in obesity: A meta-analysis. Psychological Medicine, 46(11), 2423–2434. 10.1017/S0033291716000866 [DOI] [PubMed] [Google Scholar]
  12. Andersson H (2018). Social media apps are “deliberately” addictive to users. BBC. [Google Scholar]
  13. Baggio S, Starcevic V, Studer J, Simon O, Gainsbury SM, Gmel G, & Billieux J (2018). Technology-mediated addictive behaviors constitute a spectrum of related yet distinct conditions: A network perspective. Psychology of Addictive Behaviors. Advance online publication. 10.1037/adb0000379 [DOI] [PubMed] [Google Scholar]
  14. Becirevic A, Reed DD, Amlung MT, Murphy JG, Stapleton JL, & Hillhouse JJ (2017). An initial study of behavioral addiction symptom severity and demand for indoor tanning. Experimental and Clinical Psychopharmacology, 25(5), 346–352. 10.1037/pha0000146 [DOI] [PubMed] [Google Scholar]
  15. Bickel WK, Johnson MW, Koffarnus MN, MacKillop J, & Murphy JG (2014). The behavioral economics of substance use disorders: Reinforcement pathologies and their repair. Annual Review of Clinical Psychology, 10, 641–677. 10.1146/annurev-clinpsy-032813-153724 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Bickel WK, Odum AL, & Madden GJ (1999). Impulsivity and cigarette smoking: Delay discounting in current, never, and ex-smokers. Psychopharmacology, 146(4), 447–454. 10.1007/pl00005490 [DOI] [PubMed] [Google Scholar]
  17. Bravo AJ, Pearson MR, Pilatti A, Read JP, Mezquita L, Ibáñez MI, & Ortet G (2018). Impulsivity-related traits, college alcohol beliefs, and alcohol outcomes: Examination of a prospective multiple mediation model among college students in Spain, Argentina, and USA. Addictive Behaviors. Advance online publication. 10.1016/j.addbeh.2018.02.009 [DOI] [PubMed] [Google Scholar]
  18. Broadbent J, & Dakki MA (2015). How much is too much to pay for internet access? A behavioral economic analysis of internet use. Cyberpsychology, Behavior, and Social Networking, 18(8), 457–461. 10.1089/cyber.2014.0367 [DOI] [PubMed] [Google Scholar]
  19. Buono FD, Sprong ME, Lloyd DP, Cutter CJ, Printz DMB, Sullivan RM, & Moore BA (2017). Delay discounting of video game players: Comparison of time duration among gamers. Cyberpsychology, Behavior, and Social Networking, 20(2), 104–108. 10.1089/cyber.2016.0451 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Correia CJ, Simons JS, Carey KB, & Borsari BE (1998). Predicting drug use: Application of behavioral theories of choice. Addictive Behaviors, 23(5), 705–709. 10.1016/S0306-4603(98)00027-6 [DOI] [PubMed] [Google Scholar]
  21. Dahne J, Murphy JG, & MacPherson L (2017). Depressive symptoms and cigarette demand as a function of induced stress. Nicotine & Tobacco Research, 19(1), 49–58. 10.1093/ntr/ntw145 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Delmonico DL, & Miller JA (2003). The internet sex screening test: A comparison of sexual compulsives versus non-sexual compulsives. Sexual and Relationship Therapy, 18(3), 261–276. 10.1080/1468199031000153900 [DOI] [Google Scholar]
  23. Demetrovics Z, Szeredi B, & Rózsa S (2008). The three-factor model of internet addiction: The development of the Problematic Internet Use Questionnaire. Behavior Research Methods, 40(2), 563–574. 10.3758/BRM.40.2.563 [DOI] [PubMed] [Google Scholar]
  24. Few LR, Acker JD, Murphy CM, & MacKillop J (2012). Temporal stability of a cigarette purchase task. Nicotine & Tobacco Research, 14(6), 761–765. 10.1093/ntr/ntr222 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Gilroy SP, Kaplan BA, Reed DD, Koffarnus MN, & Hantula DA (2018). The demand curve analyzer: Behavioral economic software for applied research. Journal of the Experimental Analysis of Behavior. Advance online publication. 10.1002/jeab.479 [DOI] [PubMed] [Google Scholar]
  26. Gray JC, Amlung MT, Acker JD, Sweet LH, & MacKillop J (2014). Item-based analysis of delayed reward discounting decision making. Behavioural Processes, 100, 256–260. 10.1016/j.beproc.2014.01.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Green L, Myerson J, Lichtman D, Rosen S, & Fry AF (1996). Temporal discounting in choice between delayed rewards: The role of age and income. Psychology and Aging, 11(1), 79–84. 10.1037/0882-7974.11.1.79 [DOI] [PubMed] [Google Scholar]
  28. Hauser DJ, & Schwarz N (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. 10.3758/s13428-015-0578-z [DOI] [PubMed] [Google Scholar]
  29. Henrich J, Heine SJ, & Norenzayan A (2010). Most people are not WEIRD. Nature, 466, 29. 10.1038/466029a [DOI] [PubMed] [Google Scholar]
  30. Herrnstein RJ (1974). Formal properties of the matching law. Journal of the Experimental Analysis of Behavior, 21(I), 159–164. 10.1901/jeab.1974.21-159 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Herrnstein RJ (1990). Rational choice theory: Necessary but not sufficient. American Psychologist, 45(3), 356–367. 10.1037/0003-066X.45.3.356 [DOI] [Google Scholar]
  32. Herrnstein RJ, Loewenstein G, Prelec D, & Vaughan W (1993). Utility Maximation and Melioration: Internalities in Individual Choice. Journal of Behavioral Decision Making, 6, 149–185. 10.1002/bdm.3960060302 [DOI] [Google Scholar]
  33. Higgins ST, Heil SH, & Lussier JP (2004). Clinical implications of reinforcement as a determinant of substance use disorders. Annual Review of Psychology, 55(March 2017), 431–461. 10.1146/annurev.psych.55.090902.142033 [DOI] [PubMed] [Google Scholar]
  34. Hogarth L, & Field M (2020). Relative expected value of drugs versus competing rewards underpins vulnerability to and recovery from addiction. Behavioural Brain Research, 394, Article 112815. 10.1016/j.bbr.2020.112815 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Howdle D (2020). The cost of fixed-line broadband in 206 countries. Cable.Co.Uk. https://www.cable.co.uk/broadband/pricing/worldwide-comparison/
  36. Hox J, Moerbeek M, & van de Schoot R (2010). Multilevel Analysis: Techniques and Applications (2nd ed.). Routledge. 10.4324/9780203852279 [DOI] [Google Scholar]
  37. Hu L, & Bentler PM (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. 10.1080/10705519909540118 [DOI] [Google Scholar]
  38. International Telecommunication Union. (2017). Individuals using the internet (% of population). The World Bank. [Google Scholar]
  39. Jacobs EA, & Bickel WK (1999). Modeling drug consumption in the clinic using simulation procedures: Demand for heroin and cigarettes in opioid-dependent outpatients. Experimental and Clinical Psychopharmacology, 7(4), 412–426. 10.1037/1064-1297.7.4.412 [DOI] [PubMed] [Google Scholar]
  40. Kline RB (2016). Principles and practice of structural equation modeling (4th ed.). Guilford Press. [Google Scholar]
  41. Koffarnus MN, Franck CT, Stein J, & Bickel WK (2015). A modified exponential behavioral economic demand model to better describe consumption data. Experimental and Clinical Psychopharmacology, 23(6), 504–512. 10.1037/pha0000045 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Kuss DJ, Harkin L, Kanjo E, & Billieux J (2018). Problematic smartphone use: Investigating contemporary experiences using a convergent design. International Journal of Environmental Research and Public Health. Advance online publication. 10.3390/ijerph15010142 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Kwon M, Lee JY, Won WY, Park JW, Min JA, Hahn C, Gu X, Choi JH, & Kim DJ (2013). Development and Validation of a Smartphone Addiction Scale (SAS). PLOS ONE. Advance online publication. 10.1371/journal.pone.0056936 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. La Rue F (2011). Report of the Special Rapporteur on the promotion and protection of the right to freedom of opinion and expression. https://www.ohchr.org/en/issues/freedomopinion/pages/opinionindex.aspx
  45. Lemley SM, Kaplan BA, Reed DD, Darden AC, & Jarmolowicz DP (2016). Reinforcer pathologies: Predicting alcohol related problems in college drinking men and women. Drug and Alcohol Dependence, 167, 57–66. 10.1016/j.drugalcdep.2016.07.025 [DOI] [PubMed] [Google Scholar]
  46. Lemmens JS, Valkenburg PM, & Peter J (2009). Development and validation of a game addiction scale for adolescents. Media Psychology, 12, 77–95. 10.1080/15213260802669458 [DOI] [Google Scholar]
  47. Leventhal AM, Bello MS, Unger JB, Strong DR, Kirkpatrick MG, & Audrain-McGovern JE (2015). Diminished alternative reinforcement as a mechanism underlying socioeconomic disparities in adolescent substance use. Preventive Medicine: An International Journal Devoted to Practice and Theory, 80, 75–81. 10.1016/j.ypmed.2015.05.021 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Lin MP, Wu JYW, You J, Hu WH, & Yen CF (2018). Prevalence of internet addiction and its risk and protective factors in a representative sample of senior high school students in Taiwan. Journal of Adolescence, 62(November 2017), 38–46. 10.1016/j.adolescence.2017.11.004 [DOI] [PubMed] [Google Scholar]
  49. MacKillop J, Amlung MT, Few LR, Ray LA, Sweet LH, & Munafò MR (2011). Delayed reward discounting and addictive behavior: A meta-analysis. Psychopharmacology, 216(3), 305–321. 10.1007/s00213-011-2229-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. MacKillop J, Few LR, Murphy JG, Wier LM, Acker JD, Murphy CM, Stojek MK, Carrigan M, & Chaloupka F (2012). High-resolution behavioral economic analysis of cigarette demand to inform tax policy. Addiction, 107(12), 2191–2200. 10.1111/j.1360-0443.2012.03991.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. MacPhillamy DJ, & Lewinsohn PM (1982). The pleasant events schedule: Studies on reliability, validity, and scale intercorrelation. Journal of Consulting and Clinical Psychology, 50(3), 363–380. 10.1037/0022-006X.50.3.363 [DOI] [Google Scholar]
  52. Manyika J, & Roxburgh C (2011). The great transformer: The impact of the internet on economic growth and prosperity. McKinsey & Company. [Google Scholar]
  53. Mazur JE, & Herrnstein RJ (1988). On the functions relating delay, reinforcer value, and behavior. Behavioral and Brain Sciences, 11(4), 690–691. 10.1017/S0140525X00054091 [DOI] [Google Scholar]
  54. Meshesha LZ, Pickover AM, Teeters JB, & Murphy JG (2017). A longitudinal behavioral economic analysis of non-medical prescription opioid use among college students. The Psychological Record, 67(2), 241–251. 10.1007/s40732-017-0235-2 [DOI] [Google Scholar]
  55. Mezquita L, Bravo AJ, Ortet G, Pilatti A, Pearson MR, & Ibáñez MI (2018). Cross-cultural examination of different personality pathways to alcohol use and misuse in emerging adulthood. Drug and Alcohol Dependence, 192(September), 193–200. 10.1016/j.drugalcdep.2018.08.004 [DOI] [PubMed] [Google Scholar]
  56. Mulhauser K, Miller Short E, & Weinstock J (2018). Development and psychometric evaluation of the Pornography Purchase Task. Addictive Behaviors, 84(December 2017), 207–214. 10.1016/j.addbeh.2018.04.016 [DOI] [PubMed] [Google Scholar]
  57. Murphy JG, & MacKillop J (2006). Relative reinforcing efficacy of alcohol among college student drinkers. Experimental and Clinical Psychopharmacology, 14(2), 219–227. 10.1037/1064-1297.14.2.219 [DOI] [PubMed] [Google Scholar]
  58. Muthén LK, & Muthén BO (2017). Mplus user’s guide (8th ed.). Muthén & Muthén. 10.1111/j.1600-0447.2011.01711.x [DOI] [Google Scholar]
  59. Negash S, Sheppard NVN, Lambert NM, & Fincham FD (2016). Trading later rewards for current pleasure: Pornography consumption and delay discounting. Journal of Sex Research, 53(6), 689–700. 10.1080/00224499.2015.1025123 [DOI] [PubMed] [Google Scholar]
  60. Perrin A, & Duggan M (2015). Americans’ internet access: 2000–2015. Pew Research Center. [Google Scholar]
  61. Petry NM, & Bickel WK (1998). Polydrug abuse in heroin addicts: A behavioral economic analysis. Addiction, 93(3), 321–335. 10.1046/j.1360-0443.1998.9333212.x [DOI] [PubMed] [Google Scholar]
  62. Rachlin H (1997). Four teleological theories of addiction. Psychonomic Bulletin & Review, 4(4), 462–473. 10.3758/BF03214335 [DOI] [Google Scholar]
  63. Rachlin H, Battalio R, Kagel J, & Green L (1981). Maximization theory in behavioral psychology. Behavioral and Brain Sciences, 4(3), 371–388. 10.1017/S0140525X00009407 [DOI] [Google Scholar]
  64. Reed DD, Kaplan BA, Becirevic A, Roma PG, & Hursh SR (2016). Toward quantifying the abuse liability of ultraviolet tanning: A behavioral economic approach to tanning addiction. Journal of the Experimental Analysis of Behavior, 106(1), 93–106. 10.1002/jeab.216 [DOI] [PubMed] [Google Scholar]
  65. Salzer AR, Gelino BW, & Reed DD (2019). Left-digit pricing effects in a high-resolution examination of hypothetical operant demand for alcohol. Behavioural Processes. Advance online publication. 10.1016/j.beproc.2019.05.020 [DOI] [PubMed] [Google Scholar]
  66. Saunders JB, Hao W, Long J, King DL, Mann K, Fauth-Bühler M, Rumpf HJ, Bowden-Jones H, Rahimi-Movaghar A, Chung T, Chan E, Bahar N, Achab S, Lee HK, Potenza M, Petry N, Spritzer D, Ambekar A, Derevensky J, … Poznyak V (2017). Gaming disorder: Its delineation as an important condition for diagnosis, management, and prevention. Journal of Behavioral Addictions, 6(3), 271–279. 10.1556/2006.6.2017.039 [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. Saville BK, Gisbert A, Kopp J, & Telesco C (2010). Internet addiction and delay discounting in college students. The Psychological Record, 60, 273–286. http://opensiuc.lib.siu.edu/tpr/vol60/iss2/5/ [Google Scholar]
  68. Snider SE, Mellis AM, Poe LM, Kocher MA, Turner JK, & Bickel WK (2020). Reinforcer pathology: Narrative of hurricane-associated loss increases delay discounting, demand, and consumption of highly palatable snacks in the obese. Psychology of Addictive Behaviors, 34(1), 136–146. 10.1037/adb0000498 [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Stein JS, Koffarnus MN, Snider SE, Quisenberry AJ, & Bickel WK (2015). Identification and management of nonsystematic purchase task data: Toward best practice. Experimental and Clinical Psychopharmacology, 23(5), 377–386. 10.1037/pha0000020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Sze YY, Stein JS, Bickel WK, Paluch RA, & Epstein LH (2017). Bleak present, bright future: Online episodic future thinking, scarcity, delay discounting, and food demand. Clinical Psychological Science, 5(4), 683–697. 10.1177/2167702617696511 [DOI] [PMC free article] [PubMed] [Google Scholar]
  71. Tabachnick BG, & Fidell LS (2013). Using multivariate statistics (6th ed.). Pearson. [Google Scholar]
  72. Tang Z, Zhang H, Yan A, & Qu C (2017). Time is money: The decision making of smartphone high users in gain and loss intertemporal choice. Frontiers in Psychology, 8, Article 363. 10.3389/fpsyg.2017.00363 [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Tham SM, Ellithorpe ME, & Meshi D (2020). Real-world social support but not in-game social support is related to reduced depression and anxiety associated with problematic gaming. Addictive Behaviors, 106-(January), 106377. 10.1016/j.addbeh.2020.106377 [DOI] [PubMed] [Google Scholar]
  74. The World Bank. (2016). World Development Report 2016: How the internet promotes development. https://www.worldbank.org/en/publication/wdr2016
  75. Tian M, Tao R, Zheng Y, Zhang H, Yang G, Li Q, & Liu X (2018). Internet gaming disorder in adolescents is linked to delay discounting but not probability discounting. Computers in Human Behavior, 80, 59–66. 10.1016/j.chb.2017.10.018 [DOI] [Google Scholar]
  76. Twenge JM (2019). The sad state of happiness in the United States and the role of digital media. World Happiness Report, 86–95. https://worldhappiness.report/ed/2019 [Google Scholar]
  77. Vuchinich RE, & Tucker JA (1988). Contributions from behavioral theories of choice to an analysis of alcohol abuse. Journal of Abnormal Psychology, 97(2), 181–195. 10.1037/0021-843X.97.2.181 [DOI] [PubMed] [Google Scholar]
  78. Vuchinich RE, & Heather N (Eds.). (2003). Choice, behavioural economics and addiction. Pergamon/Elsevier Science. 10.1016/B978-0-08-044056-9.X5040-4 [DOI] [Google Scholar]
  79. Weinstock J, Mulhauser K, Oremus EG, & D’Agostino AR (2016). Demand for gambling: Development and assessment of a gambling purchase task. International Gambling Studies, 16(2), 316–327. 10.1080/14459795.2016.1182570 [DOI] [Google Scholar]
  80. West DM (2015). Digital divide: Improving Internet access in the developing world through affordable services and diverse content [executive summery]. Center for Technology Innovation at Brookings. [Google Scholar]
  81. Young KS, Pistner M, O’Mara J, & Buchanan J (1999). Cyber-disorders: The mental health concern for the new millennium. CyberPsychology & Behavior, 2(5), 475–479. 10.1089/cpb.1999.2.475 [DOI] [PubMed] [Google Scholar]
  82. Zvorsky I, Nighbor TD, Kurti AN, DeSarno M, Naudé G, Reed DD, & Higgins ST (2019). Sensitivity of hypothetical purchase task indices when studying substance use: A systematic literature review. Preventive Medicine: An International Journal Devoted to Practice and Theory, 128, 105789. 10.1016/j.ypmed.2019.105789 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Materials
NIHMS1705724-supplement-Supplemental_Materials.docx (42.5KB, docx)

RESOURCES