Robustness of Decision-Making Competence: Evidence from two measures and an 11-year longitudinal study

Abstract

Decision-making competence is the ability to follow normative principles when making decisions. In a longitudinal analysis, we examine the robustness of decision-making competence over time, as measured by two batteries of paper-and-pencil tasks. Participants completed the youth version (Y-DMC) at age 19 and/or the adult version (A-DMC) eleven years later at age 30, as part of a larger longitudinal study. Both measures are comprised of tasks adapted from ones used in experimental studies of decision-making skills. Results supported the robustness of these measures and the usefulness of the construct. Response patterns for Y-DMC were similar to those observed with a smaller initial sample drawn from the same population. Response patterns for A-DMC were similar to those observed with an earlier community sample. Y-DMC and A-DMC were significantly correlated, for participants who completed both measures, 11 years apart, even after controlling for measures of cognitive ability. Nomological validity was observed in correlations of scores on both tests with measures of cognitive ability, cognitive style, and environmental factors with predicted relationships to decision-making competence, including household SES, neighborhood disadvantage, and paternal substance abuse. Higher Y-DMC and A-DMC scores were also associated with lower rates of potentially risky and antisocial behaviors, including adolescent delinquency, cannabis use, and early sexual behavior. Thus, the Y-DMC and A-DMC measures appear to capture a relatively stable, measurable construct that increases with supportive environmental factors and is associated with constructive behaviors.

Day-to-day, we make decisions that range from the relatively trivial to ones with major effects on our lives. Traditionally, normative theories of decision making have posited how people should make decisions that maximize the expected utility of their outcomes (e.g., Bernoulli, 1954; von Neumann & Morgenstern, 1953; Simon, 1978). Descriptive research in behavioral decision making has long focused on identifying when and why individuals systematically deviate from the principles posited by traditional normative theories, and on the efficiency of the heuristic strategies that sometimes replace them (e.g., Einhorn & Hogarth, 1981; Hastie & Dawes, 2010; Kahneman, Slovic, & Tversky, 1982). For example, people may violate the normative ‘sunk cost’ principle by continuing to invest in unprofitable options, especially when they have made large unrecoverable investments (Arkes & Blumer, 1985). People may also violate the traditional normative principle of ‘description invariance,’ as when they rate a team as more successful when described in terms of its 60% success rate rather than its 40% failure rate (Dunegan, 1993), perhaps reflecting conversational norms that prescribe using success rates to describe successful teams and failure rates to describe failing ones (Sher & McKenzie, 2006).

As a result of its focus on understanding how individual violations occur, the field long left open the question of whether there are stable individual differences in the ability to avoid such violations, across time and contexts. Such decision-making competence (DMC) would be similar to, but distinct from, other cognitive abilities such as IQ or executive cognitive function (e.g., Stanovich & West, 1998, 2008; West, Toplak, & Stanovich, 2008).

Suggestively, studies that used multiple items to measure adherence to a given normative principle have found relative internal consistency for participants’ ability to resist violations of the sunk cost principle (Stanovich, 1999), resist framing effects that violate description invariance (Levin, Gaeth, Schreiber, & Lauriola, 2002), apply decision rules consistently (Bröder, 2000), and express appropriate confidence in their knowledge (Bornstein & Zickafoose, 1999; Klayman, Soll, González-Vallejo, & Barlas, 1999; Stankov & Crawford, 1996, 1997; West & Stanovich, 1997; Wolfe & Grosch, 1990). As with most experimental research, these studies report little else about participants’ traits, behaviors, and life circumstances. The one exception is that several studies that have also measured cognitive ability, typically finding it to be positively correlated with performance on decision-making tasks, such as ones testing resistance to sunk costs (Larrick, Nisbett, & Morgan, 1993), resistance to framing (Smith & Levin, 1996), and under/overconfidence (Stanovich & West, 2000; Wolfe & Grosch, 1990).

Twenty years ago, we received an unusual opportunity to examine the validity of tasks used in behavioral decision research. We administered a battery of seven such tasks to 110 males, at the age-19 assessment of a longitudinal study conducted by the Center for Education and Drug Abuse Research (CEDAR; Tarter & Vanyukov, 2001). As described below, CEDAR provided a wealth of other measures, including ones with predicted relationships with decision-making competence. Results from our initial study suggested individual differences in performance that are correlated across behavioral decision-making tasks, as reflected in the primary factor of an exploratory factor analysis across task scores (Parker & Fischhoff, 2005). These scores (called Y-DMC, for Youth Decision-Making Competence) were related to other theoretically postulated measures, such as maladaptive risk-taking behavior and supportive home environment, even after controlling for two measures of cognitive ability (Parker & Fischhoff, 2005).

In order to expand that initial study, we developed an adult version of Y-DMC, called A-DMC, with items suited to a general adult population, and improving some elements of Y-DMC (Bruine de Bruin, Parker, & Fischhoff, 2007; Parker, Bruine de Bruin, & Fischhoff, 2015). Administration of A-DMC to a diverse community sample of adults revealed patterns similar to those in the initial Y-DMC study. Performance on the tasks was positively correlated, yielding a common factor in an exploratory factor analysis (Bruine de Bruin et al., 2007). Moreover, the overall A-DMC score predicted life experiences, as reported on a Decision Outcome Inventory (DOI) developed in the study, even after controlling for measures of cognitive ability (Bruine de Bruin et al., 2007). Thus, Y-DMC and A-DMC are measures of decision-making competence that appear to demonstrate three key aspects of construct validity (Cronbach & Meehl, 1955; Messick, 1989; Nunnally & Bernstein, 1994). Namely, the tasks (a) span a theoretical domain, focusing on skills identified in behavioral decision research as ones that would indicate decision quality; (b) have structural validity, revealed in the correlations between performance on the constituent tasks; and (c) reveal convergent, discriminant, and predictive validity.

In the ensuing period, CEDAR continued to administer Y-DMC to participants, as part of its age-19 assessment, adding 416 individuals to our original sample of 110. It also administered A-DMC to 214 individuals completing their final CEDAR assessment at age 30, including 146 who had completed Y-DMC eleven years earlier. The next section summarizes the original Y-DMC and A-DMC studies, followed by the new data and analyses.

Original studies

The Center for Education and Drug Abuse Research (CEDAR; Tarter & Vanyukov, 2001) enrolled high-risk youth (defined as having fathers who met clinical diagnostic criteria for substance use disorder) and low-risk youth (defined as having fathers with no diagnosed substance use disorder), at age 9–13, with the aim of following them until age 30. The initial CEDAR sample recruited only males, adding females in subsequent years. Participants underwent periodic intensive assessments. At the age-19 assessment, 110 of the initial male CEDAR sample completed a battery of seven tasks adapted from the behavioral decision research literature, chosen to capture key decision-making skills, and formulated to reduce shared method variance (Table 1).¹

Table 1.

Decision-making competence tasks used for Y-DMC and A-DMC

Task	Performance Criterion
Resistance to Framing	Consistency in choice across equivalent, positively- and negatively-worded questions
Resistance to Sunk Cost	Considering only future consequences when making choices
Applying Decision Rules	Using specified decision rules in choosing among multi-attribute options
Under/overconfidence	Correspondence between confidence and knowledge
Consistency in Risk Perception	Consistency between risk judgments and probability theory
Recognizing Social Norms	Correlation between judged and measured social norms

As reported more fully in Parker & Fischhoff (2005), performance was correlated across tasks, revealed a strong first factor in an exploratory factor analysis, and showed good test-retest reliability on a subsample of participants who completed the materials twice. Scores on the common factor, called Y-DMC, were positively correlated with measures of both fluid and crystallized intelligence (Cattell, 1987; Horn, 1985). They were also correlated with other measures relevant to the decision-making competence construct, even after controlling for cognitive ability, suggesting a distinct construct. More specifically, semi-partial correlations that controlled for the two measures of cognitive ability found that individuals with higher Y-DMC scores also reported less polarized thinking, more constructive behavioral coping strategies (Epstein & Meier, 1989; Katz & Epstein, 1991), and greater self-monitoring (suggesting greater cognitive flexibility; Graziano et al., 1987; Snyder, 1974; Snyder & Cantor, 1980). Semi-partial correlations with the same controls found that individuals with higher Y-DMC scores reported fewer risky and antisocial behaviors² suggestive of poor decision making, such as externalizing and delinquent behavior, cannabis use, and early sexual activity. Finally, semi-partial correlations found that individuals with higher Y-DMC scores had better family and social environments, in ways that should promote the development of decision-making skills.

As reported more fully in Bruine de Bruin, Parker, and Fischhoff (2007), the Adult Decision-Making Competence (A-DMC) measure adapted the Y-DMC tasks to be more suitable for adults and their life experiences. As with Y-DMC, A-DMC task scores were significantly correlated with one another, and their composite score was positively correlated with a measure of fluid intelligence. Holding age, SES, and cognitive ability constant, individuals with higher A-DMC scores reported fewer negative life outcomes (e.g., foreclosure, getting kicked out of an apartment) on the Decision Outcome Inventory (see also Parker et al., 2015).

Subsequent research has found further support for the validity of the Y-DMC and A-DMC measures. Higher Y-DMC scores have been associated with greater ability to make choices consistent with maximizing expected value (Parker & Weller, 2015) and with fewer emotional, behavioral, and peer-related difficulties, reported two years after assessment (Weller, Moholy, Bossard, & Levin, 2014). Studies have also found better performance on A-DMC tasks to be associated with higher scores on tests of cognitive abilities (e.g., Del Missier et al., 2013, 2015; Frederick, 2005) and self-reports of theoretically productive decision-making styles (e.g., Carnevale, Inbar & Lerner, 2012; Dewberry, Juanchich & Narendran, 2013; Parker, Bruine de Bruin & Fischhoff, 2007). One study found that individuals with higher A-DMC scores reported seeing less expected value in various risk behaviors and engaging in fewer of them (Weller, Ceschi, & Randolph, 2015). A-DMC has been used to assess the decision-making competence of US policy leaders (Carnevale et al., 2012), Swedish adults with ADHD (Mäntylä, Still, Gullberg & DelMissier, 2012), US psychiatric patients at risk for suicide (Szanto et al., 2015), Slovak students (Bavolar, 2013), and US students in high school history classes with and without a focus on decision making (Jacobson et al., 2012).

The current study

The new CEDAR data allow further analyses of the psychometric properties of the Y-DMC and A-DMC measures used in the original studies, and of the decision-making competence construct that they assess (Bruine de Bruin et al., 2007; Parker & Fischhoff, 2005). These new data also provide a first examination of how stable the construct is over time, for participants who completed the closely related measures at the age-19 and age-30 assessments. Several studies have used cross-sectional designs to assess decision-making competence in individuals at various life stages (e.g., Bruine de Bruine, Parker, & Fischhoff, 2012; Finucane & Gullion, 2010; Weller, Levin, Rose, & Bossard, 2012). However, such designs confound age groups and cohorts. CEDAR’s longitudinal design allowed us to follow individuals as they aged through a formative developmental period.

Thus, we had three main research aims:

• Aim 1: Assess the internal validity of Y-DMC and A-DMC, as revealed in the relationships among scores on their component tasks.

• Aim 2: Assess the nomological validity of Y-DMC and A-DMC, in terms of whether they correlate in expected ways with constructs reflecting participants’ (a) general cognitive abilities, (b) social environments, (c) cognitive styles, and (d) risk behaviors.

• Aim 3: Assess the stability of decision-making competence over time and across assessment tools, in terms of the relationship between age-19 Y-DMC and age-30 A-DMC, controlling for cognitive ability.

Aims 1 and 2 expand results from prior work (Bruine de Bruin et al., 2007; Parker & Fischhoff, 2005) to new samples, providing context for our novel longitudinal analyses. Aim 3 examines the stability of relationships across an 11-year age span, and across two related instruments.

Method

Longitudinal data collection

The Center for Education and Drug Abuse Research (CEDAR) conducted a longitudinal study examining the etiology of substance abuse (Tarter & Vanyukov, 2001). Beginning in 1989, it recruited 9–13 year-olds (M = 11.4, SD = 0.9) at high or low risk for developing substance-use disorders, defined by whether a participant’s father had a history of substance-use disorders. High-risk youth were identified through clinical settings, advertising, and a marketing firm; low-risk youth were drawn through random-digit dialing and advertising. The initial baseline assessment included the child and the child’s parents, and covered domains such as health behavior, psychopathology, cognitive functioning, and family and social environment. Subsequent assessments were conducted every 2–3 years through age 30.³ In 1998, CEDAR added Y-DMC to the age-19 assessment; in 2007, CEDAR added A-DMC to the age-30 assessment. The National Institute of Drug Abuse discontinued funding CEDAR in 2015. As a result, although most CEDAR participants completed Y-DMC, not all completed (or will ever complete) A-DMC.

Y-DMC and A-DMC samples

Our study involved the three CEDAR samples (Table 2): First, the Y-DMC initial sample (N=108)⁴ included male youth who completed Y-DMC at age 19, as reported in Parker and Fischhoff (2005). Second, the Y-DMC expansion sample (N=416) included male and female youth who completed the age-19 Y-DMC assessment at later times, whose data have not been published previously. Third, the A-DMC sample (N=214) included participants who completed A-DMC at age 30; among them, 146 (73.4%) had also completed Y-DMC at age 19 (the others had completed the age-19 assessment before the initial Y-DMC data collection began). Lngitudinal analyses reported here involve those 146 participants. A-DMC scores were similar for participants with and without Y-DMC scores, t(212)=.91, ns.; Y-DMC scores were similar for participants with and without A-DMC score, t(522)=−1.51, ns.

Table 2.

Characteristics of the Y-DMC and A-DMC samples.

Sample Characteristic	Y-DMC Initial Sample (Age 19)	Y-DMC Expansion Sample (Age 19)	A-DMC Sample (Age 30)
Sample size	108	416	214
% Male	100.0%	56.0%	88.8%
Age
Age-19 assessment	M = 18.6(SD = 0.6)	18.9(SD = 0.4)	M = 18.7(SD = 0.6)
Age-30 assessment	M = 30.1(SD = 0.4)	M = 30.0(SD = 0.3)	M = 30.1(SD = 0.4)
% with high-school diploma or GED:
Age-19 assessment	56.1%	81.6%	61.0%
Age-30 assessment	95.8%	96.6%	96.2%
% with bachelor’s degree or greater, age-30 assessment	47.2%	48.7%	47.4%
% Caucasian	86.1%	71.1%	79.1%
% in High-risk Group	36.1%	45.2%	32.7%

Note: Gender and race information is missing for eight Y-DMC respondents and three A-DMC respondents. Education information is missing for one Y-DMC and A-DMC respondent.

The Y-DMC expansion sample included female participants (44%), who had not been recruited in CEDAR’s early stages. The Y-DMC expansion sample was somewhat less Caucasian (71.1%) than the Y-DMC initial sample (86.1%), χ²(1)=7.75, p<.05, r=.12, slightly older at the age-19 assessment (mean 18.9 vs. 18.6 years), t(513)=7.13, p<.001, d=.77, more likely to have graduated high school or received a GED by the age-19 assessment (81.6% vs. 56.1%, partly reflecting the older mean age), χ²(1)=30.4, p<.001, r=.24, but no more likely to have graduated high school or received a GED by the age-30 assessment (96.6% vs. 95.8%), χ²(1)=.07, ns., r=.01. There were no differences between the two samples in their age at the age-30 assessment, likelihood of having a bachelor’s degree by age 30, or membership in the low-risk vs. high-risk group for developing substance use disorders (p>.05).

In the analyses that follow, we focus on the new data, in the Y-DMC expansion sample and the age-30 sample, considering the Y-DMC initial and combined samples (including both initial and expansion samples) for context when helpful.

Both Y-DMC and A-DMC included six tasks assessing the ability to follow normative principles of decision making (Bruine de Bruin et al., 2007; Parker & Fischhoff, 2005). They are publicly available for download from the Society for Judgment and Decision Making (http://www.sjdm.org/dmidi/Youth_-_Decision_Making_Competence.html; http://www.sjdm.org/dmidi/Adult_-_Decision_Making_Competence.html). Below, we briefly describe each task and its scoring. Further detail can be found in the original papers (Bruine de Bruin et al., 2007; Parker & Fischhoff, 2005) and online supplementary material.

Resistance to Framing measured whether participants valued options similarly despite normatively irrelevant variations in their descriptions. As such, items were paired, with each representing an equivalent choice presented differently. Participants received both items in each pair, separated by other tasks. Y-DMC included five pairs of binary-choice items, with the overall Resistance to Framing score reflecting the number of consistent choices (0–5). Two of the five item pairs represent valence-framing problems, where the same outcomes or attributes are framed positively or negatively (Levin, Schneider, & Gaeth, 1998). The other three represent non-valence frames. Cronbach’s alpha is low in the Y-DMC expansion sample (α=.19), as it was in the Y-DMC initial sample (α=.30; Parker & Fischhoff, 2005).

Suspecting that the diverse forms of framing in the Y-DMC set were sampling more than one domain, we designed A-DMC to use only valence framing problems.⁵ We also increased the number of pairs from five (Y-DMC) to fourteen (A-DMC) and changed from a discrete choice response mode to a six-point rating scale, hoping to discriminate better among preferences. That scale was anchored at 1=definitely would choose A and 6=definitely would choose B. The Resistance to Framing score was one minus the mean absolute difference in ratings on the fourteen question pairs. Cronbach’s alpha for A-DMC was much higher (α=.70) than for Y-DMC, presumably reflecting these changes and similar to results with the A-DMC community sample (α=.62; Bruine de Bruin et al., 2007).

Resistance to Sunk Cost measured how well participants follow the normative prescription to ignore unrecoverable past expenditures when making decisions (Arkes & Blumer, 1985). Y-DMC was scored as the number of choices, across two items, that resisted sunk cost. Cronbach’s alpha was very low in the Y-DMC expansion sample (α=.13), as it had been in the Y-DMC initial sample (α=.03; Parker & Fischhoff, 2005). A-DMC expanded this scale, using ten items instead of two. Additionally, the latter scale implemented a 6-point rating scale to assess how likely respondents were to choose the sunk-cost (versus normatively correct) option. Presumably reflecting those changes, Cronbach’s alpha improved but remained lower than typically sought for scales (α=.55), replicating previous findings (α=.54 in the A-DMC community sample; Bruine de Bruin et al., 2007).

Applying Decision Rules asked participants to indicate which option to select from a choice set, using decision rules from Payne, Bettman, and Johnson (1993). For example, one A-DMC item states that, “Lisa wants the Blu-ray player with the highest average rating across features,” and provides ratings of each option on features such as picture quality, sound quality, and programming options. A-DMC differed from Y-DMC by having more items (ten vs. seven), more options per item (five vs. three), an updated cover story (Blu-ray players vs. Walkmen), and numeric rather than graphical rating displays. The overall score reflected the percent of items for which the choice followed the decision rule. Cronbach’s alpha was .58 for the Y-DMC expansion sample (α=.68 for the Y-DMC initial sample; Parker & Fischhoff, 2005) and .67 for the A-DMC sample (α=.73 for the A-DMC community sample; Bruine de Bruin et al., 2007).

Under/overconfidence assessed how well participants assess the extent of their own knowledge. For each of 42 statements for Y-DMC and 34 for A-DMC, participants indicated whether it was true or false, and then assessed their confidence in their answer, on a scale anchored at 50% (just guessing) and 100% (absolutely sure). Y-DMC items covered topics relevant to adolescents, including (a) general knowledge (e.g., history, geography), (b) the effects of alcohol and drugs, and (c) potential consequences of risky sexual behavior and HIV/AIDS transmission. A-DMC items included topics taken from advice books targeting adults’ life decisions (e.g., finances, health). The overall Under/overconfidence score equaled one minus the absolute difference between mean confidence and percentage correct, so that higher scores reflected better performance. Cronbach’s alpha was .76 for the Y-DMC expansion sample (α=.79 for the Y-DMC initial sample; Parker & Fischhoff, 2005) and .80 for the A-DMC sample (α=.77 for the A-DMC community sample; Bruine de Bruin et al., 2007).

Consistency in Risk Perception measured participants’ ability to follow the normative rules of probability theory when assessing risks. Each item asked participants to judge the probability of an event occurring on a linear scale ranging from 0% (=no chance) to 100% (=certainty). Item pairs included nested subset and superset events, complementary events, conjunctions of two events, and disjunctions of two events. A-DMC had more items than Y-DMC (twenty vs. five). Scores reflected the percentage of item pairs assigned consistent probabilities. Cronbach’s alpha was.45 for the Y-DMC expansion sample (α=.50 for the Y-DMC initial sample; Parker & Fischhoff, 2005) and .65 for the A-DMC sample (α=.72 for the A-DMC community sample; Bruine de Bruin et al., 2007).

Recognizing Social Norms measured how well participants assess peer social norms, following previous measures (Jacobs, Greenwald, & Osgood, 1995; Loeber, 1989), and is the same for Y-DMC and A-DMC. Participants indicated how many “out of 100 people your age” would agree that “it is sometimes OK” to engage in each of 16 undesirable behaviors (e.g., to steal under certain circumstances). After completing other tasks, participants indicated whether they thought that “it is sometimes OK” to engage in each behavior. The score for each participant was the rank-order correlation (from −1 to 1) between the estimated and actual percentage of endorsers in the sample, as computed across the 16 behaviors. Cronbach’s alpha was .68 for the Y-DMC expansion sample (α=.40 for the Y-DMC initial sample; not previously reported) and .57 for the A-DMC sample (α=.64 for the A-DMC community sample; Bruine de Bruin et al., 2007).

Overall score

For both Y-DMC and A-DMC, the overall score reflected the mean of the six task scores, after standardizing each with z transformations. This algorithm was used by Bruine de Bruin et al. (2007), but differed from Parker & Fischhoff (2005), who extracted a factor score from a principal components analysis (i.e., creating a weighted average). Y-DMC scores were largely unaffected by this change in computation, with the two calculations correlated at r = .93 (p<.001) in the Y-DMC combined sample. We used the procedure proposed by Nunnally and Berstein (1994, pp. 266–274) to estimate the internal consistency of composite scores, which recognizes that linear combinations of loosely-correlated measures can have greater reliability than the original measures (much as a summative scale is more reliable with more items). This estimate was computed as an average of the task alphas weighted by factor scores from a principal components analysis. For the DMC composite measure, this estimate of alpha was .77 for the Y-DMC expansion sample (vs. α=.76 in the Y-DMC initial sample; Parker & Fischhoff, 2005), with a mean inter-item correlation of .13, and .81 for the A-DMC sample (vs. α=.83 in the A-DMC community sample; Bruine de Bruin et al., 2007), with a mean inter-item correlation of .15.

Covariates

We drew on the large CEDAR database to establish the nomological validity of the Y-DMC and A-DMC assessments. We report all covariates reported in Parker and Fischhoff (2005), along with several substitutions chosen to improve comparisons, as noted below. The ages at which each covariate was assessed appear as footnotes to Table 4.

Table 4.

Correlations of Y-DMC and A-DMC with covariate measures.

	Y-DMC, pairwised			Y-DMC, controlling for cognitive ability	A-DMCe	A-DMC, controlling for cognitive ability
	Initial	Expansion	Combined	Combined
Sample Size	108	416	524	516	214	211
Cognitive Ability
ECFa	.39***	.42***	.43***	n/a	.30***	n/a
Vocabulary (WISC-R)a	.46***	.46***	.47***	n/a	.46***	n/a
Social and Family Influences
Paternal substance use (yes=1; no=0)a	−.37***	−.14**	−.19***	−.08*	−.20**	−.05
Household Socio-Economic Statusa	.35***	.29***	.30***	.09*	.31***	.05
Neighborhood disadvantagea	−.32***	−.43***	−.42***	−.21***	−.42***	−.17**
Social supportc	.31**	.15**	.18***	.07	.30***	.12*
Positive peer environmentc	.30**	.13*	.16***	.05	.19*	.08
Cognitive Style
Polarized thinkingc	−.30**	−.30***	−.28***	−.09*	−.28***	−.12*
Self-consciousnessb	.25*	.09+	.12**	.10*	.05	.07
Self-monitoringc	.20+	.22*	.19**	.21***	−.03	−.01
Behavioral copingc	.25*	.12*	.14**	.01	.26***	.09
Risk Behavior
Childhood antisocial disorders c	−.23*	−.22***	−.20**	−.08*	−.05	.03
Externalizing behaviorb	−.29**	−.21***	−.23***	−.09*	−.19*	−.11
Delinquencyb	−.24*	−.34***	−.33***	−.14**	−.20**	−.05
Ln(# of alcohol uses, typical month)
Age-16 assessment	−.11	−.11*	−.10*	−.07	.02	.01
Age-19 assessment	−.16	−.03	−.05	−.03	−.16*	−.11
Age-22 assessment	−.01	.13*	.10*	.06	.08	.05
Ln(# of cannabis uses, typical month)
Age-16 assessment	−.21*	−.19***	−.18***	−.12**	.00	.00
Age-19 assessment	−.11	−.24***	−.21***	−.12**	−.18*	−.08
Age-22 assessment	−.15	−.16**	−.15**	−.07	−.25**	−.13
Ln(# times had sex, last year)c	−.19+	−.28***	−.26***	−.14***	−.24***	−.09
Ln(# sexual partners)c	−.26*	−.29***	−.29***	−.16***	−.31***	−.15*

Note: Columns 4 and 6 present semipartial correlations, controlling for cognitive ability. Change in R² is the square of these semipartial coefficients, and ranges from .00 to .05 for Y-DMC and .00 to .03 for A-DMC.

^***p< .001;

^**p < .01;

^*p < .05;

⁺p < .10.

^aMeasured at baseline assessment;

^bMeasured at age-14 assessment;

^cMeasured at age-16 assessment;

^dMeasured at age-19 assessment;

^eMeasured at age-30 assessment.

Cognitive ability

Following Parker & Fischhoff (2005), we used the vocabulary assessment of Wechsler Intelligence Scale for Children - Revised (WISC-R; Wechsler, 1972) as a measure of crystallized cognitive ability. Fluid cognitive ability was represented by executive cognitive functioning (ECF), assessed by combining the block design test from the WISC-R with other measures of inhibition and executive functioning, including the Porteus maze, vigilance, motor restraint, and forbidden toy tasks (see Giancola and Parker, 2001, for details).

Social and family influences

We used five measures related to conditions that should foster the development of decision-making abilities (Parrill-Burnstein, 1978; Baron & Hershey, 1988; Jones et al., 1997): (a) whether the participant’s father had a history of substance use disorders; (b) household socio-economic status (Hollingshead, 1975); (c) neighborhood disadvantage, defined by Census estimates of the percentage of households below the poverty level, the percentage of households headed by females, the percentage of adults over age 24 without college degrees, and the percentage of households not occupied by owner (following Ridenour et al., 2009); (d) social support, as assessed by the Interpersonal Support Evaluation List (ISEL) (Cohen & Hoberman, 1983); (e) peer environment, including peers’ problem behavior (e.g., drug or alcohol use) and constructive behavior (e.g., participation in school clubs and athletics; Tarter, 1991).

Cognitive style

As in Parker & Fischhoff (2005), we considered four measures of self-reported cognitive style. Two are from the Constructive Thinking Inventory (Epstein & Meier, 1989; Katz & Epstein, 1991): Polarized Thinking, which assesses difficulty with seeing nuances in situations (e.g., “I tend to classify people as either for me or against me”), and Behavioral Coping, which captures behavioral strategies for coping with adversity (e.g., “I am the kind of person who takes action rather than just thinks or complains about a situation”). Both assessed the kind of open-minded, flexible thinking that should accompany decision-making competence (Baron, 1988; Stanovich & West, 1998, 2000). The other two were the Self-consciousness Scale, which assesses introspection and concern for how others view one, hence should reflect attention to social norms (e.g., “I care a lot about how I present myself to others,” Fenigstein et al., 1975), and the Self-monitoring Scale, which asked about the tendency to monitor and critique one’s own behavior in social situations (e.g., “There are many things I would only tell to a few of my friends,” Graziano et al., 1987; Snyder, 1974; Snyder & Cantor, 1980).

Risk behavior

We focused on measures of risk and antisocial behavior that can incur significant physical, emotional, and societal costs. Psychiatric diagnoses that reflect disruptive and antisocial behaviors included Oppositional Defiant Disorder and Conduct Disorder up through age 16 (using a diagnostic instrument derived from Endicott & Spitzer, 1978; Spitzer, Williams, & Gibbon, 1987). Because respondents who miss a visit could return for future visits and the psychiatric interviews retrospectively covered the intervening time, we included all available data. Externalizing behavior (e.g., fighting, teasing) and delinquency (e.g., stealing, running away) were reported by the participant’s mother at age 14, using the Child Behavior Checklist (Achenbach & Edelbrock, 1983). The number of times the respondent had had sexual intercourse and number of sexual partners were self-reported for the year prior to the age-16 assessment (using the natural logarithm of overall scores +1, to account for positive skewness). For self-reported alcohol and drug use, we followed CEDAR’s convention of using estimates derived from the Drug Use Screening Inventory (Kirisci, Tarter, & Tse-Chi, 1994; Tarter, 1990). At the age-16, age-19, and age-22 assessments, respondents reported their alcohol and drug use in a typical month, with the response options of 0, 1–2, 3–9, 10–20, and more than 20 times.

Analytic approach

Our analyses followed our three research aims. First, we used confirmatory factor analysis on the new CEDAR data for the Y-DMC expansion sample and A-DMC, to examine whether we could replicate sufficiency of fit for the single-factor model identified in exploratory factor analyses on previous data (Bruine de Bruin et al., 2007; Parker & Fischhoff, 2005). Second, we examined the nomological validity of overall Y-DMC and A-DMC scores in terms of their correlations with the posited covariates, presenting results separately for the Y-DMC initial and expansion samples. Third, we compared Y-DMC and A-DMC scores for the 146 participants who completed both tests, assessing the stability of decision-making competence over time, using zero-order and semi-partial correlations controlling for cognitive ability.

Attrition analysis

The logistical challenges of longitudinal data collection led to varying amounts of missing data across variables. We used all available data, with the resulting sample sizes for nomological validity analyses (i.e., correlations) ranging from 89–108 for the Y-DMC initial sample, 334–416 for the Y-DMC expansion sample, 427–516 for the Y-DMC combined sample, and 176–214 for the A-DMC sample.⁶ Overall, 29.7% of CEDAR participants had at least one missing score. These individuals had slightly lower mean scores for Y-DMC (M=−.08, SD=.54 vs. M=.03, SD=.52, t(522)=2.30, p < .05) and A-DMC (M=−.07, SD=.61 vs. M=.03, SD=.51, t (144)=1.28, ns.).

The online supplementary material include additional task details, descriptive statistics, and correlations among the tasks.

Results

Aim 1: Assess the internal validity of Y-DMC and A-DMC, as revealed in the relationships among scores on their component tasks

Y-DMC (Age 19)

Table 3 presents a confirmatory factor analysis on the Y-DMC expansion sample, which revealed a good fit for a single overall factor. All Y-DMC tasks loaded significantly (p < .01), except for Resistance to Sunk Cost (which also had low internal consistency). The overall fit was similar when Resistance to Sunk Cost was removed from the model (CFI=.95, RMSEA=.05, SRMR, .04, χ²(10)=19.3, p<.05). In order to maintain comparability to past studies, we kept Resistance to Sunk Cost in the composite measure.

Table 3.

Single-factor solutions for confirmatory factor analyses of Y-DMC and A-DMC task scores.

Task	Y-DMC (Expansion Sample Only)	A-DMC
Resistance to Framing	.30** (.07)	.40** (.09)
Resistance to Sunk Cost	−.09 (.07)	.31** (.08)
Applying Decision Rules	.75** (.09)	.84** (.17)
Under/overconfidence	.49** (.09)	.17* (.09)
Consistency in Risk Perception	.36** (.08)	.36** (.11)
Recognizing Social Norms	.40** (.07)	.27** (.09)
CFI	.953	1.00
RMSEA	.046	.000
SRMR	.034	.026
χ2 (df)	16.93* (9)	4.90 (9)

Note. Standardized estimates presented; standard errors for parameter estimates in parentheses.

^**p<.01.

^*p<.05.

A-DMC (age 30)

The confirmatory factor analysis for A-DMC also showed a good fit and yielded results similar to those with Y-DMC. The main exception is that Resistance to Sunk Cost loaded positively and significantly, which may reflect its improved measurement.

Aim 2: Assess the nomological validity of Y-DMC and A-DMC, in terms of whether they correlate in expected ways with other constructs

Y-DMC (Age 19)

Table 4 shows significant zero-order Pearson correlations between overall Y-DMC score and both measures of cognitive ability, for the Y-DMC initial sample (as reported in Parker & Fischhoff, 2005), the Y-DMC expansion sample, and the combined sample. Table 4 shows that participants with higher Y-DMC scores also had more favorable social and family environments, as measured by the absence of paternal substance abuse, higher household socio-economic status, less neighborhood disadvantage, greater social support, and more positive peer environment.⁷ Higher Y-DMC scores were also correlated with self-report measures of cognitive style plausibly related to better decision making. Participants with higher Y-DMC scores showed less of several problematic behavior tendencies, as measured by CEDAR staff (childhood antisocial disorders) and reported by the youth’s mother (externalizing behavior, delinquency). They self-reported more cannabis use, more sex, and more sexual partners. Y-DMC scores showed little relationship to self-reported alcohol use. Generally speaking, these relationships were similar in the two samples. Controlling for the two measures of cognitive ability reduced these correlations, but many of the associations remain significant – most notably neighborhood disadvantage, self-monitoring, and several risk behaviors.

A-DMC (age 30)

The fifth column of Table 4 shows correlations between A-DMC and the CEDAR covariates. Overall, they were very similar to those for Y-DMC, assessed eleven years earlier, with covariates from earlier CEDAR assessments. Perhaps the most notable difference was the weaker correlation with childhood antisocial disorders, assessed at age 16. A-DMC was more strongly correlated with cannabis use self-reports collected closer in time to its assessment. Controlling for the two measures of cognitive ability had a similar effect with A-DMC as with Y-DMC, leaving the signs unchanged, but taking most below the threshold of statistical significance.

Aim 3: Assess the stability of decision-making competence over time and across assessment tools

Overall Y-DMC scores at age 19 were strongly correlated with A-DMC scores at age 30 (r=.50, p<.001), despite the 11-year time difference and the modest differences in the measures (with A-DMC drawing on Y-DMC results to improve its design). Controlling for cognitive ability reduces this correlation somewhat, but it remains strongly significant (r=.33, p<.001). There were also positive correlations for scores on corresponding versions of most of the six individual tasks: Resistance to Framing (r=.17, p<.05), Resistance to Sunk Cost (r=.18, p<.05), Applying Decision Rules (r=.55, p<.001), Under/Overconfidence (r=−.03, ns.), ⁸ Consistency in Risk Perception (r=.07, ns.), and Recognizing Social Norms (r=.29, p<.01). As might be expected for subtasks involving fewer responses, these correlations were smaller than those for the overall decision-making competence score. Excluding the most highly correlated task, Applying Decision Rules, left overall scores that were still well correlated over time (r=.35, p<.001).

Discussion

We evaluated the robustness of decision-making competence using two instruments, Y-DMC, designed for young adults, and A-DMC, designed for the general adult population. We reported new data from the CEDAR longitudinal study, adding 416 Y-DMC participants in its age-19 assessment, to our original 110 (Parker & Fischhoff, 2005), and 214 A-DMC respondents in its age-30 assessment, for comparison with our original community sample (Bruine de Bruin et al., 2007). These samples included 146 individuals who completed both Y-DMC at age 19 and A-DMC at age 30. Combined with the earlier studies, these new data allowed us to examine the robustness of the construct across samples and time.

Addressing Aim 1, responses of the Y-DMC expansion sample were similar to those of the original Y-DMC sample, in terms of the internal validity of the six tasks and the results of a confirmatory factor analysis. Responses of the CEDAR A-DMC sample were similar to those of the earlier community sample, in the same ways. The main difference between results with the two scales was that Resistance to Sunk Cost had a significant loading on the A-DMC factor, but not on the Y-DMC factor, presumably reflecting its improved measurement in A-DMC (more items, more response options per item).

Addressing Aim 2, patterns of nomological validity in both new samples paralleled those for the original samples. Participants with higher Y-DMC or A-DMC scores had greater cognitive abilities and more supportive family and social environments, which may promote the development of decision-making competence. They displayed more constructive cognitive styles, which may be expected to reflect concurrent skills. Finally, they reported fewer instances of what may be deemed (at least by some) health-risking and antisocial behavior, potentially reflective of poorer decision making. The pattern of these correlations remained, albeit in weakened form, after controlling for the two measures of cognitive ability, with some of the strongest residual correlations being neighborhood disadvantage, self-reported self-monitoring, and several risk behaviors. The correlations emerged despite the covariates being assessed from 3 to 20 years before or after the DMC measures.

Addressing Aim 3, overall Y-DMC scores at age 19 predicted overall A-DMC scores at age 30. This relationship remained after controlling for the two measures of cognitive ability. This stability occurred despite the Y-DMC and A-DMC assessments using somewhat different instruments. A-DMC sought to improve on Y-DMC by increasing the number of items per task and the number of response options per item. Those changes may account for the highest correlations across time being with the two tasks that were perhaps most similar, Applying Decision Rules and Recognizing Social Norms.

Thus, the construct of decision-making competence captured in these two measures appears to be robust over samples and over time, while showing nomological validity in its correlations with other measures that are plausibly related to the ability to follow traditional normative principles. These results strengthen the case for the external validity of such tasks, when used in behavioral experiments. They also suggest the value of using DMC measures in studies of decision making. Two topics for such research that emerge from the present results are how decision-making competence relates to other cognitive abilities and how social environments (family, neighborhood) and decision-making competence affect one another.⁹

Limitations

The longitudinal data collection conducted by CEDAR created a unique scientific resource that allowed us to examine decision-making competence over time and in the context of a rich suite of other relevant variables and outcomes. CEDAR had a remarkable retention rate (73%), considering the 11-year time span and demands on participants. If that attrition were non-random, then it might bias our results in some way. However, we know of no specific concerns – and are somewhat reassured by the similarities in results across samples and for participants with and without missing covariate data.

To avoid capitalization on chance, we identified the correlates that we would consider in advance and report on all variables that we analyzed. We focused on major patterns, rather than small differences in results (especially given the small size of the Y-DMC initial sample). Nonetheless, our aims required many analyses, suggesting caution in interpreting results beyond the general patterns.

Whereas the Y-DMC and A-DMC assessments were designed to assess the same underlying construct, the specifics of their component measures differ sufficiently so that we can only examine the stability of performance in relative terms (via correlations) and not its absolute level (i.e., are people better decision makers at 30 than at 19). A-DMC is a superior instrument overall, judging by its psychometric properties (although Cronbach’s alpha for some tasks remains low). However, its items are designed for adults, whereas Y-DMC is designed for late adolescents. Both have content best suited for North American, English-speaking audiences, which has prompted translations (Bavolar, 2013; Del Missier et al., 2012, 2013, 2015; Mäntylä et al., 2012), whose feasibility is supported by the construct validity of these two measures.

The present analyses consider a unidimensional characterization of decision-making competence, as supported by past research and evaluated with CFA, but do not preclude the potential for multiple dimensions. The original Y-DMC and A-DMC studies reported exploratory factor-analytic solutions that extracted more than one factor, but these factors lacked clear interpretability, due to both the limited number of tasks and cross-loadings across factors. Examining the dimensionality of decision-making competence is an important challenge for future research, using a larger set of tasks tapping different skills. That research might also use Item Response Theory methods at the individual DMC scale level, in order to refine the measurement of component tasks. These methods, given an appropriate item pool, have the potential to create more efficient tests, both in terms of the range of ability levels measured and the length of the test.

As mentioned, the performance standard for both assessments is adherence to traditional normative theories of decision making, for maximizing expected utility. These tasks do not address the question of individual differences in judicious use of heuristic processes, another topic for future research.

Implications and Recommendations

We interpret these results as evidence for the robustness of the decision-making competence construct and its component tasks, as captured in these two related measures. We believe that one contributor to this robustness is that those tasks build on ones developed in behavioral decision research experiments that have refined their format and presentation, while examining the effects of alternative formulations on behavior. That collective experience helped us to create tasks that would be well understood and to situate each in the skills domain that it was meant to represent. A second plausible contributor is that the tasks assess performance, rather than self-reported behavior (Appelt et al., 2011). Although self-report and performance tasks often demonstrate convergent validity, perceived and actual performance can differ (e.g., Parker et al., 2007; Weller et al., 2012).

Behavioral decision research has traditionally viewed decision making through a situational lens, examining how experimental manipulations evoke general cognitive processes. Our results suggest the feasibility and potential value of studying individual differences and their relationships to real-world antecedents, concomitants, and consequences. Moving forward, as a research community, we face the collective challenge of linking research in the lab and the world, across individuals’ lives.

Biographies

Andrew Parker is a Senior Behavioral and Social Scientist at the RAND Corporation and Director of the RAND Center for Decision Making under Uncertainty. His research applies core concepts in behavioral decision research to the understanding of decision makers’ behavior in real-world situations. His research interests include decision-making competence, aging, risk perception, health decision making, and public health emergency preparedness.

Wändi Bruine de Bruin holds a University Leadership Chair in Behavioural Decision Making at Leeds University’s Business School, where she co-directs the Centre for Decision Research. She is Collaborating Professor at Carnegie Mellon University’s Department of Engineering & Public Policy. Her research focuses on decision-making competence, risk perception and communication.

Baruch Fischhoff is the Howard Heinz University Professor at Carnegie Mellon University’s Department of Engineering & Public Policy and the Institute of Politics and Strategy. He is a member of the National Academy of Sciences and National Academy of Medicine. His research interests include risk management, adolescent decision-making, national security, health, and environment.

Joshua Weller is an Assistant Professor at Tilburg University’s School of Social and Behavioural Sciences, and Senior Research Scientist at Decision Research. His research focuses on how decision-making abilities change from childhood to older age, as well as on how affective and cognitive processes contribute to decision making and risk perceptions.