Identifying the Effects of Unjustified Confidence versus Overconfidence: Lessons Learned from Two Analytic Methods

Abstract

One of the most common findings in behavioral decision research is that people have unrealistic beliefs about how much they know. However, demonstrating that misplaced confidence exists does not necessarily mean that there are costs to it. This paper contrasts two approaches toward answering whether misplaced confidence is good or bad, which we have labeled the overconfidence and unjustified confidence approach. We first consider conceptual and analytic issues distinguishing these approaches. Then, we provide findings from a set of simulations designed to determine when the approaches produce different conclusions across a range of possible confidence-knowledge-outcome relationships. Finally, we illustrate the main findings from the simulations with three empirical examples drawn from our own data. We conclude that the unjustified confidence approach is typically the preferred approach, both because it is appropriate for testing a larger set of psychological mechanisms as well as for methodological reasons.

One of the most common findings in behavioral decision research is that people have unrealistic beliefs about how much they know. Specifically, people tend to overestimate their knowledge and ability in many different domains, with this overestimation increasing with harder tasks and decreasing with easier tasks (to the point of actual underestimation; Alba & Hutchinson, 2000; Keren, 1991; Lichtenstein, Fischhoff, & Phillips, 1982; Yates, 1990). More generally, it is common to observe a lack of correspondence between knowledge and confidence in that knowledge.

In principle, these results seem problematic. Since confidence influences behavior and confidence levels are unwarranted, researchers have argued that this lack of correspondence between confidence and knowledge should reduce decision makers’ ability to appropriately leverage their knowledge (Bruine de Bruine, Parker, & Fischhoff, 2007; Parker & Fischhoff, 2005; Yates, 1990). Hence, much past research has focused on the conditions under which confidence does not correspond to knowledge (e.g., the existence of overconfidence). However, demonstrating that misplaced confidence exists does not necessarily mean that there are costs to it.

Lately, there has been an increase in attention to decision making and decision-making biases as predictors of real-world behavior and outcomes – essentially asking the question of whether biases are necessarily bad. Applied to the current topic, is misplaced confidence in knowledge really a problem? Although intuitively it seems that not having an accurate understanding of your knowledge should be problematic, there are reasons to believe that it may not be, at least in some situations. For example, Von Winterfeldt and Edwards (1973) showed that variations in subjective probabilities for continuous decision alternatives have little impact on decision outcomes. Applied to discrete choices, Smith and Dumont (1997) referred to the concept of an “action threshold” and discussed that it was an open question whether overconfidence would lead to different actions and thus worse consequences. Elsewhere, Price and Stone (2004) showed that exhibiting unwarranted confidence can make one seem more competent, not less, which could have benefits in social situations. More generally, the conditions under which confidence that does not correspond to knowledge actually causes problems is not well understood. Addressing this issue requires treating the level of confidence that does not correspond with knowledge as an independent rather than a dependent variable.

This paper takes this question – is misplaced confidence good or bad? – and then asks how one should answer it. Simply assessing the relationship between confidence and some downstream outcome variable is not sufficient. Since confidence frequently covaries with actual knowledge (e.g., Alba & Hutchinson, 2000), examining this zero-order correlation confounds the effects of confidence and knowledge. Our first instinct (and we suspect the first instinct of many others) was to use the overconfidence score as a predictor of real-world behaviors or outcomes. Overconfidence is usually assessed using a battery of knowledge questions (often with two options, such as true and false), with confidence assessed on each answer (often on a 50% to 100% scale), with the overconfidence score computed as the difference between mean confidence and percent correct.¹

As we will demonstrate, even though this approach is intuitively appealing, the overconfidence statistic has the property of being equally affected by confidence and knowledge. This is appropriate only if the researcher is, in fact, equally interested in confidence and knowledge. However, in many if not most instances this is not the case. Rather, as will be discussed further below, the interest appears to be on the impact of confidence, and more specifically the impact of differences in confidence that do not reflect differences in actual knowledge. It is important to note that the overconfidence score does not reflect this goal well, since individual differences in overconfidence could be entirely driven by individual differences in knowledge. In contrast, examining the effect of confidence when controlling for knowledge, which we’ve labeled the unjustified confidence approach, more clearly examines whether there are costs or benefits to misplaced confidence. This approach is more common in social psychological applications (e.g., Jaccard, Dodge, & Guilamo-Ramos, 2005).

More generally, the appropriateness of the overconfidence and unjustified confidence approaches relies on the proposed mechanism for why the effect of misplaced confidence occurs. Commonly, these mechanisms focus on the effect of confidence. For example, one might expect confidence to affect subsequent information search, independent of actual knowledge. What drives information search in this case is one’s feelings of confidence (i.e., perceived knowledge); indeed, one’s actual knowledge level is unknown to the decision maker. From a researcher’s perspective, actual knowledge is thus relevant only as a variable to be controlled in order to show that any effects are due to confidence rather than to knowledge. Other mechanisms, however, do focus on overconfidence. For example, research on disconfirmation of expectations (e.g., Anderson & Sullivan, 1993; Oliver, 1980) suggests that a person should be satisfied with their performance (say, on a test) only to the extent that it exceeds their expected performance (i.e., their confidence). Thus, changes in the relative difference between confidence and knowledge (rather than in confidence controlling for knowledge) should predict eventual satisfaction. We present below how McGraw, Mellers, and Ritov (2004) propose a mechanism similar to this in their work.

In this paper, we consider the difference between these two ways of capturing how confidence that does not correspond with knowledge is related to some outcome variable. We start with a discussion of the conceptual and analytical differences between the overconfidence and unjustified confidence approaches. We then present key findings from a set of simulations designed to determine when the two approaches produce similar or different conclusions. Finally, we illustrate these findings using three empirical examples from our own research. We conclude with recommendations for when to use each of these approaches in predicting real-world behaviors and outcomes.

Overconfidence versus Unjustified Confidence

This section first presents conceptual definitions and then contrasts overconfidence (OC) and unjustified confidence (UJC) as predictors of an outcome variable,² highlighting three inter-related issues: the role of knowledge, the functional form (and associated measurement scale), and the analytic relationship to the outcome variable.

Definitions

The overarching construct we are trying to assess is confidence that does not correspond to knowledge. The overconfidence measure is often used in behavioral decision research, although it has most commonly been used as a dependent measure. When used as an independent variable in a predictive model, overconfidence represents individual differences in the degree of confidence-knowledge bias across individuals. McGraw, Mellers, and Ritov (2004), for example, used this score as a predictor and found that participants in a basketball-shooting task who were more overconfident experienced less pleasure from their performance on the task. In this case, according to McGraw et al., the difference between expectation (confidence) and performance is what causes displeasure.

In contrast, the unjustified confidence approach to determining the effects of confidence that does not correspond to knowledge is more common in other fields, such as social psychology, consumer research, and economics. Unjustified confidence reflects individual differences in confidence that are not warranted by corresponding individual differences in knowledge. This approach uses the variance in confidence not explained by actual knowledge as the predictor, which can be computed as the residuals after predicting confidence from knowledge. For example, Jaccard et al. (2005) found that perceived birth-control knowledge, after controlling for actual knowledge, predicted later pregnancy rates.

Overconfidence as a predictor

Overconfidence treats knowledge and confidence as equally important and focuses on their relative magnitude, as reflected in a difference score, subtracting knowledge from confidence. Doing so requires measuring knowledge and confidence on equivalent ratio scales. For example, a common procedure is the two-alternative forced choice method, where respondents first indicate whether a statement is true or false and then indicate their confidence in that answer on a probability scale. A sample item from Parker and Fischhoff (2005) is:

Alcohol kills brain cells.
This statement is [True / False].
50% just guessing	60%	70%	80%	90%	100% absolutely sure

Across a set of such items, researchers calculate the percentage correct (knowledge) and the mean confidence judgment (confidence). Because the resulting knowledge and confidence scores share a common ratio scale, one can be meaningfully subtracted from the other. The signed difference between knowledge and confidence has been called overconfidence, calibration-in-the-large, and bias. This measure offers a meaningful zero point (being “appropriately confident” on average), bracketed by over- and underconfidence.

To use OC as a predictor, one computes the zero-order correlation between the OC measure and the outcome variable. Given the way overconfidence is calculated, this zero-order correlation with the outcome variable is thus equally influenced by one’s confidence and one’s knowledge.

Unjustified confidence as a predictor

Unjustified confidence, in contrast, treats knowledge as a variable to be controlled. Because it uses residual variance in confidence, the UJC approach does not require measuring confidence and knowledge on the same scale or measuring confidence on a ratio scale. For example, Radecki and Jaccard (1995) measured knowledge as number of correct answers on an 84-item multiple-choice quiz and confidence as responses to a 3-item scale with questions like “In general, how much to you think you know about the topic of birth control?” (with responses anchored at 1 = not at all knowledgeable and 11 = extremely knowledgeable).

Because UJC is defined as a residual, and is hence a function of not just the individual but the entire sample, it is not particularly useful as an individual score in itself, in the way that positive and negative overconfidence scores have distinct meanings. UJC is, however, useful as a predictor. To use UJC as a predictor, one computes the zero-order correlation between UJC and the outcome variable, or equivalently, the semipartial correlation between confidence and the outcome variable while controlling for knowledge. Thus, with this approach, the resulting (semipartial) correlation indicates the extent to which differences in confidence that do not correspond to differences in knowledge are related to a particular outcome.

Formally Contrasting the Two Approaches

Mathematically, OC is a constrained case of UJC. To see this, consider two regression equations. The first demonstrates the overconfidence approach to predicting an outcome:

$$\text{OUTCOME}={\beta }_0+{\beta }_1(\text{CONFIDENCE}-\text{KNOWLEDGE})+\epsilon .$$ (1)

The second demonstrates the unjustified confidence approach to predicting an outcome:

$$\text{OUTCOME}={\beta }_0+{\beta }_1\text{CONFIDENCE}+{\beta }_2\text{KNOWLEDGE}+\epsilon .$$ (2)

The two analyses are equivalent if and only if β₂ = − β₁ in Equation (2). Thus, the OC correlation is a special case of the UJC semipartial correlation. Table 1 summarizes the key differences between the two research strategies.

Table 1.

Overconfidence versus unjustified confidence.

	Overconfidence	Unjustified Confidence
Definition	Individual differences in the degree of confidence-knowledge bias	Individual differences in confidence that are not warranted by corresponding individual differences in knowledge
Role of knowledge	Of equal conceptual importance to confidence	Treated as a control variable
Functional form	Mean confidence – percent correct	Residual variance in confidence, controlling for knowledge
Measurement scale	Must be equivalent ratio scales	Can be non-equivalent and less than ratio scale
Analytic relationship to outcome	Zero-order correlation between overconfidence and outcome	Semipartial correlation between confidence and outcome, controlling for knowledge

Implications of the Overconfidence—Unjustified Confidence Distinction

As we have asserted above, the key conceptual difference between the approaches entails the role of knowledge. In the OC approach, knowledge is as important as confidence, whereas in the UJC approach knowledge is considered a variable to be controlled. As we will show, the two analytic approaches can produce different results, and sometimes substantially different results. Unfortunately, this distinction has not always been recognized, including by us in our own work.³ A frequent problem is drawing conclusions about confidence when one uses overconfidence as a predictor. In most cases, this occurs when the mechanism under investigation is a confidence mechanism (rather than an overconfidence mechanism), but the authors use an overconfidence approach. For example, in their work on classroom performance, Renner and Renner (2001) discuss the importance of confidence in determining how much one prepares for an exam. In our opinion, this focus on confidence is correct – a student’s decision of how much to study is determined by their confidence level, not their overconfidence level (which they don’t know). Renner and Renner’s analysis, however, focused on overconfidence, showing how students who gave confidence judgments became less overconfident and more accurate (had higher quiz scores) over time. Their conclusion, however, focused on confidence – “a decrease in confidence was accompanied by an increase in accuracy” (p. 31). But note they never tested whether confidence significantly decreased over time, and in one of their two studies, mean confidence actually increased over the course of the term.

Another example is provided by the work of Grimes (2002). This study found that overconfidence in one’s academic performance in an economics course was related to previous experience with the material, and concluded that “awareness of having studied similar course material in the past apparently enhanced student confidence in future performance” (p. 24). Again, however, this study never examined the relationship with confidence; instead, the relationship with overconfidence could be due to either a relationship with confidence or with knowledge.

In the above cases, our interpretation of the studies is that the authors were predominantly interested in testing a confidence mechanism. The authors, very reasonably, were concerned with actual knowledge as a potentially confounding variable, and correctly wanted to control for it; indeed, these studies are more sophisticated and better done than other studies that examine the effect of confidence without controlling for knowledge. Yet, consistent with our initial instinct, in most cases they did this by testing effects of overconfidence. So doing, however, opens up the possibility that any observed effects are due solely to knowledge, not to confidence.

In other cases, authors seem to equate the effects of confidence and overconfidence. Karelaia and Hogarth (2010), for example, defined confidence as estimated score minus actual score, i.e., they are defining confidence as what we are referring to as overconfidence. They proceed to examine the effect of confidence on entry into markets, while controlling for knowledge, which is equivalent to our unjustified confidence analysis. They correctly draw conclusions about confidence, but incorrectly conclude about overconfidence as well, “we found no relation between overconfidence and excess entry.” But they never tested the relationship between overconfidence and excess entry.

In general, then, there seems to be a conflation as to the effects of overconfidence and of confidence controlling for knowledge. Although understandable, this confusion has the potential to lead to conclusions that are not consistent with the analyses reported. To determine whether this is an academic concern that would rarely arise or whether it is a serious issue that arises frequently we next conducted a set of simulations designed to determine how often the two analytic approaches would produce different results.

Simulations

The goal of the simulations was to systematically explore the space of possible relationships among confidence, knowledge, and an outcome, examining the conclusions that would be reached by the OC and UJC approaches. Toward this end, we systematically varied the correlation between (a) confidence and the outcome, (b) knowledge and the outcome, and (c) confidence and knowledge, as well as (d) the relative variance of confidence and knowledge. We then focused on two key indices, corresponding to the two analytic approaches, as follows:

• 1.The zero-order Pearson correlation between overconfidence (equal to confidence minus knowledge) and the outcome variable.
• 2.The semipartial correlation between confidence and the outcome variable, controlling for knowledge.

Specifically, we randomly generated simulated datasets using the Splus statistical programming language. Each dataset consisted of 1,000,000 observations on three normal variables (knowledge, confidence, and an outcome) with user-defined correlation matrices and variances, within three sets of simulations summarized in Table 2. The first set of simulations considers combinations of the knowledge-outcome and confidence-outcome correlations, where confidence and knowledge are uncorrelated and have equal variances. We constrained the correlation between confidence and the outcome to be either positive (r = .5), zero (r = 0), or negative (r = −.5); the correlation between knowledge and the outcome was either zero or positive (r = .5).⁴ The second set of simulations takes the most interesting cases from the first set and examines results for cases where confidence has twice, the same, or half the variance of knowledge, with confidence and knowledge again being uncorrelated. The third set of simulations assumes equal variances for confidence and knowledge (as in Set 1), but correlations between them are either zero or positive (ranging from .3 to .7).⁵ Key results from each of these sets, respectively, are presented in Tables 3 through 5. For each result, we report the relevant parameters, the indices mentioned above, and the following statistics, as they provide additional insight in some situations:

• 3.The semipartial correlation between knowledge and the outcome variable, controlling for confidence, which represents the marginal predictive validity of knowledge after accounting for confidence.
• 4.The multiple R from the regression analysis when the outcome variable is regressed onto knowledge and confidence, which represents the overall predictive validity of knowledge and confidence when used together.

Table 2.

Parameters Varied for Simulation Sets 1, 2, and 3.

Set	Confidence- Outcome Correlation	Knowledge- Outcome Correlation	Ratio of Variances	Knowledge- Confidence Correlation
1	−.5, 0, .5	0, .5	1	0
2	−.5, 0, .5	0, .5	.5, 1, 2	0
3	−.5, 0, .5	0, .5	1	0, .3, .5, .7

Table 3.

Set 1 simulations, holding variances constant with zero confidence-knowledge correlation.

	Parameters		OC Approach Results	UJC Approach Results	Comparative Statistics
Case #	rCONF,OUT	rKNOW,OUT	rOC,OUT	rCONF,OUT KNOW	rKNOW,OUT CONF	Multiple R
1	.50	0	.35	.50	.00	.50
2	−.50	0	−.35	−.50	.00	.50
3	0	.50	−.35	.00	.50	.50
4	.50	.50	.00	.50	.50	.71
5	−.50	.50	−.71	−.50	.50	.71

NOTE: r_X,Y represents the zero-order correlation between X and Y. r_X,Y|Z represents the semipartial correlation between X and Y controlling for Z. The Multiple R is generated from regressing the outcome onto confidence and knowledge.

Table 5.

Select set 3 simulations, allowing confidence and knowledge to covary.

	Parameters			OC Approach Results	UJC Approach Results	Comparative Statistics
Case #	rCONF,KNOW	rCONF,OUT	rKNOW,OUT	rOC,OUT	rCONF,OUT|KNOW	rKNOW,OUT|CONF	Multiple R
1	0	.50	0	.35	.50	.00	.50
	.3	.50	0	.42	.52	−.16	.52
	.5	.50	0	.50	.58	−.29	.58
	.7	.50	0	.65	.70	−.49	.70
4	0	.50	.50	.00	.50	.50	.71
	.3	.50	.50	.00	.37	.37	.62
	.5	.50	.50	.00	.29	.29	.58
	.7	.50	.50	.00	.21	.21	.54
5	0	−.50	.50	−.71	−.50	.50	.71
	.3	−.50	.50	−.85	−.68	.68	.85
	.5*	−.50	.50	−1.00	−.86	.86	1.00
	.7*	−.50	.50	na.	na.	na.	na.

NOTE: r_X,Y represents the zero-order correlation between X and Y. r_X,Y|Z represents the semipartial correlation between X and Y controlling for Z. The Multiple R is generated from regressing the outcome onto confidence and knowledge.

^*Cannot be solved for, as the Choleski decomposition is not full rank. The point at which the computations break down is .5, so the values reported for a knowledge-confidence correlation of .5 are actually computed at r = .499.

The full simulation results are available in the Supplementary Material. For clarity sake, we highlight below the main points drawn from the simulations as “general findings,” presenting only the results relevant to the particular conclusion.

General finding 1: The OC approach conflates the effects of knowledge versus confidence, whereas the UJC approach focuses on confidence

Because the OC approach treats confidence and knowledge equivalently, effects of confidence and effects of knowledge will be indistinguishable using the OC approach. A comparison of the result from Case 2 in Table 3 (where confidence and knowledge are uncorrelated) to that from Case 3 provides perhaps the most striking example of this result. In Case 2, confidence is negatively related and knowledge is unrelated to the outcome, and both the OC approach (r_OC,OUT) and UJC approach (r_{CONF,OUT|KNOW} ) lead to the same basic conclusion. In Case 3, knowledge is positively related and confidence unrelated to the outcome. Here, the OC approach provides identical results to those from Case 2. Thus, if one were interested in knowledge just as a variable to be controlled (rather than in overconfidence per se), the OC results would be highly misleading, suggesting there is an effect of confidence when there is not one. The Case 3 results with the UJC approach, however, show no relationship between unjustified confidence and the outcome.

For the same reason, the OC approach can lead one to conclude there is no effect of confidence when there is in fact one. Case 4 provides the situation where confidence and knowledge are both positively related to the outcome. Here, the conclusion reached from the OC approach is that there is no relationship, due to the fact that the confidence and knowledge correlations are in the same direction and equal in magnitude. The UJC approach, however, shows that there is a relationship between confidence and the outcome when controlling for knowledge.

General finding 2: When knowledge is unrelated to the outcome, it adds noise to OC as a predictor, but UJC is unaffected by the inclusion of knowledge

The previous general finding showed that when knowledge was correlated with the outcome, effects of OC could be due to knowledge rather than to confidence. When knowledge is uncorrelated with the outcome, inclusion of knowledge in the OC statistic will not produce spurious results per se, but it will dampen any effects of confidence.⁶ This can be seen in Case 1 (Table 3), where confidence is positively correlated with the outcome and knowledge is uncorrelated with the outcome. Comparing the OC-outcome correlation (r_OC,OUT) with the confidence-outcome semipartial correlation, controlling for knowledge (r_{CONF,OUT|KNOW}), we see that the OC-outcome correlation is less (r = .35) than the semipartial correlation (r = .50). This difference results from the fact that knowledge is not predictive of the outcome, and hence simply adds noise to the OC prediction. Since knowledge is ignored in the UJC approach, the full predictive power of confidence is depicted in that analysis. The same effect can be seen in Case 2, where confidence is negatively related to the outcome.

General finding 3: OC, as a measure, is largely determined by the component with the largest variance, but UJC is unaffected by the relative variance of confidence and knowledge

Changing the relative variance of confidence and knowledge has no influence on the semipartial results, since the correlation is a standardized statistic. However, the OC-outcome correlation is affected, since the overconfidence measure is determined primarily by the variable (confidence or knowledge) with the greater variability. This greater influence on the overconfidence measure, in turn, produces a greater influence on the OC-outcome correlation. Hence, when the variability in confidence is greater than the variability in knowledge, the OC-outcome correlation is closer to the confidence-outcome correlation, and when the variability in knowledge is greater, the OC-outcome correlation is closer to the knowledge-outcome correlation.

These effects are demonstrated in Table 4, which focuses on Cases 1 and 4. In each case, we contrast when the variability in confidence is twice that of knowledge, when they are equal, and when the variability in confidence is half that of knowledge (the full range of cases are presented in the Supplementary Material). As discussed previously, the results of the UJC approach are equivalent to the confidence-outcome correlations, since confidence and knowledge are uncorrelated. Thus, the results from the OC approach are closer to the results from the UJC approach in the situation where the variability in confidence is greater than the variability in knowledge. For example, in Case 1, when the variability in knowledge is twice that of confidence, the OC results are fairly different from those of the UJC approach (.29 vs. 50). However, when the variability in confidence is twice that of knowledge, the two analyses produce similar results (.41 vs. .50). Case 4 shows this effect even more dramatically, when the effects of confidence and knowledge on overconfidence are in opposite directions. As the relative variability in knowledge and confidence changes, the OC-outcome correlation goes from positive to negative. With extreme differences in variability (e.g., 20:1, not shown), the OC-outcome correlation ranges from +.50 to −.50. Thus, in the extreme situation where the variability in confidence is considerably greater than the variability in knowledge, the OC and UJC approach produce nearly identical results, but when the variability in knowledge is much greater than in confidence, the two approaches produce vastly different results.

Table 4.

Select set 2 simulations, allowing variances to be different.

	Parameters			OC Approach Results	UJC Approach Results	Comparative Statistics
Case #	VarCONF/VarKNOW	rCONF,OUT	rKNOW,OUT	rOC,OUT	rCONF,OUT|KNOW	rKNOW,OUT|CONF	Multiple R
1	2	.50	0	.41	.50	.00	.50
	1	.50	0	.35	.50	.00	.50
	.5	.50	0	.29	.50	.00	.50
4	2	.50	.50	.12	.50	.50	.71
	1	.50	.50	.00	.50	.50	.71
	.5	.50	.50	−.12	.50	.50	.71

NOTE: Var_X is the variance of X. r_X,Y represents the zero-order correlation between X and Y. r_X,Y|Z represents the semipartial correlation between X and Y controlling for Z. The Multiple R is generated from regressing the outcome onto confidence and knowledge.

General finding 4: As confidence and knowledge become more positively correlated, the two analytic approaches produce more similar results

Overall, the primary influence of a positive correlation between confidence and knowledge is to reduce the difference between the two analytic approaches. This can be seen in Table 5, which presents Cases 1, 4, and 5. As the correlation between confidence and knowledge is increased from 0 to .7, the results of the two approaches become more similar. Indeed, when confidence and knowledge are uncorrelated, examining the effect of unjustified confidence is identical to examining the effect of confidence, but as the confidence-knowledge correlation approaches 1.0, the results of the UJC and OC approaches converge.⁷

In addition, overall predictive ability (as reflected in Multiple R) also generally increases (in Cases 1 and 5, as well as 2 and 3, presented in the Supplementary Material). The one exception to this is Case 4, where, as the correlation between confidence and knowledge increases, the two become redundant with each other and the relative weight is split between the two.

General finding 5: As confidence and knowledge become more positively correlated, they can act as suppressors on each other

The simulations highlight another issue relevant to the two approaches. As evident in Table 5, many of our cases show suppression. In Case 1, for example, as the knowledge-confidence correlation increases, the confidence semipartial correlation (r_{CONF,OUT|KNOW}) grows larger than the zero-order correlation between confidence and the outcome. Balancing this out is a similar suppression effect with the knowledge semipartial correlation (r_{KNOW,OUT|CONF}), which departs from zero and becomes more negative with the increased collinearity. The OC-outcome correlation (r_OC,OUT) demonstrates a similar pattern, with the OC-outcome correlation increasing as the correlation between confidence and knowledge increases. This same general pattern of results holds in all of our cases except for Case 4, and as noted above are also reflected in greater Multiple R. These conditions (correlated predictors with different correlations with the outcome) are in keeping with the conditions where suppression is known to occur (c.f., Cohen & Cohen, 1983) and are explored more thoroughly in the Supplementary Material.

Empirical Studies

To provide concrete examples of the above findings, we next present brief descriptions of three empirical studies taken from our own research that cut across very diverse content domains. The first study compares teens’ knowledge and confidence about sex and drugs with self-reports of risk behavior and socio-economic status. The second study addresses stock-market knowledge and confidence and their relationship to subsequent investment judgments and choices. The third study looks at college basketball knowledge and confidence and their relationship to subsequent betting behavior. Together, they provide real-world examples of the findings above.

The three studies are also of considerable interest in their own right, providing evidence regarding the effects of confidence that does not correspond to knowledge. To conserve space, the methodological descriptions are necessarily brief, and the results are limited to the primary substantive and methodological issues under consideration. One of the datasets is described in detail in a previous paper (Parker & Fischhoff, 2005), although the specific results here are original. Additional detail, including the primary measures, are included in the Supplementary Material, and complete methodologies for the other two datasets (both original to this paper) are available upon request from the first author.

In each of the three studies, we focus on the relationship between responses to a Knowledge-Confidence Assessment (KCA) – designed to elicit domain-specific knowledge and confidence – and key outcome variables. Each KCA involves multiple items and uses the two-alternative forced choice method. Respondents are first asked whether a given statement is true or false, and are then asked to judge the likelihood that they, in fact, chose the correct response, on a scale from 50% to 100%. For each respondent, we compute three variables. Knowledge is the proportion of true/false questions that the respondent correctly answered. Confidence is the mean of all confidence judgments for that individual. Overconfidence is the difference between confidence and knowledge, such that an overconfidence score of zero represents appropriate confidence (on average), positive scores reflect overconfidence, and negative scores reflect underconfidence.

Study 1: Adolescent Risk

Method

As part of a larger project on Youth Decision-Making Competence (Y-DMC), these data were collected at the Center for Education and Drug Abuse Research (CEDAR), an NIH-funded center focusing on the etiology of substance abuse. To this end, CEDAR is implementing an ongoing longitudinal study of families with and without a paternal history of substance abuse (see, e.g., Tarter & Vanyukov, 2001). Respondents are youth recruited at age 10–12, with subsequent interviews every 2–3 years. Their ages at the time of this study were between 18 and 19 (their fourth interview), and the KCA was collected as part of a battery of paper-and-pencil tasks at the end of the main CEDAR assessment. See Parker and Fischhoff (2005) for a more thorough description of the methodology. The current dataset includes the 110 respondents from Parker and Fischhoff (2005), plus an additional 95 respondents not previously available, bringing the total sample size to 205.

The Y-DMC KCA consists of 42 items, with three sets of 14 items focusing on general knowledge (e.g., “A robin’s eggs are orange”), alcohol and drug use (e.g., “Snorting cocaine can cause lung cancer”), and sex and AIDS (e.g., “You can only get the AIDS virus (HIV) from someone who is gay”).

One of the strengths of the CEDAR project is the rich set of comparison measures, collected from the respondents and their families across the 6–10 years of their participation in the study. We focus on one measure of risk behavior – marijuana use – and a measure of socio-economic status (SES; as described in Hollingshead, 1975). Marijuana use is the total number of reported episodes of marijuana use, up through age 16 (Skinner, 1982). For ease of comparison across studies, we reverse coded marijuana use (i.e., multiplied by −1) so that higher scores indicated more positive behavior.

Results and discussion

Performance on the KCA was relatively poor, with a percent correct of .52 and a mean confidence of .60, resulting in a mean overconfidence of .08. Knowledge and confidence were substantially correlated (r = .44), with knowledge having a somewhat larger variance. Hence, these results represent the case where variances differ slightly (with knowledge more dominant in the overconfidence score), as well as relatively strong collinearity.

Table 6 summarizes the results from all three empirical studies, with Adolescent Risk in the top section. Those who used marijuana less were both less confident (r = −.15) and marginally more knowledgeable (r = .13) than those who used marijuana more. Thus, lack of marijuana use demonstrates Case 5, where (in the zero-order sense) confidence and knowledge are associated with the outcome in opposite directions. Both unjustified confidence and overconfidence are associated with poorer outcomes (more marijuana use). Note the effect of unjustified confidence is greater than would be suggested by just examining the zero-order correlation, due to the suppression caused by the collinarity of knowledge and confidence. Substantively, the primary conclusion is that both overconfidence and unjustified confidence are positively associated with risk behavior.

Table 6.

Results of three empirical studies.

			Parameters				OC Approach Results	UJC Approach Results
Study	Outcome Variable	Case #	VarCONF/ VarKNOW	rCONF,KNOW	rCONF,OUT	rKNOW,OUT	rOC,OUT	rCONF,OUT|KNOW
Adolescent Risk	Lack of marijuana use	Case 5	.005/ .008	.44***	−.15*	.13†	−.26***	−.23***
	SES	Case 4	.005/ .008	.44***	.16*	.34***	−.23**	.01
Investment	Fund prediction	Case 3	.007/ .005	.08	−.08	.24*	−.22*	−.10
Basketball	Pick accuracy	Case 4	.018/ .018	.62***	.41***	.28**	.16	.31***

NOTE: r_X,Y represents the zero-order correlation between X and Y. r_X,Y|Z represents the semipartial correlation between X and Y controlling for Z. The Multiple R is generated from regressing the outcome onto confidence and knowledge.

^*p < .05

^**p< .01

^***p < .001

Table 6 also shows that coming from a high SES environment was positively associated with both knowledge (r = .34) and confidence (r = .16), representing Case 4 (potentially leaning towards Case 3). SES shows a strong negative correlation with overconfidence, due to the strong knowledge-SES zero-order correlation and the greater amount of variability in the knowledge measure versus the confidence measure. However, SES is uncorrelated with unjustified confidence, due to the relatively low correlation between confidence and SES, combined with the high confidence-knowledge correlation. Since confidence and knowledge are correlated in the same direction with SES, and the correlation with knowledge is much greater, the semipartial correlation with confidence becomes negligible.

Using real-world data, these results highlight how easily the wrong conclusions could be reached if one is interested in investigating the effects of unjustified confidence and attempts to control for confidence by using the overconfidence approach. As evident in Table 6, the overconfidence results are nearly identical for marijuana use and SES, yet for very different reasons. In particular, whereas unjustified confidence does seem to be related to marijuana use, the overconfidence-SES relationship was due entirely to the knowledge measure. Indeed, the zero-order correlations between confidence and the outcome measure are in different directions for marijuana use and SES. It is only by examining the semipartial correlations that the difference between the two outcome measures becomes apparent.

Study 2: Investment

Method

In the Fall of 2001, 109 students enrolled in an introductory marketing-management class at a large, mid-Atlantic university participated for course extra credit. Most respondents were college juniors and seniors majoring in business-related fields. As part of a larger set of self-administered tasks, participants were tested on their knowledge and confidence regarding the recent performance of different stocks. Specifically, the investment KCA consisted of 50 items, each asking respondents to judge the historical performance of two stocks (e.g., “In the five years prior to 6/31/01, Sony (SNE) stock increased more in value (proportionately) than did the Gap (GPS).”).

Participants also predicted the value of real sector funds four months hence. Respondents were informed that the respondents with the best performance on the prediction task would be rewarded $25. Fund Prediction is the mean absolute deviation of predicted and actual sector-fund values after four months.⁸ For exposition purposes, Fund Prediction is reverse-scaled (i.e., multiplied by −1), so that higher numbers consistently indicate more accurate responses.

Results and discussion

The investment KCA was also very difficult, with a mean percent correct of just .52 and a mean confidence of .68, resulting on average in .16 overconfidence. Unlike the adolescent-risk study, investment knowledge and confidence were largely uncorrelated (r = .08), with confidence having a slightly larger variance.

Fund predictions were better for those with greater knowledge (r = .24), but not for those with greater confidence (r = −.08), providing an example of Case 3. As shown in Table 6, the effect of unjustified confidence on investment performance is negative but non-significant. This result reflects the lack of collinearity between confidence and knowledge, which results in a semipartial correlation that is virtually identical to the zero-order correlation. There is, however, a significant relationship between overconfidence and fund prediction, due to the significant knowledge-outcome correlation and the fact that confidence, although only weakly related to investment performance, is working in the same direction to influence the overconfidence score as is knowledge.

In sum, overconfidence is associated with lower performance on the investment task. Similar to the results with SES, however, this effect is due primarily to the relationship with knowledge and not with confidence, as indicated by the negligible relationship with unjustified confidence.

Study 3: Basketball

Method

In the Spring of 2003, immediately before the Sweet 16 round of the NCAA basketball tournament, 120 students from an introductory marketing-management class at a large, mid-Atlantic university participated for course extra credit. Most respondents were college juniors and seniors majoring in business-related fields. As part of a larger set of tasks, respondents were tested on their knowledge and confidence regarding pre-season rankings of college basketball teams. Specifically, the basketball KCA consisted of 25 items, each asking respondents to judge the relative pre-season rankings of two college basketball teams (e.g., “According to the Associated Press (AP) rankings, before the 2002–2003 college basketball season, Tulsa was ranked better than Indiana.”).

Respondents were then asked to pick the winner of each of the games in the Sweet 16 round. Pick Accuracy is the number of Sweet 16 games (out of 8) for which the respondent correctly picked the winner. Respondents were informed that those with the best performance would be rewarded $25.⁹

Results and discussion

In this study, the mean percent correct on the KCA was .58, with a mean confidence of .69, leading to, on average, .11 overconfidence. Knowledge and confidence were highly correlated (r = .62) and had equal variances.

Both knowledge and confidence were positively related to performance on pick accuracy (r = .28 for knowledge; r = .41 for confidence). Thus, these results reflect an example of Case 4, although the relationship with confidence is somewhat greater than with knowledge. Accordingly, the UJC approach produces stronger results than does the OC approach. Note that the UJC relationship exists despite the large correlation between confidence and knowledge, which has substantially dampened the semipartial correlation.

These results provide an example of where a significant relationship with confidence could have been missed if one only examined the relationship with overconfidence. Although in the same direction as the unjustified confidence semi-partial correlation, the OC correlation is considerably weaker, given that the knowledge – pick accuracy relationship is in the same direction as the confidence – pick accuracy relationship.

Thus, the results with the basketball task are substantially different than with either of the other tasks. First, there is a stronger relationship with unjustified confidence than with overconfidence, unlike the results in the previous situations. Second, unjustified confidence is positively related to positive outcomes, not negatively or unrelated as in the previous studies.

General Discussion

We have attempted to show that there are two distinct perspectives that can be taken when considering the consequences of confidence that does not correspond to knowledge – one focused on the effect of overconfidence (OC) and one on the effect of unjustified confidence (UJC). We began by distinguishing these two perspectives and proceeded to compare the analytic approaches for addressing each of them (summarized in Table 1). We then used a set of simulations to understand the implications of the two analytic techniques, highlighting a set of general findings. Finally, we presented three empirical studies, each examining the correlates of overconfidence and unjustified confidence in specific substantive domains.

The concluding sections first address the choice of analytic method for prediction. Then, we turn to the substantive issue underlying this paper – is confidence that does not correspond to knowledge helpful or harmful? We close with suggested future directions.

Which Analytic Approach is Preferable?

It is clear from our simulations and empirical results that the choice of analytic technique matters – i.e., that the conclusion one reaches with each approach will often be different, and sometimes substantially different. So given that, which approach should be used? Conceptually, the choice of analytic technique should follow from the research question or mechanism under investigation. If one is interested in the effect of confidence and wishes to control for knowledge, one should use the unjustified confidence approach. If one is interested in the effect of the difference between confidence and knowledge, one should use the overconfidence approach. Our review of the literature suggests that most researchers are interested in the effect of confidence controlling for knowledge. The reason for this is that most of the mechanisms under investigation – lack of consideration of additional information, risk taking, etc. – follow from one’s confidence level rather than its deviation from knowledge. Since it is the person’s confidence level, not their knowledge level, that is known to the decision maker, it is confidence that is driving their behavior, whether that is to take risks, seek out additional information, or some other action.

Nonetheless, some mechanisms are hypothesized to result directly from the mismatch of confidence and knowledge. As discussed previously, one example of this type of mechanism can be seen in the work of McGraw et al. (2004), who explicitly were interested in changes in pleasure due to the mismatch between performance and expectations. Other potential mechanisms result from the evaluation of a person’s confidence assessments when both their confidence and knowledge levels are known. From a consumer of probability information perspective (see, e.g., Yates, Price, Lee, & Ramirez, 1996), does the perceived credibility of others and trust in their judgments decrease as a mismatch between confidence and knowledge increases? For example, Tenney, Spellman, and MacCoun (2008) suggest that, when judging credibility, people attend not only to overall levels of confidence and knowledge, but also to the match between them, initially presupposing that those making confidence assessments are well-calibrated but overriding that assumption when evidence of overconfidence becomes available.

Thus, ideally the choice of analytic technique should be based on the theoretical issues under investigation. That said, not all researchers collect data with well-defined research questions, and exploratory work can be useful at early stages of investigations. In these cases, it is helpful to know which will typically produce the stronger results – the partial correlation of confidence with the outcome or the zero-order correlation between overconfidence and the outcome – and why.

On the basis of our simulations, in most cases the unjustified confidence approach will produce stronger results. When confidence and knowledge are related to an outcome in the same direction (see Case 4, Table 3), these effects will cancel out each other within the overconfidence approach, but can work in tandem within the unjustified confidence approach. Further, when knowledge is unrelated to the outcome (Cases 1 and 2), it merely adds noise when using the overconfidence approach. It is only when there is a substantial knowledge-outcome relationship in the absence of a confidence-outcome relationship (Case 3) or a confidence-outcome relationship in the opposite direction (Case 5) that the overconfidence analysis will produce stronger results. Again, it is important to note that the reason for this stronger relationship in these situations rests on the role of knowledge, and so the interpretation of the results must acknowledge that fact. Finally, the overconfidence score is more susceptible to differential variance in confidence and knowledge, the effects of which were displayed in the adolescent-risk empirical study. Whether that will produce stronger results for the unjustified confidence or overconfidence approach depends on the precise case under investigation and whether confidence or knowledge has the greater variability.

Overall, our view is that overconfidence is typically more attractive as a dependent variable, where over- versus underconfidence (i.e., on which side of the zero point an individual falls) is an interesting outcome. When used as a predictor variable, however, such thresholds have no meaning, at least when used in correlation-based analyses. Even in the dependent-variable case, however, there may be benefits from considering confidence as the dependent variable, controlling for knowledge. Such an analysis makes less stringent metric assumptions on confidence, providing greater robustness if responses on the probability scale are not strictly probabilities in respondents' minds.

Finally, although the analytical approaches will often produce different results, it is worth re-emphasizing that they are related. Literally, examining the effect of unjustified confidence is identical to examining the effect of confidence when confidence and knowledge are uncorrelated, but is identical to examining the effect of overconfidence when confidence and knowledge are perfectly correlated. Thus, in a very real sense, the OC and UJC approaches become indistinguishable when confidence and knowledge are highly correlated. As has often been noted, however, confidence and knowledge are frequently relatively uncorrelated (Alba & Hutchinson, 2000). Since the two analytic approaches differ the most in such situations, we would expect frequent divergence in results of the two approaches.

Is Confidence that Does Not Correspond to Knowledge Necessarily Bad?

As should be evident, the answer to this question depends on whether one is interested in examining the effects of overconfidence or of unjustified confidence. In terms of overconfidence, the results across the studies and measures are reasonably consistent. With the exception of the basketball study, greater overconfidence was associated with negative outcomes or influences in each of the studies. For example, people who displayed a larger difference between their confidence and knowledge scores made less (hypothetical) money on the investment task than did people who had a smaller difference between their perceived and actual knowledge. Overconfidence was similarly related to marijuana use and lower SES. These results should not be particularly surprising, given the substantial positive relationships between knowledge and the criterion variables in most cases. Since overconfidence treats knowledge and confidence equivalently, there will tend to be a negative relationship between overconfidence and the criterion variable as long as the (positive) relationship with knowledge is stronger than the relationship with confidence. Furthermore, as discussed below, when confidence has a negative relationship with the outcome, the OC-outcome relationship will be even stronger – a result amplified by suppression effects when confidence and knowledge are positively correlated (General Finding 5).¹⁰

The results with unjustified confidence, however, were less consistent across the measures and studies. In terms of marijuana use, it appears that people who think they know more, holding constant how much they actually do know, are more apt to exhibit risky behavior than people who are less confident about their knowledge. Since marijuana use is generally considered maladaptive (see Parker & Fischhoff, 2005), unjustified confidence appears to be problematic in this situation. In contrast, unjustified confidence in the basketball study was actually a positive predictor of performance, perhaps because confidence influenced some (unmeasured) psychological or behavioral mechanism (e.g., willingness to act on the knowledge they do possess) that was in turn beneficial in this situation. Unjustified confidence, however, was unrelated to SES or performance on the investment task. The inconsistent results across studies suggest that unjustified confidence has different effects in different situations and that the effects of unjustified confidence, working through psychological and behavioral mediators, may be moderated by situational factors (see Stone et al., 2004).

Future Directions

As we have emphasized throughout, the choice of analytic approach should be based on the research question being asked, and use of an analytic technique inappropriate for the research question under investigation can easily lead to an incorrect conclusion. Indeed, the analyses presented here contrast only two approaches to characterize confidence that does not correspond to knowledge. Other approaches, for example using calibration indices (Yates, 1990) or the absolute value of overconfidence (Bruine de Bruin, Parker, & Fischhoff, 2007; Parker & Fischhoff, 2005), may be appropriate for other specific research questions, and future research should consider the implications of those approaches. However, the two approaches considered here are logical extensions of the existing literature and are appropriate for addressing a wide range of issues.

The choice of analytic method, of course, has implications for measurement of confidence, in that only the overconfidence approach requires that confidence and knowledge are measured on a ratio scale. Our general preference would be for ratio-scale measurement, given that it gives greater flexibility in analytic approach (OC is impossible without it), but future research should consider the implications of different measurement scales.

Note that, in addition to differing in terms of measurement, the overconfidence approach has typically been associated with item-level confidence judgment, whereas the unjustified confidence approach has been associated with aggregate confidence judgments. Substantial research suggests differences between item-level judgments (of the sorts used here and in most behavioral decision research) and aggregate confidence judgments (of the sorts more commonly in social psychology and elsewhere), such that item-level judgments are more likely to show overconfidence (e.g., Gigerenzer, Hoffrage, & Kleinbölting, 1991; Sniezek & Buckley, 1991; Stone, Rittmayer, Murray, & NcNeil, 2011). It is thus plausible that the two types of confidence assessments would produce different correlations with downstream outcomes. To examine this issue, however, it is crucial that researchers do not confound the analytic approach with the type of confidence judgment, instead using the relevant analytic approach to examine both the impact of item-level and aggregate confidence.