Physicians’ metacognition in medical decision-making

Abstract

A large literature reports that physicians are overconfident when making medical decisions. However, the validity of these findings is questioned due to methodology pitfalls leading to poor ecological validity. Moreover, most previous studies do not distinguish between (i) metacognitive bias, the general tendency to report high or low confidence irrespectively of actual accuracy, and (ii) metacognitive sensitivity, the capacity to discriminate between one’s own correct and incorrect decisions. Here, we aim to overcome these limitations by drawing from state-of-the-art paradigms and tools from the cognitive science of metacognition. To examine overconfidence and to disentangle metacognitive bias from metacognitive sensitivity, we developed a carefully controlled set of case-vignettes of patients with headaches that was completed by 52 physicians. We found that physicians’ confidence, reliably discriminated their correct from their incorrect decisions, demonstrating genuine metacognitive sensitivity. However, physicians were also overconfident, and this overconfidence was most pronounced among the least accurate participants. Moreover, the costliest errors (i.e. missed urgent cases) were held with higher confidence than unnecessary referrals. Finally, unlike previous reports, we found limited interindividual variability depending on specialty, gender, or seniority level, although our study was not powered for these comparisons.

Highlights

• Physicians were overconfident, replicating previous reports, yet their confidence reliably discriminated correct from incorrect decisions—genuine metacognitive sensitivity, quantified here for the first time with signal detection theory metrics.

• Metacognitive measures showed limited variation by specialty, gender, or seniority (the study was not powered for these comparisons), though specialties differed markedly in their urgency-referral criterion.

• The costliest errors—missed urgent cases—were held with higher confidence than unnecessary referrals.

• Refining medical training to assess and cultivate metacognitive sensitivity—rather than merely mitigating overconfidence—could help better align confidence judgments with diagnostic accuracy.

Introduction

Decision-making is a cornerstone of medical practice, profoundly influenced by a multitude of factors such as time constraints, stress, cognitive and emotional load, personality traits, fatigue, and uncertainty. These elements shape the professional environment of physicians, impacting their clinical reasoning and decision-making processes (Croskerry 2022). A significant challenge in a context of uncertainty is a lack of direct, immediate feedback, preventing physicians from refining their future medical strategies. In this context, metacognition, our ability to evaluate and reflect on our own cognitive capacities and decisions, is crucial. Metacognition enables the self-assessment of physicians’ medical performance, for instance by generating explicit or implicit confidence judgments that, in turn, influence a number of processes, such as exploring alternative options or seeking a second opinion (Boldt et al. 2019). Biased confidence judgments can have important negative consequences, e.g. when an overconfident doctor becomes blind to signs or symptoms that go against their initial diagnosis. While advances in medical technology and artificial intelligence offer promising tools to mitigate such biases, they are not to entirely replace the nuanced decision-making processes that experienced physicians bring to the table; in particular, this capacity for accurate confidence seems still bound to humans (Griot et al. 2025).

Overconfidence in physicians has been repeatedly reported over the past decades (Berner and Graber 2008, Aiyer et al. 2025). A systematic review identified overconfidence as one of the most common cognitive biases in medical contexts, with a prevalence between 46.3% and 70% (Saposnik et al. 2016). To examine the link between physicians’ objective performance and their subjective perception of it, classic experiments consist of surveys with knowledge questions, clinical case vignettes, or clinical procedures, each associated with a retrospective or concurrent confidence rating. However, analysis strategies and computed indicators vary from one study to another (correlations, calibration/bias, and to a lesser extent discrimination/sensitivity), and results about the link between accuracy and confidence remain unclear (Friedman et al. 2005, Davis et al. 2006, Meyer et al. 2013, Naguib et al. 2019, Richardson et al. 2020, Lakhlifi et al. 2023).

However, the variability and complexity of these empirical results is related to a number of theoretical and practical experimental limitations. A first issue lies in the low ecological validity of experiments, where overconfidence may be an artefact resulting from artificial stimuli whose content, difficulty, and frequency poorly represent real-world situations (Gigerenzer 1991). A second hurdle is the partially intertwined nature of objective performance and subjective confidence (Maniscalco and Lau 2012). Confidence partly stems from the same inputs that determine performance, e.g. the objective difficulty of the case. To put it simply, easier cases will lead both to better performance and (rightfully so) higher confidence. In contrast, behavioral studies of metacognition have typically relied on comparing subjective confidence estimates to objective accuracy across a sequence of decisions to derive various indicators of metacognitive abilities (Schraw 2009, Maniscalco and Lau 2012, Norman et al. 2019, Rouault et al. 2019, Rahnev et al. 2020). Two main families of indices have been proposed: metacognitive bias reflecting one’s general tendency to report high or low confidence (notions of calibration, overconfidence), and metacognitive sensitivity, reflecting one’s capacity to discriminate between their own correct and incorrect decisions (notions of resolution, discrimination, metacognitive accuracy, metacognitive efficiency) (Schraw 2009, Fleming and Dolan 2012, Maniscalco and Lau 2012, Fleming and Lau 2014, Norman et al. 2019, Rouault et al. 2019, Rahnev et al. 2020). However, the analysis of confidence in clinical decision-making has almost exclusively focused on the former and overlooked the latter. This second notion may indeed reflect a crucial facet of metacognitive ability for accurate medical decisions, for instance, being able to seek additional evidence or a second opinion when confidence is rightfully low. Moreover, different combinations of metacognitive bias and metacognitive sensitivity can arise. While showing an overall high (overconfidence) or low (underconfidence) metacognitive bias, individuals can at the same time either adequately assess their accuracy by expressing stronger confidence in correct than incorrect decisions (high metacognitive sensitivity), or confound the quality of their decisions by reporting overlapping confidence judgments for correct and incorrect decisions (low metacognitive sensitivity). A third pitfall arises from analysis strategies, where some effects have been shown to arise from statistical artifacts (Olsson 2014, Gignac and Zajenkowski 2020, Dunkel et al. 2023); moreover, the strength of the overconfidence effect depends on what confidence level is defined as optimal.

Although these considerations are extensively discussed in metacognition research, they have not yet been addressed in clinical decision-making research (though see Naguib et al. 2019, Lakhlifi et al. 2023 for exceptions). This indicates a need for a better integration of theoretical insights and methodological improvements into protocols studying physicians’ decision-making and their associated confidence.

Hence, we developed a new experimental design to avoid a bias toward demonstrating overconfidence and to enable the study of metacognitive sensitivity in clinical decisions. We developed 60 two-alternative forced-choice case vignettes of patients with acute headaches, which were administered online to 52 physicians. We designed the task, instructions, confidence scales, and vignette content to be as ecological and representative of real-world scenarios as possible (Kostopoulou et al. 2019). Critically, our protocol allows for the application of type-I and -II signal detection theory analyses to disentangle metacognitive bias from metacognitive sensitivity (Fleming and Lau 2014). We found that physicians were not as overconfident as previously estimated, and that their confidence could discriminate between their own correct and incorrect judgments. Contrary to previous reports and popular belief, we found limited variations according to medical specialty, gender, and seniority level. These findings temper previous reports of overconfidence in physicians and highlight the practical importance of cultivating metacognitive sensitivity in medical training to enhance diagnostic accuracy and patient outcomes.

Methods

Study design

This preregistered experiment (https://osf.io/q7zrm) consisted of an online questionnaire chunked into six sessions (Fig. 1). This research received approval from our local ethics committee (CER-Sorbonne Université; protocol CER-2022-Lakhlifi-METADECIDOC) and complies with European General Data Protection Regulation. Participants gave their informed consent. Since the study was conducted through an online platform, participants were presented with an information page on the welcome screen of the survey. At the end of the page, they were asked to click on a box to indicate that they have read and understood the information provided and consent to participate in the study.

Figure 1.

Experimental paradigm. Fifty-two physicians were presented with case-vignettes describing an acute non-traumatic headache. The text could contain zero, one, two, or three red flags according to clinical official guidelines, for which the expected response was nonurgent (0) and urgent (1, 2, or 3) respectively. All case vignettes were validated by neurologists specializing in headaches. Participants indicated their decision (binary) together with their decision confidence on a continuous visual analog scale ranging from “I am not confident at all: I make my decision randomly” to “I am totally confident: I am sure of my decision.” At the end of the survey, participants were further asked for a global retrospective estimate of confidence, that is their expected proportion of correct responses overall (see Methods).

Participants registered their contact data and demographic profile, before being presented with instructions and an example. They reported their gender, age, level of experience (number of years of practice since the end of residency), additional qualifications, type of practice (public/private, hospital/medical office), medical specialty (neurologists, emergency physicians, or general practitioners), research activity (and if any, the percentage of research in their professional time). Participants were also asked to indicate the average number of patients presenting with headaches they see per month. This step allowed us to check whether potential participants met inclusion criteria before continuing (see Exclusion Criteria section below). Following the validation of their profile, participants received a daily email with a link to the survey, with a reminder of task instructions followed by one of the five sessions of cases. Each session was composed of 12 case vignettes to examine. The 60 case vignettes were pseudo-randomly distributed across five sessions (with a balance of 25%–75% of signal-present(/absent) trials within session). The order of sessions was randomized across participants (Latin square design). The order of the 12 vignettes was randomized within each session for each participant. The collection of identifying data and the management of email invitations were handled via REDCap (Harris et al. 2009, 2019) while other demographic data and responses to case vignettes were collected via LimeSurvey (LimeSurvey GmbH, n.d.). Participant recruitment was supported by the Paris Brain Institute PRISME Core Facility (RRID:SCR_026394), Paris, France.

Case vignettes

Our case vignettes and protocol were inspired by previous work evaluating general practitioners’ referral rates based on signal detection theory (Kostopoulou et al. 2019). We conducted a pilot study evaluating an initial set of 30 vignettes based on the International Classification of Headache Disorders’ official diagnostic guidelines and academic publications on a convenience sample of nine residents and attendants in neurology and emergency medicine. We focused on headache since it is a frequent symptom, often disabling, with numerous possible causes, ranging from benign to life-threatening, thus offering variability in the required response. We then controlled the signal (clinically relevant information) and the noise (irrelevant information) to build a set of 60 literature-based ecologically relevant and experimentally tailored vignettes. The final set of vignettes was based on a mix of SNNOOP10-list (Do et al. 2019), the International Classification of Headache Disorders, 3rd edition (ICHD-3) (Headache Classification Committee of the International Headache Society (IHS) 2018), and guideline recommendations (Moisset et al. 2016, 2018, Mawet et al. 2018). Each of these 60 vignettes contains multiple pieces of information detailing the symptom sets of a patient suffering from an acute nontraumatic headache. In light of available information about the patient’s profile and symptoms, participants were instructed to make a binary triage decision about whether the case required or not urgent investigation management (urgent: within the next 24 hours, non-urgent: in the next day(s)/weeks). According to official guidelines on which the case vignettes were created, the presence of typical red flags in the patient’s presentation should lead physicians to consider the case as urgent. Half of our vignettes were considered “urgent” as they entailed at least one red flag (signal-present trials) and could in majority be suspected of being secondary serious headaches (high likelihood of a serious intracranial underlying cause). These vignettes could have one, two, or three red flags, which constituted different difficulty levels. The other half was mainly composed of cases of primary headaches (signal-absent trials) that did not represent emergency situations, for which patients could be safely sent home with instructions to have an examination in the following days/weeks if the pain persisted. Participants were not informed of the half–half distribution.

Cues introduced as red flags in signal-present vignettes included: (i) age above 50, (ii) onset of headache is sudden or abrupt, (iii) positional headache, headache with exertion, (iv) pattern change of headache, new headache, (v) systemic symptoms (e.g. fever); (vi) neurological deficit or dysfunction (e.g. decreased consciousness), (vii) neck stiffness or neck pain, (viii) pathology of the immune system, immunosuppression, (ix) eye ptosis or pain, (x) pregnancy, (xi) neoplasm, and (xii) potential infection (e.g. travel abroad). Four of these symptoms never appeared alone as a red flag but in combination with another red flag (v, ix, xi, xii). Fever appeared with two other red flags, as it is considered an “orange” flag only in isolation (28). The reason these symptoms did not appear as a single red flag is that to maintain ecological validity, each case vignette was designed around a plausible potential diagnosis with common symptom presentations, and for certain diagnoses, it was rare to see one red flag alone. Examples of “urgent” and “non-urgent” vignettes are provided in the supplementary materials. We ensured that the set was balanced between urgent and non-urgent cases in terms of the case patient’s gender (chi-square test, P = .796), age (two-sided t-test, P = .765) and vignette text length (two-sided t-test, P = .179). We also verified the impact of the vignette patient’s demographics. Unsurprisingly, the patient’s age was linked to participants’ accuracy (Spearman correlation, rho = 0.26, P = .045). The patient’s gender did not affect participants’ accuracy (Wilcoxon-Mann–Whitney test, W = 542.5, P = .17), nor did vignette’s length (Spearman correlation, rho = 0.0095, P = .94). Mean confidence in each vignette was not related to the patient’s age (Spearman correlation, rho = 0.11, P = .41), vignette’s length (Spearman correlation, rho = −0.17, P = .20) or gender (Wilcoxon–Mann–Whitney test, W = 539.5, P = .19).

Data collection

Participants were asked to examine and solve the case vignettes as they would have done in their daily practice in their usual work environment. Case vignettes were two-alternative forced choices presented one at a time. Each vignette included a short reminder of instructions, the main question to be answered (“Do you think that this case requires urgent (within the next 24 hours) or non-urgent (within the next few days/weeks) diagnostic management?”) followed by two response buttons, “urgent” and “non-urgent” (Fig. 1). After each decision and on the same screen, participants were asked to explicitly report their confidence in their decision on a visual analog scale ranging from “I am not confident at all: I make my decision randomly” (collected as 50%) to “I am totally confident: I am sure of my decision” (collected as 100%) (Fig. 1). The use of two-option questions combined with a confidence judgment between 50% and 100% is a frequent practice in judgment and decision-making studies (Juslin et al. 2000, Detsky et al. 2006, Locker et al. 2006, Hainer and Matheson 2013, Moisset et al. 2016, 2018, Mawet et al. 2018, Olesen 2018, Cutrer et al. 2019, Do et al. 2019, Wootton et al. 2021) and has already been used in studies evaluating physicians’ confidence (Naguib et al. 2019). This confidence scale match accuracy units, ranging from chance level to maximum confidence. No numerical values were indicated on the scale to prevent any anchoring biases. Both the triage decision and the confidence judgment were compulsory to proceed to the next case. Participants had unlimited time to answer each case. At the end of the last session, participants were asked to give an overall global retrospective estimation of the proportion of trials they believed to have correctly solved (Gigerenzer 1991) henceforth referred to as “global confidence judgment.”

Participants

Emergency physicians, general practitioners and neurologists with an official national registration number and practicing in France constituted our population of interest. Participants were recruited between 7 March 2023 and 11 June 2023 through a combination of professional mailing lists (direct contacts, Haute Autorité de Santé, Fédération Française de Neurologie, Collège de la Médecine Générale, Société Française de Médecine d’Urgence, Collège Français de Médecine d’Urgence. . .) and social media (LinkedIn, Twitter). The survey distribution was boosted to foster participation by a communication strategy emphasizing the need for more research to better understand triage decisions while facing patients with acute headaches, without any mention of the interest in confidence. Participants who completed the whole study were compensated for 50€.

Sample size

Our design relies on 60 vignettes, a higher number than all previous studies exploring physicians’ confidence in their decisions (most had fewer than 10 cases), and that typically tested 25 to a few hundred participants. Considering this and practical constraints (time pressure, funding for participants’ compensations, limited pool of potential participants of our targeted specific profiles), a sample size goal was set at 50–60 participants to ensure reasonable statistical power with a balanced distribution of medical specialties (at least 15 GPs, 15 neurologists, and 15 emergency physicians) (Fleming 2017).

Exclusion criteria

No session met our preregistered exclusion criterion about response time (sessions with <10 s to answer to at least 8 case vignettes out of 12, as this represents a strong clue of rapid and random completion without reading the case). According to a binomial law with P = .5 and 60 trials, the probability of obtaining the lowest accuracy rate 38/60 observed among our participants by responding randomly to all vignettes is 0.026. Therefore, we did not exclude any participant based on accuracy. Three participants reported confidence judgments above 95% for 50 or more decisions out of 60; another participant gave confidence judgments above 99% for all decisions. Considering that the accuracy of these four participants was high (at least 45/60 correct responses), their high confidence could be justified, and we thus decided not to exclude them from the analyses.

Statistical analysis of behavior

We computed several preregistered (https://osf.io/q7zrm) indicators from our measured variables to depict all possible relevant facets of physicians’ metacognitive abilities.

First, for accuracy, analyses of metacognitive capacities require cases to have an objectively correct answer. However, clinical decision-making and especially triage decisions are inherently complex tasks with a variable level of uncertainty. Therefore, we preregistered two possible definitions for correctness, according to: (i) official guidelines (with verification from experts in the field); (ii) the participants’ majority response (>50% of participants). We expected minimal discrepancies in responses’ correctness according to these two definitions, which was indeed the case (supplementary materials). Based on signal detection theory, we computed for each participant their type-1 correct responses according to official guidelines by pooling type-1 hit rates (“urgent” response for signal-present stimulus, i.e. with at least one red flag) and type-1 correct rejections (“non-urgent” response for signal-absent stimulus, with zero red flag). Similarly, type-1 incorrect responses were computed from type-1 misses (“nonurgent” response for signal-present stimulus, with at least one red flag) and type-1 false alarms (“urgent” response for signal-absent stimulus, with zero red flag). Type-1 discrimination (d’) and type-1 bias (c criterion) were calculated using type-1 hit (HR1) and false alarm rates (FA1): type-1 d′ = z(HR1) − z(FA1); type-1 c = −0.5 × (z(HR1) + z(FA1)), with z the inverse of a cumulative normal distribution. For four participants, the real number of false alarms was 0 but was forced to 1 for calculation purposes. All other participants had at least one false alarm.

Second, for each participant we extracted metacognitive bias (also referred to as calibration index in the literature) as the difference between their average confidence and their average accuracy across decisions (Fleming and Lau 2014, Rouault et al. 2022). This is to compare their percentage of accuracy (with 50% representing chance level) to their average confidence judgment (ranging from 50% to 100%).

Third, for each participant, we computed a “global metacognitive bias” (analogous to the one above based on local confidence), as the difference between their unique global retrospective estimate provided at the end of all sessions and their average accuracy.

Fourth, to examine whether participants’ confidence reports were reliably associated with their objective performance, we focused on metacognitive sensitivity using two indices. In general, metacognitive sensitivity is said to be good when correct decisions are endorsed with high confidence and incorrect decisions are held with lower confidence.

A first index, discrimination, was calculated as the difference between participants’ mean confidence in their correct and incorrect responses. We also compared confidence between type-1 hits, correct rejections, misses, and false alarms.

A second index of metacognitive sensitivity based on signal detection theory was computed: meta-d’. It indicates how well each participant’s confidence discriminates between their own correct and incorrect decisions, independently from their tendency to use high or low confidence reports on the scale (which corresponds to the metacognitive bias; Fleming 2017). This approach posits a generative model of confidence data within a signal detection theory (SDT) framework. Fitting the model to data returns a parameter, meta-d’, which reflects the level of first-order performance (known as d’) that would have led to the observed confidence rating data under an ideal observer model. Meta-d’ can then be compared to actual first-order d’ (as we did here by computing the Mratio, meta-d’/d’) to give a measure of metacognitive ‘efficiency’, which quantifies the level of metacognitive sensitivity relative to first-order performance. By using metacognitive efficiency as our measure of metacognition, it is meaningful to compare metacognitive scores across individuals. We employed a hierarchical Bayesian framework for fitting meta-d’, with all convergence values Ȓ < 1.006 indicating satisfactory convergence (Fleming 2017). We directly report the ratio meta-d’/d’, called metacognitive efficiency, which corresponds to metacognitive sensitivity divided by objective d’.

The commonality between both the discrimination metric and the metacognitive efficiency metric, is that they capture how well an individual’s confidence ratings discriminate between their own correct and incorrect decisions; they carry the same psychological meaning. The difference between both metrics is that the latter further accounts for first-order performance, which could otherwise conflate the findings.

All statistical analyses were conducted using R version 4.2.2 (R Core Team 2022). Continuous variables were reported as mean and standard deviation (SD), and range (min and max) for normally distributed data, or as median and interquartile range (IQR) for non-normally distributed data. Intergroup comparisons were performed using Welch’s two-sample t-test, paired t-test, Wilcoxon–Mann–Whitney test, Wilcoxon signed-rank test, chi-square test, one-way analysis of variance (ANOVA) and Kruskal–Wallis test as appropriate. The level of statistical significance was set at P < .05. Regarding interindividual differences, we conducted complementary Bayesian analyses on d′, type 1 response c criterion, local metacognitive bias, global metacognitive bias, and discrimination ability to better quantify the evidence for both the null and alternative hypotheses. These analyses were performed using the BayesFactor R package (v0.9.12-4.7). For variables with more than two groups (medical specialty and seniority level), Bayesian ANOVAs were conducted. For binary variables (gender and research activity), Bayesian independent-samples t-tests were used. Bayes factors (BF₁₀) were computed using the default Jeffreys–Zellner–Siow (JZS) prior with a Cauchy scale parameter of r = 0.707.

Finally, to investigate whether objective performance influenced confidence over and above vignette difficulty, we conducted an additional linear mixed-effects analysis (lme4 R package, v1.1-35.1). Confidence ratings were the dependent variable. Response accuracy (correct versus incorrect), number of red flags (0, 1, or ≥2), and their interaction were entered as fixed effects. To account for the repeated-measures structure of the data, the model included crossed random intercepts for participants and vignettes. Fixed effects were assessed using Type II Wald χ² tests (car R package, v3.1-2). Post hoc estimated marginal means comparisons with Holm correction were also examined (emmeans R package, v1.8.9).

Data and code availability

The complete stimuli set (in French) and the dataset are available upon reasonable request.

Results

Experimental design

Fifty-two participants (31 women and 21 men) between 26 and 64 years old (median = 35.5 [IQR = 31.75–42]) completed the experiments. There were 19 general practitioners (GPs), 18 emergency physicians, and 15 neurologists with no gender or age imbalance across medical specialties (ANOVA for age, chi-square test for other categorical variables; all P > .05, Supplemental Table 1). Most of them worked in a public hospital (29, mainly emergency physicians and neurologists, i.e. 56% of the sample) or a medical office (19, mainly GPs, i.e. 37% of the sample, Supplemental Fig. 1). Nineteen participants had <5 years of professional experience (37%), 13 participants between 5 and 10 years (25%), 9 participants between 11 and 15 years (17%) and 11 participants >16 years of experience (21%). The majority of participants (38, i.e. 73%) reported devoting all their professional time to clinical practice; among the 14 participants who are involved in research, 10 participants declared spending >10% of their time doing research. Participants reported varying levels of exposure to patients presenting with acute nontraumatic headaches, with a mean of 23.8 (SD = 26.3) cases per month. Participants took a median duration of 57 min [IQR = 44–80] to complete the five-session survey, with a median of 11 min [IQR = 8–15] for each session of 12 case vignettes. Most participants respected the suggested planning of doing one session per day for five consecutive days (median time between two sessions: 1.5 days [IQR = 1.25–2]).

Decision accuracy

Fifty-two physicians were presented with case vignettes describing a patient consulting for an acute nontraumatic headache. Accuracy varied across the vignettes: for each vignette, the median number of physicians answering correctly according to the guidelines was 44 out of 52 (84.6%; [IQR = 36.5–49], range = 14–52). We obtained virtually identical results when defining accuracy as the group’s majority response, instead of the official guidelines, as preregistered (again 84.6%; [IQR = 37.75–49], range = 27–52). Consequently, we focus the remainder of the paper on accuracy defined according to official guidelines and report all results with our alternative definition of accuracy in the supplementary materials. Accuracy was relatively high, with a mean of 78.6% correct and a median of 78.3% correct [IQR = 75.0%–83.3%] (range = 63.3%–88.3%). Individual decision sensitivity (hit rate) and 1—specificity (false alarm rate) coordinates were plotted, together with theoretical receiver operating characteristic curves for different d’ values for visualization purposes (Fig. 2A). Mean d’ across participants was 1.769, confirming good discriminability.

Figure 2.

(A) Scatterplot of decision sensitivity (hit rate) and 1—specificity (false alarm rate) for the 52 participants with the group mean d’ (plain line) and theoretical receiver operating characteristic curves produced by different values of d’ ( dotted lines). In our design, d’ could theoretically vary from 0 to 3.67. Each dot represents one, two, or three participants (N = 52). (B) Participants’ accuracy (percentage of correctly solved trials) (left) and mean confidence over the 60 vignettes (right) (N = 52). Both variables are expressed in percentages between 50% and 100%. There were 46 overconfident and 6 underconfident participants. ^***P < .001. (C) Scatter plot of participants’ individual metacognitive bias as a function of their accuracy. Each dot represents a participant (N = 52); dots with a metacognitive bias above zero represent participants with overconfidence. The line represents a linear regression fit, with the shaded gray area indicating the 95% confidence interval. Marginal density plots on the top and right margins illustrate the distributions of accuracy and metacognitive bias, respectively. Metacognitive bias correlated with accuracy on the 60 vignettes (R and P indicate Pearson’s correlation coefficient and the statistical significance).

As expected, the number of red-flags reflecting the strength of signal for each stimulus (i.e. vignette) had a significant impact on participants’ accuracy (Kruskal–Wallis test, P = .017). Participants were less accurate when facing vignettes containing only one red flag (median = 57.7% [IQR = 42.3%–84.6%]) compared to vignettes with 0 (median = 86.5% [IQR = 82.7%–94.2%]; Dunn’s post hoc test with Benjamini–Hochberg [BH] multiple test correction, P = .020) and two or more red flags (median = 87.5% [IQR = 70.7%–95.7%]; BH–Dunn’s post hoc test, P = .027), plausibly facing a higher level of uncertainty with vignettes entailing a unique red flag compared to vignettes with no or several red flags.

Local confidence

Participants overall reported high confidence judgments in their triage decisions with a mean of 87.5% (sd = 5.8) and a median of 88.2% [IQR = 83.3%–90.6%] (range = 73.3%–99.5%). Local median confidence across vignettes (reported on a scale ranging from 50 to 100%) was 87.2 ([IQR = 85.6–89.0]. This reported mean confidence in each case vignette was not affected by the number of red flags (Kruskal–Wallis test, χ2 = 1.75, df = 2, P = .42) or the vignette text length (Spearman correlation, rho = −0.17, P = .20).

Metacognitive bias

Participants’ level of subjective confidence was significantly higher than their objective accuracy (one-sided paired Wilcoxon signed-rank test, V = 1290, P = 2.3e-8) (Fig. 2B). We summarized this information in a metacognitive bias index (difference between average confidence and average accuracy; see Methods). For most participants, metacognitive bias was positive, with a mean of 8.9 (sd = 8.1) and a median of 8.5 [IQR = 2.6–14.5] (range = −8.4 to 24.8), reflecting participants’ miscalibration in the form of overconfidence. At the group level, however, we did not observe any correlation between participants’ accuracy and their average confidence at the group level (Spearman correlation, rho = −0.07, P = .40), indicating variable degrees of overconfidence.

Metacognitive bias negatively correlated with participants’ accuracy (Fig. 2C). The higher the doctors’ performance, the lower their overconfidence (Pearson correlation, R = −0.69, P = 1.5e-8).

Global retrospective confidence: an alternative to local confidence judgments

Complementing local confidence judgments for each case, we also asked participants retrospectively, at the end of the survey, to estimate the proportion of trials they believed they had correctly answered overall. This measure is an alternative form of confidence that participants had in their knowledge or decisions, at a global scale rather than a local, decision level. The comparison of this estimate with the actual average accuracy provided an alternative ‘global’ metacognitive bias/calibration index.

Global median confidence across participants was 83.3 on a 50–100 scale [IQR = 83.3–91.7]. As highlighted in previous work (Gigerenzer 1991), in our sample also overconfidence disappeared when measured at a global level. This analysis revealed that participants significantly underestimated their performance at the global level (global confidence against accuracy, paired Wilcoxon signed-rank test, V = 1 331, P = 5.2e-9) (Fig. 3). Indeed, the indices calculated as the difference between their estimated and real accuracy were mostly negative with a mean index of −9.6 (sd = 19.1) (one-sample t-test for negativity, P = .00035) and a median of −10.0 [IQR = −18.3 to 2.1]), in contrast to metacognitive bias at the local level (Fig. 2).

Figure 3.

Two forms of confidence. The vertical axis represents the difference between confidence (local decision by decision, left; or retrospective global, at the end of the survey, right) and accuracy. Each dot represents one participant (N = 52).

Metacognitive sensitivity

Besides physicians’ metacognitive biases at the local and global confidence levels, a key goal of our study was to evaluate their metacognitive sensitivity, i.e. their capacity to discriminate between their correct and their incorrect decisions by reporting higher confidence in the former than in the latter. We used a two-way repeated-measures ANOVA with a linear mixed-effects model to compare the individual mean confidence judgments across the types of response defined by correctness (trial correctly or incorrectly solved) and urgency (urgent or nonurgent objective categorization). We further included the urgency factor in case it modulates the effect of correctness on confidence Lee et al. 2018. This may be particularly important here since the two possible types of errors do not carry the same risk; a missed urgent case is not equivalent to an unnecessary urgent referral. Type-II Wald chi-square tests revealed a significant difference in mean confidence judgments between correct and incorrect answers (χ2 = 85.63, df = 1, P < 2.2e-16), with higher reported confidence in correct as compared to incorrect answers (Fig. 4A). In addition, we found a significant difference in mean confidence judgments between objectively urgent (hits and misses) and objectively nonurgent (correct rejections and false alarms) cases (χ2 = 11.13, df = 1, P = .00085) (Fig. 4A). Mean confidence (estimated marginal means) for hits, correct rejections, misses, and false alarms was, respectively, 89.0, 87.7, 84.2, and 81.7. There was no statistically significant interaction between these two factors (χ2 = 1.12, df = 1, P = .29), meaning that higher confidence judgments were reported toward objectively urgent cases relatively to nonurgent cases, independently from correctness.

Figure 4.

Physicians’ metacognitive sensitivity. (A) Boxplots of participants’ mean confidence according to correct (hits and correct rejections, green) and incorrect (misses and false alarms, purple) answers. Each dot represents one participant (N = 52). Hits and false alarms represent cases categorized as urgent by participants while correct rejections and misses were identified as nonurgent. Hits and misses are cases that were objectively urgent according to official guidelines (at least one red flag); correct rejections and false alarms are cases that were objectively nonurgent (no red flag). ^****P < .0001 based on type II Wald chi-square test for linear mixed-effects model fit of mean confidence values. (B) Group-level metacognitive efficiency (H-Mratio = meta-d’/d’) estimated hierarchically (see Methods).

We further computed two indices to formally quantify this metacognitive sensitivity: discrimination ability (difference between confidence in correct versus incorrect answers at the individual level) and meta-d’/d’ (see Methods). First, median participants’ discrimination ability was 4.0 [IQR = 1.6–6.3] at the group level. Metacognitive sensitivity measured as discrimination ability correlated with participants’ accuracy (Pearson correlation, R = 0.29, P = .034) (Fig. 5A). This means that higher objective accuracy made it a bit easier for participants to detect their errors.

Figure 5.

(A) Participants’ metacognitive sensitivity as a function of their accuracy. (B) Participants’ metacognitive bias as a function of their metacognitive sensitivity. Metacognitive sensitivity corresponds to the measure of discrimination ability in both these plots. Each dot represents one participant (N = 52). Lines represent a linear regression fit with a shaded gray area illustrating 95% confidence intervals. Marginal density plots on the top and right margins illustrate the distributions of accuracy, metacognitive bias, and metacognitive sensitivity. R and P respectively indicate Pearson’s correlation coefficient and the statistical significance.

Second, we estimated metacognitive efficiency by computing the standard index of meta-d’/d’ using a hierarchical approach (Fleming and Lau 2014, Fleming 2017). We found an average M-ratio of 0.598 over the group (Fig. 4B) with satisfactory convergence of mean R^ = 1.006. This is slightly higher than what is reported in metaperception studies, typically in the range of 0.55–0.70, and lower than what is typically reported in metamemory studies (semantic, episodic, etc.), in the range of 1.0–1.2 (e.g. Mazancieux et al. 2020, Lund et al. 2025 for recent well-powered studies). This may be related to the generally high accuracy and confidence levels observed at the group level in the present study, leaving highly accurate participants with less range on the scale to adjust their ratings between perceived correct and incorrect responses.

Third, we verified that participants’ accuracy influenced confidence on top of vignette difficulty level using a linear regression model (see Methods). Importantly, if confidence were solely driven by vignette difficulty, operationalized as the amount of evidence present in the “red flags” of the case, we would not expect confidence to also vary as a function of response accuracy. The analysis revealed a significant interaction between response accuracy and number of red flags (χ²(2) = 18.93, P < .001; Supplemental Table 4), indicating that the effect of vignette difficulty on confidence differed depending on whether the response was correct or incorrect. Post hoc estimated marginal means comparisons showed that, for incorrect responses, confidence was significantly lower when one or more red flags were present compared with no red flags (both P < .001). In contrast, for correct responses, confidence did not significantly vary as a function of the number of red flags (all P > .24). Moreover, confidence was significantly higher for correct than for incorrect responses for vignettes containing no red flags (P < .001) and for vignettes containing two or more red flags (P = .0014; Supplemental Table 4). Together, these findings indicate that vignette difficulty impacted confidence but did not fully explain the observed relationship between confidence and accuracy. After accounting for vignette difficulty, confidence remained higher for correct than for incorrect responses.

Finally, we examined the association between metacognitive bias and metacognitive sensitivity since previous reports (outside the clinical context) revealed that they were not independent (Fleming and Lau 2014). As expected, we found a significant correlation between participants’ metacognitive bias and their metacognitive sensitivity (Pearson correlation, R = −0.45, P = .0078) (Fig. 5B). This correlation remains significant when removing three outlier participants (rightmost dots on Fig. 5B) (Pearson correlation, R = −0.33, P = .20). Participants with higher overconfidence had a weaker capacity to discriminate between their own correct and incorrect responses (and had lower accuracy, Fig. 2C).

Interindividual variability

Even though it was not the primary focus of our study, and therefore our statistical power is limited, we examined the influence of participants’ demographic characteristics (gender, seniority level, medical specialty, and research activity) on their d’, type 1 response c criterion, local and global metacognitive bias, and metacognitive sensitivity. To anticipate our findings, overall, we found little to no associations (Supplemental Tables 2 and 3).

At the level of decisions, mean d’ did not significantly vary across medical specialties (P = .072). c-criterion values were also extracted for each participant. Kruskal–Wallis test and subsequent post hoc comparisons revealed that c criterion significantly differed between specialties (P < .001), meaning that neurologists were more inclined to refer urgently compared to emergency physicians and GPs (Supplemental Table 3). Our Bayesian analysis confirmed a strong effect of medical specialty on response c criterion, with very strong Bayesian evidence supporting group differences (BF₁₀ = 40 094).

At the level of metacognition, none of our demographic parameters had any influence on local confidence. As for the global calibration index, it was mostly negative, with a trend of women underestimating more their performance than men (women: median = −11.7 [IQR = −24.2 to −2.5], men: median = −6.7 [IQR = −11.7 to 8.3], Wilcoxon–Mann–Whitney test, W = 220, P = .0498). Importantly, the marginal gender effect observed for global metacognitive bias in the frequentist analysis (P = .0498) was not supported by the Bayesian analysis (BF₁₀ = 0.758). None of the participants’ demographics significantly impacted their metacognitive sensitivity. Consistently, our Bayesian analyses revealed that most other comparisons yielded Bayes factors close to or below 1, indicating either limited evidence for group differences or some support for the null hypothesis (Supplemental Table 3).

Discussion

Our study aimed at bridging the gap between state-of-the-art methods in metacognition research and the real-life context of medical triage decision-making. We took particular care to establish hypotheses and preregister our analysis plan, elaborate ecologically relevant stimuli, and conceive a robust experimental paradigm with carefully chosen parameters. This allowed us to test several hypotheses regarding physicians’ confidence in their triage decisions for patients with acute nontraumatic headaches. This is to our knowledge the first study deciphering metacognitive sensitivity in medical practitioners using signal detection theory and characterizing decision-by-decision fluctuations in local confidence. We found that physicians had overall high accuracy but were slightly overconfident. We also established that physicians had insight into the accuracy of their triage decisions reflected in a degree of metacognitive sensitivity. We found limited variations in self-evaluation of medical decision-making according to gender, medical specialty, or seniority level—though our study was not powered to detect these. Together, our findings carry important practical implications for clinical decision-making.

Many previous studies investigating clinicians’ perception of their own performance rely on computing correlation coefficients between accuracy and confidence, or calculating calibration indices of the difference between average accuracy and average confidence. At least three studies using experimental paradigms similar to the present work already concluded that physicians exhibit overconfidence, as reflected by significantly higher confidence compared to their actual accuracy (Meyer et al. 2013, Silveira et al. 2022), even when a significant positive relationship between accuracy and confidence was observed (Cheung et al. 2018). Consistently, our data replicate this miscalibration of confidence among physicians. Our individual (mis)calibration indices were characterized by an important effect size (mean and median difference between confidence and accuracy around 8 points, on a scale of 50%–100%).

As previously reported (Gigerenzer 1991), an alternative way to probe confidence based on a single retrospective global estimation of the perceived number of correctly solved vignettes—rather than local confidence judgments elicited after each decision—interestingly led to the disappearance of the observed overconfidence at the local level and instead revealed underconfidence at the global level. Dissociations between local and global confidence levels have actually been reported in other domains beyond medical decision-making, indicating that global confidence is unlikely to be merely a straight average of local confidence (e.g. Gigerenzer 1991, Rouault et al. 2019). We speculate that a first possibility is that local confidence is an immediate judgment, made simultaneously or little after the decision is made, without much additional deliberation, whereas the global confidence judgment is made at a longer temporal scale, reflecting an internal estimate about one’s ability or skills to solve the vignettes, more detached from the immediate decision context. A second interesting possibility is metacognitive judgements can be made across a hierarchy of levels, from local confidence in our responses or decisions, to global task confidence, all the way up to self-confidence (for a review, see Seow et al. 2021). Under this framework, the most local confidence judgments may also be prospectively influenced by more global levels such as prior beliefs about one’s abilities with whom one will arrive to tackle a given task, a prediction that has started to receive empirical support (Van Marcke et al. 2024). In fine, what matters is the local decision the physician makes for the patient; however, it is very likely that these local judgements are colored by global prior beliefs about themselves and their abilities, combined with the local evidence presented (e.g. the case and its number of red flags). Maintaining appropriate subjective confidence in each individual decision made under uncertainty remains crucial for physicians, as it may help minimize the risk of suboptimal decisions and adverse events. This observation underscores the value of maintaining both individual and collective deliberation and debriefing time slots in physicians’ schedules to introspect and reflect on their own decision processes.

A critical goal of this study was to disentangle metacognitive bias from metacognitive sensitivity—a key probe into physicians’ self-evaluation abilities. We found evidence that physicians were able to discriminate between correct and incorrect answers using their confidence judgments. To our knowledge, only two studies investigated metacognitive sensitivity so far in the medical context, highlighting physicians’ ability to discriminate between their correct and incorrect answers through their confidence judgments (Naguib et al. 2019, Lakhlifi et al. 2023), which are in line with our results. However, and albeit significant, the magnitude of the difference in reported confidence between correct and incorrect answers is relatively small (3–7 points of percentage). This is potentially due to the overall high levels of accuracy in our study, meaning that participants had little space to express confidence differences on the scale. Nevertheless, these small effect sizes raise questions about the real-life behavioral consequences of variations in subjective confidence. For instance, a previous study suggested that high confidence was related to decreased requests for additional diagnostic tests (Meyer et al. 2013) and the advice-taking literature indicates that confident judges are less likely to seek and utilize advice, whether from colleagues or from algorithms (Kostopoulou 2026). Further research is needed to evaluate the required level of metacognitive sensitivity for physicians to effectively ask for a second opinion from a colleague, seek additional information, or request complementary tests. Moreover, among correct answers, hits elicited higher confidence than correct rejections. This difference might be due to finding a red flag potentially strengthening participants’ confidence, whereas when finding no red flag, it might be difficult to assess the space of all possible absences. Among incorrect answers, misses elicited a higher confidence than false alarms. Attention should be devoted to this potentially problematic observation, considering the cost asymmetry between these two types of error for patients. Indeed, previous work indicates that such referral decisions trade-off costs and benefits to the patient, as well as to the healthcare system or the society, with the former typically prevailing (Kostopoulou et al. 2026); these perceived benefits and harms influenced their decisions.

Our findings indicate that physicians generally adhere to official guidelines, with minor variations in accuracy and confidence across medical specialties. Overconfidence in clinicians can hinder the pursuit of verification or assistance, potentially fostering biases such as confirmation bias (Koriat et al. 1980, Strohmer et al. 1990, Martin 2000, Owen 2008, Miller et al. 2015). However, confidence can be beneficial in high-stakes or emergency scenarios, where a lack of confidence might impede decision-making. Interventions aimed at enhancing metacognition, particularly those focused on reducing overconfidence, should be carefully considered to avoid unintended consequences on performance and physician well-being (Graber 2003, Eva and Norman 2005, Berner and Graber 2008, Johnson and Fowler 2011). Additionally, expressing doubts may also impact credibility and patient–physician relations (Berner and Graber 2008, Tenney et al. 2011).

The observed difference of c criterion among medical specialties suggests that neurologists were more inclined to refer a vignette urgently compared to GPs and EPs. This may stem from neurologists’ exposure to a higher proportion of cases requiring urgent investigation, leading them to minimize false negatives (Zwaan and Hautz 2019). In contrast, GPs and EPs, who often serve as the first point of contact for patients, must balance efficient triage with the risk of over-referral, resulting in a more liberal c-criterion. The rate of presentation of cases in the present survey, with half of the vignettes needing urgent referral, was of course quite different from the typical sample GPs would see in clinical practice and therefore is likely to have affected participants’ base rate expectations, and their accuracy in turn. Understanding these calibration differences is crucial for tailoring training and support systems to enhance the accuracy and reliability of clinical judgments across the healthcare spectrum.

Enhancing metacognition is often proposed to reduce cognitive biases in medical decisions and improve care quality and patient safety (Croskerry et al. 2013, Stark and Fins 2014, O’Sullivan and Schofield 2019, Royce et al. 2019, Zwaan and Hautz 2019). Solutions such as a longitudinal teaching curriculum aimed at medical professionals to identify their biases and promote reflective reasoning, decisional aids, and mnemonic checklists can foster metacognition and critical thinking in medical reasoning (Schraw and Roedel 1994, Mamede et al. 2008, 2010, Reilly et al. 2013, Chew et al. 2016, 2019).

Despite our carefully controlled experimental design, this work has some limitations. A first limitation concerns how representative of real-life cases our vignettes are. Although clinical case vignettes were carefully designed, for our research purpose using signal-detection theory, hard or tricky cases might be overrepresented. Accuracy and confidence can be tied to difficulty (Schraw and Roedel 1994), exemplified by the “hard–easy” effect in which individuals can be underconfident in easy tasks and overconfident in harder ones (Juslin et al. 2000).

A second limitation concerns the ecological relevance of case vignettes to assess medical decision-making. As most studies focusing on confidence in clinical decision-making (Nagendran and Chen 2019), our survey relied on clinical case vignettes, previously reported as valid instruments to investigate physicians’ decision-making (Mohan et al. 2014). Their formatting and framing are widely used for medical students, residents, and physicians’ initial and continuous training and certification. They resemble emails or phone calls in daily practice when a colleague is asking for help in the management of a patient’s case. Despite these elements and drastically increasing the number of vignettes compared to previous studies, it remains possible that our settings created artificial and simplistic contexts prone to suboptimal reasoning (Lejarraga and Hertwig 2021). We cannot rule out that the same content, if provided instead through actors or real-life patients, could lead to better calibrated and more sensitive confidence, if not increased accuracy. Moreover, we did not provide any feedback on decision accuracy or confidence. While many contexts of medical practice also lack immediate feedback, our experimental design does not allow feedback-based learning and might not be fully representative of real-life situations (Lejarraga and Hertwig 2021).

A third limitation concerns the impact of explicitly asking for subjective confidence reports on downstream decisions. Physicians do not receive any training during the medical curriculum in consciously producing and expressing metacognitive judgments of their own performance. We devoted special attention to the framing of our instructions, questions, texts, and scale selection to probe confidence using literature-informed arbitrations and iteratively piloting and refining these aspects (Juslin et al. 2000, Double and Birney 2019b). Nevertheless, variability arises when eliciting subjective, intuitive, and sometimes unconscious feelings at the individual level, and there may be additional interindividual variability in interpreting and reporting on the scale.

A last limitation is that there are reciprocal interactions by which accuracy and confidence may influence each other (Fleming and Dolan 2012). Eliciting confidence judgments during the task sometimes impact accuracy itself, a phenomenon known as confidence reactivity; either enhancing performance by activating participants’ metacognition or diminishing performance by diverting cognitive resources to the metacognitive judgments (Mitchum et al. 2016, Double and Birney 2019a, 2019b). The pathway of production of metacognitive evaluations is not yet fully understood, whether one common and unified metacognitive ability applies to various cognitive processes (Fleming and Dolan 2012, Rouault et al. 2023), or metacognitive judgments arising from the same evidence as decision accuracy (van Den Berg et al. 2016, Zylberberg et al. 2016, Webb et al. 2022). Since accuracy and confidence are related (Morgan et al. 1997, Kruger and Dunning 1999, Kunimoto et al. 2001, Kao et al. 2005, Modirrousta and Fellows 2008, Fleming and Dolan 2012, Jansen et al. 2021), we must be cautious in extending our conclusions to different types of medical decisions, targeted physicians, medical specialties, or to draw practical recommendations. The quality of care is also dependent on many other factors besides the quality of the medical decision itself as studied here at the individual level, including working conditions and available equipment. For instance, a previous study has shown that organizational context influenced referral decisions for lung cancer, with GPs from practices with higher referral positive predictive value being less likely to refer (Kostopoulou et al. 2020).

Future studies relying on similar paradigms will be useful to evaluate metacognitive sensitivity of clinicians facing different types of medical decisions such as diagnostic and treatment choice, and different decision parameters (e.g. signal strength, emotional valence, stakes, uncertainty). This will allow us to better understand the potential factors affecting the production of confidence judgments, such as experience level, sleep deprivation, working conditions, social context, and presence of feedback (Aiyer et al. 2025, Deroy and Bahrami 2026). Prior to any corrective implementations, further research is required to characterize and quantify the behavioral consequences of given levels of confidence on subsequent decision-making, and to increase the number of studies investigating physicians’ confidence in their real-life decisions (Nagendran and Chen 2019).

Author contributions

M.K., R.K., C.L., M.R., and B.R. collaborated to design the experimental paradigm. R.K., N.B., and C.L. elaborated the clinical case vignettes with the help of B.R., J.M., and C.R., and conceived the survey on LimeSurvey and RedCap with the support of the PRISME platform (Paris Brain Institute—ICM). C.L., B.R., J.M., and C.R. worked at the data acquisition by ensuring the survey distribution. C.L., F.-X.L., and B.R. analyzed the collected data, and all authors collectively interpreted them. C.L. and M.R. drafted the article manuscript. B.R. and M.K. revised it. All authors read and approved the manuscript.

Camille Lakhlifi (Conceptualization [lead], Investigation [lead], Methodology [equal], Project administration [lead], Writing—original draft [lead], Writing—review & editing [equal]), Rayan Kouzy (Conceptualization [equal], Methodology [equal], Writing—review & editing [supporting]), François-Xavier Lejeune (Data curation [lead], Formal analysis [lead], Validation [lead], Visualization [lead], Writing—original draft [equal], Writing—review & editing [supporting]), Nicolas Beuzon (Conceptualization [supporting], Methodology [supporting]), Jérôme Mawet (Conceptualization [equal], Methodology [equal], Writing—review & editing [supporting]), Caroline Roos (Conceptualization [equal], Methodology [equal], Writing—review & editing [equal]), Mehdi Khamassi (Conceptualization [equal], Formal analysis [equal], Methodology [equal], Writing—review & editing [equal]), Benjamin Rohaut (Conceptualization [lead], Data curation [equal], Formal analysis [equal], Investigation [equal], Methodology [equal], Supervision [lead], Writing—review & editing [lead]), and Marion Rouault (Conceptualization [equal], Formal analysis [equal], Methodology [equal], Supervision [equal], Writing—review & editing [lead])

Conflict of interest

None declared.

Funding

The funding for this work was provided by the Haute Autorité de Santé (C.L.’s PhD grant), INSERM, and the “Investissements d’avenir” program (ANR-10-I AIHU-06) to Paris Brain Institute—ICM (B.R.’s research annual grant). C.L. is a student from the FIRE PhD program funded by the Bettencourt Schueller Foundation and the EURIP Graduate Program (ANR-17-EURE-0012). M.R.’s work has been supported by La Fondation des Treilles and by a postdoctoral fellowship from the AXA Research Fund.

Patient and public involvement

Patients and the public were not involved in this research project.