Consensus goals in the field of visual metacognition

Abstract

Despite the tangible progress in psychological and cognitive sciences over the last several years, these disciplines still trail other more mature sciences in identifying the most important questions that need to be solved. Reaching such consensus could lead to greater synergy across different laboratories, faster progress, and increased focus on solving important problems rather than pursuing isolated, niche efforts. Here, 26 researchers from the field of visual metacognition reached consensus on four long-term and two medium-term common goals. We describe the process that we followed, the goals themselves, and our plans for accomplishing these goals. If this effort proves successful within the next few years, such consensus-building around common goals could be adopted more widely in psychological science.

Introduction

“The trouble with not having a goal is that you can spend your life running up and down the field and never score.”

Bill Copeland

The need for common goals in science

There is considerable debate among philosophers about what constitutes progress in science (Feller & Stern, 2007). Nevertheless, two broad themes appear in most accounts. First, scientific progress requires the accumulation of solid, agreed-upon empirical knowledge (Bird, 2007). Second, scientific progress requires theories and models that predict and explain the various empirical findings in a field (Guest & Martin, 2021; Muthukrishna & Henrich, 2019; van Rooij & Baggio, 2021). These two components of scientific progress are in constant interplay with each other: new findings lead to refined theories, which in turn motivate the collection of new and different empirical data to test them.

One factor that may accelerate scientific progress is the existence of common goals in a given discipline. Indeed, if most topics in a field are tackled by only one or a few labs, it becomes difficult to build both an agreed-upon empirical knowledge and robust theories. Such difficulties are apparent to various degrees in many subdisciplines of psychological and cognitive science.

Common goals could have transformative effects on research fields. They can lead to greater synergy among research groups and thus faster progress. In addition, spurious findings are more likely to be weeded out when many groups work toward a common goal. An inspiring example within psychology has been the goal of measuring the replicability of psychological science. The goal has rallied hundreds of laboratories and has led to genuine answers in a few short years (Klein et al., 2018; Open Science Collaboration, 2015) and large-scale collaborations such as the Psychological Science Accelerator. It is clear that this progress would not have been made in the absence of a common goal that served to focus the energies of many researchers. Yet, clearly defined common goals remain largely absent in basic experimental psychology.

Potential drawbacks of common goal setting

Although it is easy to identify potential benefits of common goal setting, it is also possible to think of potential drawbacks. Here we discuss several potential disadvantages of such goal setting that mostly relate to adopting an extreme approach where the common goals completely displace the creativity and innovation of individual researchers. We also explore simple measures to mitigate such drawbacks.

Perhaps the most important drawback is the potential of common goals to stymie innovation. Indeed, if individual researchers abandon their interests and only work on a small set of common goals, many important discoveries may not be made. A healthy level of diversity of goals is important for a discipline (Kording et al., 2018), while an obsession with just a few narrow paths can lead to “tunnel vision.” Yet, agreeing on common goals in no way implies that researchers should stop exploring a multitude of research questions and directions. Indeed, we believe that few, if any, researchers would abandon promising leads that fall outside of the common goals. Certainly, none of the current authors plan to do so. Similarly, we doubt that publishers or grant agencies will stop supporting research outside of the common goals and we would certainly discourage them from doing so. In the context of organizations, the existence of a “goals paradox” has been suggested, where both congruence and diversity in organizations’ goals are needed for success in collaboration (Vangen & Huxham, 2012). Similarly, goal setting in science should strive to bring about more congruence but not at the expense of diversity.

A second possible concern is that common goal-setting may overturn standard scientific practices. Indeed, research programs often evolve organically around new theories and empirical findings. If this process were fully replaced by explicitly setting goals that scientists should strive to meet, then the organic evolution of research programs would be disrupted. However, the existence of common goals does not prevent researchers from following new leads as in standard scientific practice. Instead, they can help break tendencies to only seek confirmatory evidence for one’s favorite theories (Yaron et al., 2021) and enable adversarial collaborations where researchers from different camps work together to resolve their differences (Melloni et al., 2021).

A final potential concern is about the meaning of the word “goal” and what is included under it. We do not think that there is one correct answer and common goals for different fields can be defined on many different levels. Here, we adopt a very broad conception of the term “goal” that encompasses both broad and narrow scientific questions and research directions. These goals can include topics already studied extensively as well as completely new avenues of research. It is possible that a narrower conception of the word “goal” would be more beneficial for more established fields, but such a broad definition seems preferable for newer fields such as ours.

Ultimately, assessing the advantages and disadvantages of common goal setting in science requires data. We are unaware of equivalent efforts in other fields and therefore of relevant data that we can use for this assessment. We hope that the current effort will be one critical data point that can inform our understanding of the value of common goal setting in science.

Creating common goals for the field of visual metacognition

Here, 26 researchers from the field of visual metacognition -- a field of study focused on understanding the subjective evaluation and control in visual perception -- organized around the idea of specifying common goals. We start by giving a brief timeline on the process that we followed, then discuss the specific goals that we agreed on, and end with our strategies for follow-up and evaluation.

The idea for coming up with common goals for our field was born in the summer of 2020. We gathered a group of people working on the topic of confidence and metacognition in perception. We sought to assemble a relatively small group that was diverse in terms of career stage, geographical location, and gender. We did not follow a formal methodology and did not have strict criteria for inclusion when assembling the group, so the authors represent one slice rather than a representative sample of researchers from the field.

To construct an initial list of possible goals, each person was encouraged to submit anonymous entries for what they perceived to be the most important goals in the field. We separated these into two categories: long-term goals, which aim to set a direction for the field and are not expected to be resolved for at least the next ten years, and medium-term goals where concrete progress can be expected in the next five years. This process resulted in 26 long-term goals and 39 medium-term goals. The wording of the goals was then standardized, and all goals were anonymously rated by the same group of researchers on several categories including their importance, clarity, likely success, and likelihood of wide adoption. All proposed goals and raw ratings are included as Supplementary material. The goals were then sorted based on the answers to the question “Is this goal among the 2–3 goals that should be adopted by the field?” This process resulted in six highly-rated long-term goals and six highly-rated medium-term goals. Everyone was allowed to “rescue” other goals but nobody did. All of these steps were carried out online over approximately four months.

We then held two 3-hour online workshops, three days apart, where we debated the merits of the top-rated goals from both categories. The first workshop covered the long-term goals; the second workshop covered the medium-term goals. In each case, the pros and cons of each goal were thoroughly discussed and one final round of voting took place. Based on these final ratings, each workshop ended with a decision on the consensus goals from each category. The process resulted in four long-term goals and two medium-term goals. The ratings from these meetings are also available as Supplementary material.

Finally, we discussed the best process for following up on these goals, with the discussion starting during the workshop but continuing over the next several months. Writing the current paper served to (1) formalize each goal, (2) publicly announce the goals to both generate commitment and encourage the involvement of the wider research community, and (3) inform researchers from other fields about our process in case other subfields of psychology want to engage in similar goal-setting. All goals, together with the links between them, are graphically presented in Figure 1.

Figure 1. Consensus long- and medium-term goals.

The arrows indicate how the four long-term goals are related to each of the two medium-term goals. Long-term goal 4 is the only long-term goal that is not directly connected to either of the medium-term goals, though progress on these medium-term goals could have implications for long-term goal 4 too. The arrows are bidirectional to highlight the facts that (1) progress on the medium-term goals automatically results in progress for the long-term goals, and (2) the broader long-term goals have critical subcomponents represented by the medium-term goals.

While we were able to reach a consensus, it should be noted that the process was far from easy. The large number of initially proposed goals demonstrates the existence of a large diversity of topics, approaches, and priorities in the field of visual metacognition (similar diversity exists in related fields such as computational neuroscience; Kording et al., 2018). Zeroing in on only a small minority of goals meant that the great majority of proposed goals were not selected as consensus goals regardless of how strongly the people who proposed them may have felt about them. The two workshops further demonstrated that we did not initially share a common vision for progress in the field. Arriving at a consensus strongly depended on the existence of an abundance of goodwill among the participants and the absence of “warring factions.” We include suggestions on optimizing the process of arriving at shared goals in the Supplementary material.

A very brief introduction to visual metacognition

We define “visual metacognition” broadly as the study of the subjective evaluation and control of one’s own cognitive processes and behavioral responses during visual perceptual tasks (Nelson & Narens, 1990). Most tasks in the field feature simple perceptual judgments (e.g., discriminating between two possible stimuli such as left- and right-tilted Gabor patches, though more complex tasks such as multi-alternative decisions and estimation tasks are also used). This Type-1, object-level judgment is then supplemented by a Type-2, subjective judgment, usually in the form of a confidence rating. The field has its roots in 19th-century psychophysics (Fechner, 1860; Helmholtz, 1856), which often used confidence ratings to infer the perceptual experience of the subject (Peirce & Jastrow, 1884). However, the last decade has seen both a substantial growth and a change of focus to understanding self-evaluation itself rather than simply using it as a tool to understand perception (Fleming et al., 2012; Mamassian, 2016; Rahnev, 2021). The field is rapidly maturing and growing, with many investigators from diverse fields such as computational neuroscience, animal neurophysiology, judgment and decision making, and psychometrics becoming increasingly involved. To make the current paper easier to follow for non-specialists, we provide a glossary of common terms that appear in this paper (Table 1).

Table 1.

Glossary

Term	Definition
Accumulation-to-bound models	A set of models of decision-making that assume an underlying process of accumulation of evidence to a threshold.
Metacognitive bias	An increase or decrease of confidence level despite basic task performance remaining constant.
Metacognitive efficiency	The ability to distinguish between one’s own correct and incorrect responses given a certain level of Type-1 performance
Metacognitive noise	A type of noise that affects confidence ratings but not primary decisions.
Metacognitive sensitivity	The ability to distinguish between one’s own correct and incorrect responses.
Signal detection theory (SDT)	A theory of perceptual decision making used to model choice behavior (often in two-choice tasks) that considers the across-trial variability in internal evidence for each stimulus category.
Type-1 vs. Type-2 decisions	Type-1 decisions are about the primary task, while Type-2 decisions are about the quality of the Type-1 response.
Type-1 vs. Type-2 task performance	Type-1 task performance indicates how well one’s choices predict stimulus identity, whereas Type-2 task performance indicates how well one’s subjective ratings predict one’s accuracy (i.e., metacognitive sensitivity).

Overview of the consensus goals

We agreed on four long-term and two medium-term goals. All six goals are focused on basic science. This fact largely reflects the current composition and priorities in the field but may also suggest the need for more attention towards applied research in the future. All goals should be accessible to most labs in the field as well as to researchers of all career stages. The selected goals represent a mixture of theoretical and technical components. More specifically, long-term goals 3–4 and medium-term goal 2 are largely theoretical, whereas long-term goals 1–2 and medium-term goal 1 have a dual focus on both technical and theoretical developments. No goal is purely technical -- the models, techniques, and manipulations that different goals seek to develop ultimately gain their significance from their role in answering theoretical questions. Finally, some goals are comparatively narrow (e.g., long-term goals 2 and 4), some are quite broad (e.g., long-term goal 1), and one goal (medium-term goal 2) became broad during our discussion as it was made to encompass three different but related initial entries.

It should be appreciated that the great majority of the initially proposed goals were not selected. These goals varied substantially. A post hoc analysis of these goals categorized only seven of them as closely related to the selected goals, and 49 as unrelated or very remotely related to the selected goals. Some of the most common themes among the non-selected goals included the relationship of metacognition and psychopathology (4 goals), the proper measurement of metacognitive ability (4 goals), the relationship between metacognition and consciousness (3 goals), the neural correlates of visual metacognition (3 goals) and modeling visual metacognition (3 goals). This variability demonstrates the diversity of perspectives, objectives, and methodologies in the field, and thus perhaps further underscores the need for common goal setting.

Long-term goals for the field of visual metacognition

We decided to adopt four long-term goals, and have committed to incorporating them into our research programs. We view these goals as setting a direction and do not expect that any of them will be resolved for at least the next ten years and perhaps beyond. For each goal, we explain why it is important, give a brief background on relevant research and methodologies, and put forward our current thoughts on what needs to be done to ultimately achieve that goal.

Long-term goal 1: Develop falsifiable and detailed computational models of visual metacognition

Why is this goal important?

To achieve progress in our understanding of visual metacognition, a key long-term goal is to develop detailed and falsifiable computational models that explain the implementation of visual metacognition. Both cognitive models that focus on behavior and models that explain data from neural recordings are needed. Although such modeling is a worthy goal in and of itself (by allowing, for example, to predict human behavior; Yarkoni & Westfall, 2017), it is also critical for our theoretical understanding of the mechanisms of visual metacognition. A computational model goes beyond a conceptual, verbal description and translates a specific theory into math making it more precise and unambiguous (Guest & Martin, 2021; van Rooij & Baggio, 2021). Moreover, translating verbal theories into computational models often clarifies the hidden assumptions in the theories. Within the context of visual metacognition, computational modeling can clarify which sources of evidence, internal and external, contribute to reported confidence, reveal the extent to which confidence involves normative computations or heuristics, constrain theories regarding the architecture of metacognition, etc. For such modeling to be useful, models must be sufficiently detailed, provide clear falsifiable hypotheses, and fit actual behavioral and neural data well. Given that modeling of visual metacognition is still in its infancy, this long-term goal is necessarily rather broad by encompassing both cognitive and neural models of any task that involves visual metacognition. We expect that as the field matures, it will become easier and more productive to set narrower modeling goals.

Background

Before providing a roadmap for future developments, we first discuss some of the current models of visual metacognition and their limitations and shortcomings. Much of the early work was inspired by signal detection theory or SDT (Green & Swets, 1966). This framework describes how human observers categorize noisy measurements of a signal by placing a criterion in the measurement space. By imposing additional criteria, the same framework can also be extended to explain how human observers can give a graded evaluation of the quality of their decision (Clarke et al., 1959; Galvin et al., 2003; Maniscalco & Lau, 2012). Thus, within this framework visual metacognition is directly related to the strength of the evidence in that observers will be more certain about their choice if the evidence sample lies far from the decision criterion.

An important limitation of SDT is that it does not consider within-trial dynamics, but instead only makes predictions about end-of-trial choices. Therefore, such models cannot easily account for influences of speed-accuracy tradeoffs on confidence or allow for changes of mind within the course of a trial (Resulaj et al., 2009). A natural extension of SDT that does consider within-trial dynamics is a class of models based on the accumulation-to-bound principle. Within such models, choices are thought to reflect the noisy accumulation of evidence until a threshold is reached. To account for visual metacognition, several extensions of these models have been proposed. For example, visual metacognition can be quantified as the degree of evidence extracted from additional post-decisional evidence accumulation following the initial boundary crossing (Pleskac & Busemeyer, 2010), as the difference in magnitude between two accumulators (Vickers, 1979), or as the probability that a choice was correct (Kiani & Shadlen, 2009).

An important distinction in current models is that between single-pathway, dual-pathway, and hierarchical models (Fleming & Daw, 2017; Maniscalco & Lau, 2016). According to single-pathway models, a single source of evidence, corrupted with sensory noise, informs both perceptual choices and metacognitive choices. According to dual-pathway models, perceptual and metacognitive choices reflect information corrupted by independent noise sources. Finally, according to hierarchical models, metacognitive choices are based on the corrupted signal that was used to inform the perceptual choice with additional metacognitive noise applied.

The work ahead

As the brief background above shows, several existing models of decision-making can each be extended to incorporate visual metacognition. Yet, many of these models make very similar predictions. For example, one key characteristic of visual metacognition is that choice accuracy usually monotonically increases as a function of decision confidence (Kepecs & Mainen, 2012). However, this pattern is predicted by virtually all theories of visual metacognition. As such, despite being a key aspect of metacognition, such a pattern does not appear informative to distinguish different models. Therefore, the major challenge ahead will be to find ways that allow us to behaviorally differentiate between models of visual metacognition. Two differentiable models will have certain scenarios where they make divergent predictions about behavior. Thus, in addition to giving a computational description of the model, researchers will also need to inspect the models theoretically or by using simulations to identify these key choice contexts where the models are differentiable (Shekhar & Rahnev, 2021a). Preferably, the models should also emphasize biological plausibility in that each algorithmic step can be represented as a neural process (e.g., population coding). These two elements, falsifiability and biological plausibility, would allow for behavioral and neural tests to narrow down the most likely processes underlying visual metacognition, allowing for consensus-building and a greater ability to report and compare fits to metacognitive behavior across studies.

Long-term goal 2: Develop robust protocols to manipulate one’s metacognition and investigate if such protocols facilitate adaptive performance

Why is this goal important?

This goal relates to two important questions: what is the function of visual metacognition and can visual metacognition be manipulated experimentally. As already mentioned, metacognition plays both monitoring and regulatory roles (Nelson & Narens, 1990). Research on visual metacognition has paid little attention to its specific functions, although it has been suggested that perceptual confidence might guide perceptual learning (Guggenmos et al., 2016), associative learning (Hainguerlot et al., 2018), task prioritization (Aguilar-Lleyda et al., 2020), and moderate sensory evidence accumulation (Balsdon et al., 2020). However, in most studies, visual metacognition has not been directly manipulated leaving the causal role of metacognition in behavior unclear. Developing novel protocols to robustly manipulate metacognition will have great methodological, theoretical, and even clinical significance (Moritz & Woodward, 2007).

Background

Manipulations of metacognitive efficiency

Many studies have reported manipulations that modulated metacognitive efficiency. One group of studies used manipulations related to stress. For example, it has been shown that individual predisposition to stress (i.e., cortisol) reactivity, and the administration of cortisol-like drugs, is associated with reduced metacognitive sensitivity (Reyes et al., 2015, 2020). Similarly, other studies suggested that blocking noradrenergic transmission can improve metacognitive efficiency (Allen et al., 2016), and that meditation training can improve metacognition in memory but not in perception (Baird et al., 2014; but see also Schmidt et al., 2019).

Other studies examined the effects of manipulations of cognitive load or direct stimulation of the prefrontal cortex on metacognitive efficiency. Loading the capacity of working memory systems has been shown to impair metacognitive performance for perceptual decisions (Maniscalco & Lau, 2015; Schmidt et al., 2019; but see Konishi et al., 2020). This effect may reflect the necessary role of neural circuitry involving the dorsolateral prefrontal cortex that is shared among both working memory and metacognition (Feredoes et al., 2011). Relatedly, transcranial magnetic stimulation (TMS) of the dorsolateral prefrontal cortex (Rounis et al., 2010; but see Bor et al., 2017) or anterior prefrontal cortex (Rahnev et al., 2016; Ryals et al., 2016; Shekhar & Rahnev, 2018) have also shown modulations of metacognition.

Other manipulations shown to affect metacognition include experience-dependent training in a visual imagery task (Rademaker & Pearson, 2012), the engagement of visual attention or expectation (Mei et al., 2020; Sherman et al., 2015), and changing the order of Type-1 and Type-2 confidence responses (Wierzchoń et al., 2014). Currently, there is mixed evidence on whether metacognition can be improved using feedback (Carpenter et al., 2019; de Gardelle et al., 2020; Haddara & Rahnev, 2022).

Manipulations of confidence

Several studies have attempted to selectively modulate the overall level of confidence while holding Type-1 performance and/or metacognitive efficiency constant. By causally and selectively modulating confidence, such an approach can be useful for understanding the function that perceptual confidence plays for other aspects of behavior. One popular manipulation is the positive evidence bias, in which the signal and noise components of a visual stimulus are both increased while keeping the signal-to-noise ratio approximately intact (Zylberberg et al., 2012). This paradigm has been used to show that increasing confidence does not facilitate cognitive control (Koizumi et al., 2015) or working memory (Samaha et al., 2016), thus constraining theories on how confidence relates to other higher-order cognitive processes.

However, other work has documented significant effects of confidence on other aspects of behavior. For example, increasing perceptual confidence (independently of accuracy) in a first decision biases evidence accumulation for one’s subsequent decision in favor of the initial choice (Rollwage et al., 2020). Relatedly, selectively boosting confidence increased both the attractive and repulsive serial biases typically observed across trials in visual perception tasks (Samaha et al., 2019). Confidence manipulations have also been shown to influence one’s decisions to seek additional information (Desender et al., 2018). These effects suggest that confidence in a perceptual decision, independent of decision accuracy, modulates how perceptual evidence is used to guide subsequent behavior.

The work ahead

The main challenge ahead is three-fold: validating existing manipulations of metacognitive efficiency and confidence, finding novel ways to manipulate metacognition in a way that produces generalizable effects on cognition and behavior, and developing a sound understanding of when, why, and how these effects occur. Further research is needed to test the effect of different types of feedback signals (e.g., based on the accuracy of confidence judgments) or brain markers of metacognitive skill (e.g., via neurofeedback training; Cortese et al., 2016). Another promising direction is to further develop existing neurostimulation interventions (i.e., based on TMS, transcranial direct current stimulation, or pharmacological interventions) to target the mechanisms of metacognition in a way that produces reliable changes in confidence that impact subsequent behavioral performance. We can expect progress on several of these fronts already in the next five years and have consequently discussed whether the whole goal here should be in the medium-term category. Yet, we felt that the current goal is long-term since it is important to develop multiple manipulations of metacognition, investigate whether each facilitates adaptive performance, and compare the results. This process is likely to take time. Ultimately, this line of work should reveal whether metacognitive interventions can support adaptive behavioral performance across different sensory modalities and cognitive tasks, and whether these interventions are sufficiently strong and long-lasting to allow clinical applications.

Long-term goal 3: Determine the computations underlying confidence in tasks of increasingly higher complexity

Why is this goal important?

In the real world, confidence accompanies a wide variety of decisions and is used not only as a form of self-reflection but also as a way to shape how we plan subsequent actions, learn from past errors, and communicate our decisions to others. Characterizing these processes with tasks of increasingly higher complexity will allow us to broaden our conceptualization of visual metacognition. Important next steps include examining confidence in decisions between more than two alternatives, decisions that unfold over prolonged time scales, and decisions that require actively seeking information (Desender et al., 2018; Rouault et al., 2021). In addition, increased task complexity is necessary for understanding the relationship between confidence and other forms of visual metacognition, such as introspection about task strategy, decision time, and the conscious experience of sensory stimuli (see long-term goal 4).

Background

Confidence has usually been studied by asking people to evaluate their performance on simple two-choice tasks. Typical tasks include deciding whether a stimulus is novel or familiar, comparing the orientation of two visual stimuli, or reporting the net direction of motion of randomly moving dots (Kiani & Shadlen, 2009). Focusing the study of confidence on binary decisions has made it possible to relate confidence to decision accuracy and decision time (Kiani et al., 2014). It has also led to the development of precise computational models of confidence in binary decisions (Maniscalco & Lau, 2016; Shekhar & Rahnev, 2021b; Vickers, 1979), and enabled the study of confidence in non-human animals (Kepecs et al., 2008; Kiani & Shadlen, 2009; Masset et al., 2020).

The study of confidence in simple perceptual decisions has laid solid foundations for expansion to tasks that more closely resemble its formation and use in the real world (Rahnev, 2020). Confidence affects how we plan subsequent actions, which has been studied with tasks that comprise multiple sub-decisions - akin to real-world decisions like preparing a dish or finding a route to a destination. In a task in which two correct decisions were required to obtain a reward, van den Berg et al. (2016) showed that participants adjusted the speed and accuracy of a second decision depending on their confidence in the first. This establishes a role for confidence in regulating the speed-accuracy tradeoff for subsequent decisions, a strategy that maximizes overall reward (Balsdon et al., 2020). The study of tasks in which different sources of information have to be combined to make a decision has shown that confidence is also used to infer the cause of an error. Purcell & Kiani (2016) showed that human participants integrate expected accuracy (or confidence) over multiple decisions to infer when a strategy that was useful in the past is no longer effective, and neural correlates of confidence-guided strategy selection have been found in monkeys (Sarafyazd & Jazayeri, 2019). This line of research highlights how confidence in propositions that span multiple individual decisions (“I’m good at this task”) can be built from confidence in individual decisions (“I made this decision correctly”) (Lee et al., 2021; Mamassian, 2020; Rouault et al., 2019; Zylberberg et al., 2018).

Confidence also affects how we communicate our decisions to others and how we weigh their opinions. Bahrami et al. (2010) showed that two decision-makers facing the same decision can achieve better performance than each one alone if they can exchange their confidence judgments. Confidence and metacognition influence how we judge the intention and expertise of other agents (Pescetelli & Yeung, 2021) and decide whether to seek advice or information before committing to a decision (Rouault et al., 2021). These studies have leveraged what has been learned about confidence from the study of isolated decisions to approach the more complex functions of confidence.

The work ahead

Despite recent efforts, a gap remains between the tasks used to study confidence and the complexity of both the kinds of perceptual decisions and confidence evaluations characteristic of everyday life.

In realistic contexts, percepts are formed by combining multiple cues, often weighted by their reliability (Trommershäuser et al., 2011). It is unclear whether people have metacognitive access to the uncertainty associated with low-level cues or only to the final unified percept (Deroy et al., 2016). The primary task can also have many more than two decision alternatives. Even simple extensions from binary to ternary decisions have shown that, similar to findings in executive function (Collins & Koechlin, 2012), metacognition may be limited to tracking only the best two alternatives (H.-H. Li & Ma, 2020). A related question is whether confidence only encodes a few discrete levels (Lisi et al., 2020; Zhang & Maloney, 2012) or a continuous representation of perceptual evidence (Swets et al., 1961). Paradigms involving visual search (Gajdos, Régner, et al., 2019), tracking moving stimuli (Locke et al., 2020), and active sampling (Rouault et al., 2021) can reveal the complex interplay of different cues to confidence (Boldt et al., 2017). Another aspect is determining which cues contribute to global and prospective confidence estimates (Lee et al., 2021; Mamassian, 2020; Mei et al., 2020; Rouault et al., 2019; Siedlecka et al., 2016), and how they may interact with “local” confidence in a single decision.

Normative models posit that confidence tracks the probability of a decision being correct. However, observers have been found to deviate from optimal computations (Rahnev & Denison, 2018). Relating confidence to other forms of introspection, such as observers reporting on their cognitive strategy, decision-time, or even stimulus visibility, is important for building a comprehensive theory of metacognition. Finally, the development of implicit measures of confidence would be particularly useful for the study of confidence in non-human animals (beyond the use of response times and willingness to wait for a reward; Kepecs et al., 2008; Masset et al., 2020). It has been shown that confidence is reflected in neural markers such as pupil dilation (Allen et al., 2016; Balsdon et al., 2020; Lempert et al., 2015; Urai et al., 2017), and the P300 component (Zakrzewski et al., 2019) and central parietal positivity (Boldt et al., 2019; Herding et al., 2019) obtained from electroencephalographic recordings. Further research is necessary to understand how one or a combination of these measures could be used to assess metacognitive accuracy, and how they are related to the neural computation of confidence.

Specific directions that are especially promising for immediate progress are suggested in medium-term goal 1, which is functionally equivalent to the current long-term goal. In addition, understanding the computations underlying confidence in tasks of increasing complexity will require continuous progress on modeling confidence (see long-term goal 1 and medium-term goal 2) with the ultimate goal that models of metacognition should generalize across paradigms to contribute to a unified framework.

Long-term goal 4: Determine the nature of the relationship between perceptual metacognition and perceptual consciousness

Why is this goal important?

Perceptual metacognition and perceptual consciousness are traditionally seen as closely linked; however, their relationship is not fully understood and varies dramatically across theoretical frameworks. So-called first-order theories of consciousness (e.g., Block, 2007; Lamme, 2000) posit that only recurrent activity in early sensory areas is required for consciousness and that metacognition is a post-perceptual cognitive process with no direct link with phenomenal experience. By contrast, according to higher-order theories (HOT), perceptual consciousness is linked to higher-order reflective processes that represent or monitor first-order contents stemming from sensory responses (Lau & Rosenthal, 2011). However, the meta-level representations and self-reflective processes that are critical for conscious experience in HOT need not be similar to the components of metacognitive confidence (Brown et al., 2019), and, as we will review below, metacognition can be dissociated from perceptual consciousness. The global neuronal workspace model distinguishes components of consciousness based on the global availability of information within cognitive and action systems, and self-monitoring or metacognition (Dehaene, 2014). Corroborating this distinction, a recent paper suggests that the network that subtends such global availability during conscious perception takes a different form according to whether participants are requested to decide on their perception or not (Sergent et al., 2021). However, attempts have been made to explain the role of metacognition within this framework (Shea & Frith, 2019) by suggesting that confidence is a key feature of the representations held in the global workspace, which affords a common currency to integrate information from different sensory systems (de Gardelle & Mamassian, 2014; Faivre et al., 2018) and cognitive processes that may be re-used to guide subsequent behavior and mental function.

Empirical studies often assume a link between metacognition and consciousness, as metacognitive judgments are often used to make inferences about consciousness (e.g., Norman & Price, 2015). However, there is no agreement on whether such measures exhaustively capture all conscious contents and whether they allow for differentiating conscious from unconscious perception (e.g., Seth et al., 2008; Timmermans & Cleeremans, 2015). It has also been proposed that different types of metacognitive assessments measure different phenomena. So-called introspective or first-order judgments (e.g., visibility judgments) are thought to refer directly to one’s visual experience, while second-order judgments (e.g., confidence ratings) refer to the evaluation of one’s perceptual decision accuracy (Sandberg et al., 2011). Looking for dissociations between these two processes sheds light on whether an accurate metacognitive assessment of perceptual performance depends on conscious perception (Jachs et al., 2015) or whether it can indicate the presence of conscious experience that cannot be verbalized and reported (Vandenbroucke et al., 2014).

Understanding the relationship between visual consciousness and metacognition, and pinpointing their common and distinct factors, will help both to better understand the nature and function of each construct and further develop theories in each field. Below we review the existing evidence for dissociations between perceptual consciousness and metacognition, focusing on how metacognitive judgments are made for information that is consciously experienced or not, and then provide an overview of the few studies that have attempted to examine the two phenomena simultaneously.

Background

Several lines of evidence suggest that conscious access may not be needed for the successful deployment of metacognition. For instance, Charles and colleagues (2013) assessed perceptual and metacognitive sensitivity in a number classification task across different levels of stimulus visibility. Their results showed that metacognitive processing of visual targets reported as unseen exceeded chance levels. Jachs et al. (2015) replicated these results and found that perceptual sensitivity strongly depended on visibility, while metacognitive sensitivity did so to a much lower extent. In addition, there is evidence that confidence judgments are diagnostic of visual memory accuracy even when participants display chance-level sensitivity in their first-order recognition judgments (Rosenthal et al., 2016; Scott et al., 2014). Finally, when attentional resources are constrained and participants report not seeing the target stimulus, confidence responses can discriminate between actual misses and correct rejections (Kanai et al., 2010; Meuwese et al., 2014). This dissociation between visibility and metacognition is consistent with there being a lower information threshold to make confidence estimates relative to phenomenological reports of visual experience (Zehetleitner & Rausch, 2013).

Our understanding of perceptual metacognition has mostly improved through the analysis of confidence ratings regarding discrimination tasks. Although discrimination tasks offer several practical advantages to compute metacognitive performance, only detection tasks allow a contrastive analysis of perceptual consciousness whereby the behavioral and neural responses evoked by seen vs. unseen stimuli are compared (Baars, 1997). Therefore, a simultaneous evaluation of perceptual consciousness and metacognition requires the collection of confidence ratings regarding the absence vs. presence of stimuli, which only a few studies have done. This is particularly important given that the neural underpinnings of metacognition for discrimination and detection differ qualitatively (Mazor et al., 2020). Among the studies that examined confidence in detection, an emerging pattern is that metacognitive performance is lower when judging stimulus absence vs. stimulus presence (Kanai et al., 2010; Meuwese et al., 2014), potentially in line with an asymmetric contribution of positive and negative evidence to confidence (Peters et al., 2017; Zylberberg et al., 2012) and/or unequal-variance SDT (Kellij et al., 2021; Mazor et al., 2021; Miyoshi & Lau, 2020). While the interplay between perceptual consciousness and metacognition is abundantly discussed at a theoretical level, empirical evidence bearing on this relationship is much scarcer. This interplay derives naturally from models assuming a common mechanism underlying detection and confidence responses. Recently, such a model was proposed considering a stimulus as consciously detected when a leaky evidence accumulation process reached a threshold and deriving confidence as the distance between the maximum of accumulated evidence and that threshold (Pereira et al., 2021). This latter definition of confidence notably explains how stimulus absence may be monitored and accounts for an asymmetry between positive and negative evidence mentioned above.

The work ahead

Future research needs to provide an account of how phenomenal experience, visibility, and confidence relate to computational models of human vision (Denison et al., 2020), generate and test novel predictions, and ultimately refine existing theories of consciousness. Among the hurdles of the work ahead, we note the need to match the level of performance when addressing the neurocognitive mechanisms supporting perceptual awareness and confidence (Morales et al., 2019), and develop novel paradigms that can concurrently assess both, without them being confounded with cognitive functions that are associated with reporting (e.g., attention, decision making, verbal report, response selection). There have been recent developments of so-called no-report paradigms to study the neural basis of perceptual consciousness while minimizing such confounds (Block, 2019; Tsuchiya et al., 2015) but there are currently no similar no-report paradigms for the concurrent assessment of metacognitive confidence and perceptual consciousness. While the present discussion focused on conscious contents, another line of research should also assess how metacognitive monitoring operates across distinct levels of consciousness or vigilance states.

Medium-term goals for the field of visual metacognition

In addition to the four long-term goals that set a general direction for research, we identified two medium-term goals. These medium-term goals are expected to yield progress within the timeframe of the next five years (i.e., we expect measurable progress by the end of 2026). For each of the two goals, we explain how it relates to the four long-term goals, where immediate progress appears most likely, and what we hope to achieve in the next five years. Unlike in the section on long-term goals, here we do not give extensive background for each goal since this background has already been covered in the related long-term goals.

Medium-term goal 1: Expand beyond confidence in two-choice tasks and develop models of confidence for such tasks

Why is this goal important and how does it relate to the long-term goals above?

This goal is strongly related to long-term goal 3, so much so that it can be considered a medium-term version of long-term goal 3. The present medium-term goal is also related to long-term goal 1, which outlined several models (signal detection theory, accumulation-to-bound models, single vs dual channel models) that are currently popular in explaining visual metacognition. Notably, most of these models are designed and tested in experiments where observers rate their confidence in a two-choice task. As a consequence, it is unclear whether the current models of visual metacognition can account for decision confidence in more complex cases, such as tasks with multiple alternatives or continuous judgments. Developing models that can explain visual metacognition in more complex tasks is of critical importance, not just because such models will have more ecological validity (and therefore will have wider explanatory power), but also because they may allow evaluation of the assumptions in current models in more challenging contexts. This will help researchers achieve long-term goal 1 by widening the scope of our models to a broader range of decision scenarios and providing more opportunities for divergence in model predictions. In addition, any progress on this goal will also contribute to the more general long-term goal 3.

The work ahead

Current models of visual metacognition, which mostly apply to two-choice tasks, are inherently limited in scope but it is not necessarily clear how they should be extended. Below, we present what we consider to be the four most promising directions where immediate progress can be made.

First, the most straightforward extension of current models would be to expand them from two-choice tasks to n-alternative choice tasks. For example, accumulation-to-bound models that can account for behavior in n-alternative choice tasks have been described (Ratcliff & Starns, 2013). Similarly, Li & Ma (2020) have proposed several plausible models for n-alternative decisions. Thus, a clear target for future developments would be to continue with these previous attempts and/or expand existing models, testing each model’s validity in capturing behavior in n-alternative choice tasks in a wide range of perceptual tasks.

A second more ambitious target is to expand current models so that they can explain confidence when estimating a continuous quantity, such as the confidence one has that the orientation of a stimulus was correctly reproduced. In such cases, asking the observer to report the probability they were correct seems unsatisfactory as the observer will rarely be perfect in their report. Instead, their confidence should reflect the degree of error in the estimate. Several studies have already collected data on tasks that involve estimating a continuous quantity (e.g., Graf et al., 2005; Yallak & Balcı, 2021; Yoo et al., 2018) and several such datasets are available in the Confidence Database (Rahnev et al., 2020). The next steps would involve building models of visual metacognition that explain confidence ratings in such tasks.

Third, one step further would be to examine visual metacognition of ongoing perception. Due to the subjective nature of metacognitive reports, visual metacognition is usually queried jointly with or shortly after a choice. However, this does not imply that observers have no metacognitive experiences during the choice formation itself. In fact, there is some evidence that metacognition emerges online during choice formation (Dotan et al., 2018) and that it even controls the termination of the choice formation process (Balsdon et al., 2020). Such online expressions of metacognition pose a challenge for current models of visual metacognition, which usually describe metacognition as a (post-decision) read-out of the decision process. Thus, a clear target for future work will be to develop protocols that allow for robust online measurement of metacognition, and models that can explain such reports.

Finally, the fourth target for model developments is to explain perception-action interactions. There is increasing interest in examining visual behavior in dynamic scenarios where perception and action are both at play (Bonnen et al., 2015; Huk et al., 2018), which increases decision complexity. Thus, confidence can emerge as part of perception and action loops, such as reaching a series of targets or tracking just one (Locke et al., 2020). Rather than a simple button press, the response can be highly varied or of a continuous nature. As we mentioned previously, capturing temporal dynamics and expressing confidence for continuous estimates are highly limited in the currently available frameworks.

What will achieving the goal look like?

Achieving this goal would mean that researchers interested in visual metacognition are no longer limited by the task they use. Nowadays, a lot of interesting research that is done in the field of visual metacognition falls outside the scope of existing models, especially if the experiment does not consider a simple two-choice task. Concrete progress would be having identified one or more robust paradigms for decision scenarios beyond the standard two-choice version (e.g., n-alternative choice, continuous estimates, ongoing perception, or perceptuomotor interactions), with one or more accompanying computational models of metacognition. Ideally, these computational models would be more general and adaptable to different decision scenarios, including the standard two-choice tasks favored today. This may be achieved by generalizing existing metacognitive models (i.e., SDT or accumulation-to-bound models) or with other decision-making frameworks (e.g., Bayesian frameworks).

Medium-term goal 2: Determine the computations underlying confidence and what factors influence these computations

Why is this goal important and how does it relate to the long-term goals above?

This goal is a combination of three separate goals (see Supplementary material) that were similar enough to warrant combining them. The goal, therefore, has three separate components, which are to understand: (1) what are the computations underlying confidence, (2) how do different sources of uncertainty influence metacognitive processes (regardless of whether the first-order decision is affected too), and (3) what processes (if any) selectively affect confidence while leaving the first-order decision unperturbed. The three components are interrelated such that progress on one of them is likely to translate into progress on the rest. Overall, the goal here is to understand the computations behind confidence, especially via the effects of experimental manipulations. As such, this goal will advance long-term goals 1–3 (related to developing models, developing manipulations, and determining confidence computations for complex tasks). While less directly related, progress on this goal may also have implications for long-term goal 4 (uncovering the relationship between metacognition and consciousness). This goal is therefore central to the field of visual metacognition and is likely to have wide-ranging implications.

The work ahead

There are several aspects of this goal where substantial progress can be made in the next five years. We discuss what we perceive as the most important directions related to understanding confidence computation and identifying the factors that influence this computation.

What are the computations underlying confidence?

This question is often phrased as “What does confidence reflect?” There are several competing hypotheses in the field with relatively little agreement at present. One common view is the Bayesian notion that confidence reflects the posterior probability of being correct (Aitchison et al., 2015; Fleming & Daw, 2017; Meyniel et al., 2015; Pouget et al., 2016). In other words, people base their confidence ratings on the probability that their response is correct even if this computation is noisy or biased. Another common view grounded in signal detection theory and accumulation-to-bound models is that confidence directly reflects signal strength (Bang et al., 2019; Green & Swets, 1966; Maniscalco & Lau, 2016). Here, confidence is derived from an abstract evidence axis without computing the probability that a response would be correct. Other alternatives include the view that confidence reflects the evidence for the chosen option while ignoring the evidence for all unchosen alternatives (Koizumi et al., 2015; Maniscalco et al., 2016; Peters et al., 2017; Samaha et al., 2016; Zylberberg et al., 2012) or that it reflects the difference in posterior probability of the two most likely alternatives (H.-H. Li & Ma, 2020). Several papers have compared directly two or more of these alternatives (Adler & Ma, 2018; Aitchison et al., 2015; H.-H. Li & Ma, 2020) but a consensus is yet to emerge. We believe that substantial progress is possible in the next five years on distinguishing between these possibilities.

What factors influence the confidence computation and how?

There is vibrant literature on the factors that influence confidence computation (reviewed in Shekhar & Rahnev, 2021a). Here we briefly mention the factors that have received the greatest attention and then discuss what we perceive as the most promising next steps.

Perhaps the most widely studied factors that affect confidence computations are stimulus variability and attention. However, the exact effects of each of these factors remain controversial. For example, increased variability has been found to lead both to higher-than-expected and lower-than-expected confidence (Bertana et al., 2021; Boldt et al., 2017; de Gardelle & Mamassian, 2015; Spence et al., 2016, 2018; Zylberberg et al., 2014, 2016). Similarly, different manipulations of attention have been found to either increase or decrease confidence and visibility ratings (Denison et al., 2018; Kurtz et al., 2017; Rahnev et al., 2011, 2012; Recht et al., 2019; Wilimzig et al., 2008; Zizlsperger et al., 2012). These studies have used different designs, manipulations, and sometimes collected different metacognitive measurements (e.g., confidence vs. visibility), making it difficult to pinpoint the reasons for the divergent results. Many other factors have been investigated by relatively fewer studies. For example, confidence is influenced by the confidence on previous trials (Aguilar-Lleyda et al., 2021; Rahnev et al., 2015), motor preparation and execution (Fleming et al., 2015; Gajdos, Fleming, et al., 2019), visual field location (M. K. Li et al., 2018; Solovey et al., 2015), the strength of decision-congruent evidence (Koizumi et al., 2015; Maniscalco et al., 2016; Peters et al., 2017; Samaha et al., 2016; Zylberberg et al., 2012), stimulus visibility (Rausch et al., 2018), and decision time (Kiani et al., 2014).

Despite the large number of factors already identified, many other factors that affect the confidence computation are likely yet to be discovered. A mechanistic understanding of confidence would strongly benefit (and perhaps require) the identification of all critical factors, and therefore the search should continue. The next five years can be expected to add more to the list above. Nevertheless, it also appears that the field has reached a point where more emphasis needs to be given on firmly establishing the knowledge that (we think) we have already gained. For example, few of the studies cited above have been independently replicated and there has not been much consideration of the effect sizes for each of the factors influencing confidence. Therefore, in the next five years, more attention should be paid to replicating existing effects and clarifying the effect size of each.

What will achieving the goal look like?

It is not reasonable to think that five years from now we will know the precise computations underlying confidence and all the ways it is influenced. However, it is reasonable to expect a growing emphasis on empirically adjudicating between different proposals of what confidence reflects, perhaps with an emerging consensus at least for simple two-choice experimental designs. Similarly, it is reasonable to expect the emergence of high-powered replication attempts of the different factors that influence confidence. We will consider the goal “achieved” if both of these expectations are met or at least measurable progress has been made. Such progress will have a large effect as it will ensure that the field is on a sure footing and well-positioned to build cumulative knowledge.

Final thoughts and next steps

Having described the four long-term and two medium-term consensus goals, we end with a short section where we discuss what we learned, as well as our plans for tracking and assessing progress towards achieving the goals listed here.

Thoughts on the process and results of goal setting

One of the greatest difficulties we encountered was with formulating clear and precise evaluation criteria for each goal. Indeed, currently, there is substantial latitude left for each goal. Naturally, given the generality of the long-term goals, deciding on evaluation criteria for each has been particularly challenging, though we have tried hard to establish specific evaluation criteria for the two medium-term goals. We think that difficulties with establishing concrete and rigorous evaluation criteria are likely unavoidable, especially for a relatively new field such as visual metacognition. Time will tell whether goal setting in psychological science is worthwhile only in the context of precise landmarks and evaluation criteria, or if it can have value even if such landmarks and evaluation criteria are less well defined.

Notably, the issues of replicability, estimation of effect sizes, and the use of appropriate sample sizes were only explicitly discussed in medium-term goal 2. This perhaps reflects a perception among the authors that replicability of findings in the field is likely to be relatively high, though there have been relatively few replication studies thus far to formally test this impression. Nevertheless, given the ongoing replication crisis in psychology and related disciplines (Open Science Collaboration, 2015), it may be important to pay more attention to these issues going forward.

Finally, it should be noted that we did not discuss “truths” in the field. In other words, we did not discuss which previous findings within the field are established beyond reasonable doubt and which are not. Such efforts are likely to be fruitful (e.g., see the paper on “benchmarks” in working memory by Oberauer et al., 2018) and may also be worth undertaking.

Tracking and assessing progress

We expect that formalizing these consensus goals will catalyze progress in the field, foster collaboration, and increase the chance of solving the most important problems in the field. Nevertheless, we recognize that formalizing these goals may have a limited influence without a system for tracking and assessing the progress made. It has been argued that progress in science is achieved only when a community of scientists is willing and able to hold each other accountable for the quality of their work (Ravetz, 1971). At the same time, any formal system of evaluation of individual papers or findings is likely to be inflexible and runs the risk of simply reflecting the opinions of authority figures. Any system of tracking and assessing progress should not be overly onerous (i.e., should not require an exorbitant amount of time and resources to maintain), or else it will likely be quickly abandoned.

Based on these considerations, we have decided to institute several mechanisms to help us track and assess progress towards the long- and medium-term goals that we set. First, we have created a Slack channel intended to allow for informal conversations on issues related to each goal. We invite everyone who has an interest in any of these goals to subscribe and actively participate in the ongoing discussions (link to join: bit.ly/3wsPoyl). Second, papers relevant to each long- and medium-term goal will be tracked using an online community-powered spreadsheet (bit.ly/3CJvmCA). We encourage everyone publishing relevant papers to add their papers to this spreadsheet. To obtain help with either the Slack channel or the spreadsheet, one can email visual.meta.goals@gmail.com. Third, we plan to organize a regular meeting or conference specifically for the field of visual metacognition. Fourth, we intend to write a follow-up paper in approximately five years that will assess progress towards both the long- and medium-term goals. Finally, we encourage new papers to explicitly state which of these long- and medium-term goals their findings are relevant to. This practice would be especially important for null results. Such explicit references will make future reviews and meta-analyses on the topics related to these goals substantially easier and more accurate.

Conclusion

Scientific progress requires the accumulation of agreed-upon empirical knowledge and robust theories. We believe that common goals can accelerate such progress by ensuring both a reliable body of empirical findings and the development of theories that explain existing data and make new predictions. Here 26 researchers from the field of visual metacognition agreed on such consensus goals. We identified four long-term and two medium-term goals, as well as a process for tracking and assessing progress. Only time will tell how this effort will impact our field. We hope that the formulation of these goals will enable researchers from across the field to focus our energies, increase the quality of our research, ensure that we build solid cumulative knowledge in our field, and foster more collaboration. At the very least, it should be a useful experiment that provides insight into the forces that drive science and can stir it into states of higher or lower impact. If this effort proves successful, consensus goal setting can become a model for many fields of psychological science and beyond.