Skip to main content
PLOS One logoLink to PLOS One
. 2023 Feb 16;18(2):e0278617. doi: 10.1371/journal.pone.0278617

Does decision confidence reflect effort?

Nobuhiro Hagura 1,2,*, Jamal Esmaily 3,4, Bahador Bahrami 3,5,6
Editor: Gennady S Cymbalyuk7
PMCID: PMC9934378  PMID: 36795716

Abstract

Goal directed behaviour requires transformation of sensory input to decision, and then to output action. How the sensory input is accumulated to form the decision has been extensively studied, however, the influence of output action on decision making has been largely dismissed. Although the recent emerging view postulates the reciprocal interaction between action and decision, still little is known about how the parameters of an action modulates the decision. In this study, we focused on the physical effort which necessarily entails with action. We tested if the physical effort during the deliberation period of the perceptual decision, not the effort required after deciding a particular option, can impact on the process of forming the decision. Here, we set up an experimental situation where investing effort is necessary for the initiation of the task, but importantly, is orthogonal to success in task performance. The study was pre-registered to test the hypothesis that the increased effort will decrease the metacognitive accuracy of decision, without affecting the decision accuracy. Participants judged the direction of a random-dot motion stimuli, while holding and maintaining the position of a robotic manipulandum with their right hand. In the key experimental condition, the manipulandum produced force to move away from its position, requiring the participants to resist the force while accumulating the sensory evidence for the decision. The decision was reported by a key-press using the left-hand. We found no evidence that such incidental (i.e., non-instrumental) effort may influence the subsequent decision process and most importantly decision confidence. The possible reason for this result and the future direction of the research are discussed.

Introduction

Traditionally, perceptual decision making and motor control have been studied as separate domains. This tradition has benefited researchers by simplifying assumptions that regard action as a mere output channel for reporting the higher-order decisions. However, recent developments at the theoretical and empirical level have started to demonstrate that parameters of action, which are not directly related to the sensory input per se, can influence the perceptual decision itself [1]. For example, two studies [2, 3] showed that when the response for one of the two options in a perceptual decision is made physically more effortful, participants tend to avoid the decision requiring more effort, without any change in their sensitivity at the task. Another study [4] showed that participants became more confident when a perceptual decision required more physical effort to express.

Effort can be classified as instrumental to a given task when it is necessary for better performing that task. Conversely, effort may be needed to perform the task and yet be irrelevant to the chances of success. This latter type of effort may be classified as incidental effort [5]. For example, running faster is instrumental to winning a race but thinking hard about a difficult math problem while running the same race is incidental to the race outcome. Thus, instrumental effort is directly linked to the improvement of the decision, such as in the situation where the amount of effort is proportional to the amount of evidence one can acquire. In contrast, incidental effort does not contribute to the quality of the decision per se.

A key common feature of the studies mentioned above [1–5] is that they all employed incidental effort. The amount of effort expenditure did not benefit for improving the decision accuracy. Thus, their counterintuitive observation was to show that the effort influenced the decision process even though it was irrelevant to the task, thus indicating the tight interdependency of action and perceptual decisions.

In those studies, examining the influence of incidental physical effort on the particular decision the participant has (presumably already) made meant that physical effort was required after the decision. However, the post-decision period is not the only period in which the decision and the action can interact [6]. In real life, as well as under laboratory conditions, there are situations where a decision is required while performing an effortful action. Imagine a monkey hanging from a branch with one hand and using the other hand to pick a fruit deliberating which fruit to pick. Here, the effort (i.e. keeping herself from the branch) is required during the deliberation period of the decision-making, during which perceptual evidence is being accumulated to reach the decision. Importantly, the effort is necessary to stay in the task, but does not directly change the quality of the decision evidence, thus, is incidental. The effect of such physical effort exerted during the pre-decision period on decision making is not well understood. In this study, we examined whether the physical effort required during the deliberation period of the decision making influenced the subsequent decision and the confidence for the decision.

Several different lines of research on the role effort can be employed to formulate contrasting hypotheses for the expected results. In a recent review, Inzlicht, Shenhav and Olivola [7] have done a wonderful job of describing what they called the effort paradox. On the one hand, prominent models in economics and cognitive sciences hold that effort is aversive. Animals (including humans) prefer to do as little as possible for a given reward. On the other hand, numerous studies spanning all the way from pigeons and rodents to humans, have shown that all of these various animals value effort, in and of itself. People go mountain climbing, run 7 marathons in 1 week, and spend hundreds of hours learning a musical instrument as a hobby. While few would contribute to the cost of a ticket for you to join a charity cocktail party, it is much easier to raise money for joining a charity triathlon. To the standard models of economics that seek to minimise effort this does not make sense: people who love you should be happier if you enjoyed yourself in the cocktail party. It is therefore not surprising that several contradictory hypotheses can be formulated when considering the impact of effort on decision making.

The common currency [8–13] hypothesis proposes we try to maintain the balance between expected accuracy and cost of evidence accumulation. This cost consists of time and effort. In this view, optimal decisions are the ones that achieve a reasonable level of accuracy at an acceptable cost. Increasing the cost, then, should reduce the bar for what is the acceptable level of accuracy. This would predict that under higher effort, participants would adjust evidence accumulation to collect less evidence, leading to lower accuracy, faster reaction times and lower confidence under high effort. Here, the hypothesis assumes that the brain treats the duration of effort investment during the deliberation as instrumental, and sacrifices accuracy over effort.

The second hypothesis is derived from the ideas of 17th century Dutch philosopher Baruch (Benedictus) Spinoza. In his book of Ethics [14] Spinoza proposed that individuals are motivated to be the adequate cause of their states. A consequence of this kind of motivation is that the power of actions and the satisfaction driven from the choices that those actions execute are inextricably intertwined. Spinoza’s idea of the association between power and joy is reflected brilliantly in the social psychological research on the role of self-determination and control in self efficacy [15–17]. Recent findings in neuroeconomics have shown that having a choice, in and of itself, is rewarding [18, 19]. This account is perhaps most directly supported by the evidence from a recent study by Turner et al [4] who showed that participants expressed higher confidence when deciding required more physical effort. In this view, exerting effort would have a metacognitive, evaluative impact on the decision process that does not affect the accuracy and reaction time but the evaluation of the decision. This hypothesis proposes that evidence accumulated under high effort would be evaluated as more valuable or of better quality because it has cost the participants more and the participants has obtained the evidence through effort and control. Thus, higher effort would impact neither accuracy nor reaction time per se but increase decision confidence. In this hypothesis, the brain ignores the direct benefit of the effort to the task, thus treating it as incidental effort, but using the invested effort as evidence for postdictive evaluation of the task performance.

Finally, an interference hypothesis is also plausible and would predict that higher effort would specifically interfere with the relationship between confidence and accuracy. Fleming and colleagues [20] examined the impact of transcranial magnetic stimulation in interfering with the motor execution of perceptual decisions. They found that, within a narrow and carefully designed range of parameters, one can find a TMS stimulation protocol that leaves perceptual decision accuracy, reaction times and average confidence intact but, critically, disturb metacognitive sensitivity, which is the delicate balance between reported confidence and probability of correct responses. The interference hypothesis driven from these findings, would predict lower metacognitive sensitivity under high (vs low) effort. Apart from using effort (rather than TMS), an important difference between our experimental design and those of Fleming and colleagues is that our manipulation takes place during the perceptual evidence accumulation stage whereas in their work, TMS was applied after the perceptual information had been delivered to the participants. Here, the prediction is that the incidental effort distracts the evaluation of one’s task performance.

To test these hypotheses, our participants judged the direction (leftward or rightward) of a dynamic random dot motion stimulus [21, 22] whose difficulty (i.e. coherence) level varied randomly across trials. The decision and the subjective confidence of the decision was reported by a key-press using their left-hand. While viewing the dots motion, participants held a handle with their right-hand. The handle produced force to move-away from its initial position, and the participants had to maintain its position by resisting the force (Fig 1). We designed this setup to examine how an ongoing isometric physical effort may interact with the deliberation process of decision making, independent of the decision itself and possible outcomes.

Fig 1. Schematic drawing of the experimental setup.

Fig 1

Participants held a robotic handle underneath the screen. In each trial, a random-dot motion stimulus was presented in the centre of the screen. Throughout the trial, they were required to maintain the handle position, indicated as a cursor position on the screen, in the starting position. In the force-condition trials, a force was applied to the handle towards the direction away from the participant (i.e. the arrow in the drawing).

Since we did not have any hypothesis regarding the direction of the effect, a pilot experiment was performed using a models sample of participants. Based on the exploratory analysis performed on the pilot data, the present study is pre-registered to test the following specific predictions (https://osf.io/tykcg). First, we predict that the metacognitive accuracy [23], the measure of how accurately the confidence is related to the decision accuracy, will decrease when the required effort increases. Second, we predict that the physical effort during the deliberation period will not modulate the reaction time nor the decision accuracy of the task.

Method

Participants

A total of 20 volunteers (12 females, mean age: 23.7, s.d.: 4.2 (21–39)) participated in the study. Sample size was determined using the result of the pilot experiment (see below). All were right-handed with normal or corrected-to-normal vision, with no prior history of psychiatric problems or medication. Experiments were undertaken with the understanding and written consent of each participant in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki), and with approval of the National Institute of Communications Technology (NICT) ethical committee (B210042202). Participants were recruited through local advertisements used in the Centre of Neural Networks (CiNet), and were paid 3000 JPY (approximately 10 Euros per hour) for participation. No adverse events occurred during the experiment.

Estimation of the sample size

Data of 14 participants from a pilot experiment was analysed for the sample size estimation (see below for the detail of the pilot experiment). The aim of the pilot experiment is to perform an exploratory analysis to generate a hypothesis about the direction of the effect, as well as estimating the sample size to verify the effect in the main experiment.

In this experiment, the metacognitive accuracy for the random-dot motion decision became lower when maintaining the handle position against the force compared to when there was no force applied to the handle (effect size = 0.76). Based on this result, we estimated the required sample size using the power analysis (G-power), targeting for alpha = 0.5 and power = 0.8. Taking into account the estimation noise, we pre-set the sample size to 20.

General setup

Participants were seated comfortably in front of a screen placed horizontally in front of them, which prevented the direct vision of their hands (Fig 1). The visual stimulus was presented on the screen using a projector placed above the screen. The viewing distance was 50cm. To maintain the viewing distance, the upper trunk was constrained by a harness attached to the chair.

During the experiment, participants were asked to hold the handle of the manipulandum with their right-hand (PHANToM Premium 1.5 HF, SensAble Technologies, Woburn MA, USA). The handle position was displayed as a white cursor (a circle, 6 mm in diameter) on a black background on a horizontal screen located above their hand. The movement of the handle was constrained to a virtual horizontal plane (10 cm below the screen) that was implemented by a simulated spring (1.0 kN/m) and dumper (0.1 N/ms-1).

In the centre of the screen, a random-dot motion stimulus was presented [22] (Fig 1). In a 7 deg diameter circular aperture, dots were presented at a density of 3.5 dot/deg2. The speed of the dots was 10 deg/s. For each trial, either 3.2%, 6.4%, 12.8%, 25.6%, or 51.2% of the dots moved coherently to the left or the right. All other dots moved in a random direction, picked for each dot separately between 0 and 360 deg.

Both the visual stimulus and the robotic manipulandum were controlled by an in-house software program developed using C++.

Study design and the procedure

The study was conducted as 5 (sensory evidence level) x 2 (physical effort level) within-subject factorial design. For the factor of sensory evidence, random-dot motion with 5 different levels of motion coherence (3.2% 6.4% 12.8% 25.6% 51.2%) was prepared. The direction of motion was either leftward or rightward. For the factor of physical effort level, either the handle was pushed away from the body with the force of 6N (force condition) or the force to the handle was absent (no-force condition).

To start each trial, the participant held a handle of a robotic manipulandum with their right hand in the starting position. The trials started 500 msec after setting the handle position. In the force condition trials, a force was applied to the robotic handle towards the direction moving away from the participant. Participants were asked to maintain the handle in the starting position throughout the task, meaning that the participant had to apply an equal force (6N) in the opposite direction. Therefore, if the participants did not produce 6N toward the direction of one’s body to resist the manipulandum movement, the participants’ hand would be passively moved towards the direction of the force from the starting position, or the manipulandum would slip away from the participants’ hand (and collide to the wall). In the no-force condition trials, the force was absent and the participant could effortlessly hold the handle in the starting position. Thus, in this setup, the amount of positional deviation of the manipulandum from the starting position would directly indicate the failure to produce the required amount of force.

In both trial types, once the participant had maintained the handle position for 2000 msec after the start of the trial, then a random-dot motion stimulus was displayed on the screen. Motion coherence varied from trial to trial. Participants were asked to decide about the net direction of the dot-motion and indicate their decision, as fast and accurately as possible by pressing one of the two keys on the keyboard with their left hand. Visual stimulus and the force were removed immediately after the decision (key-press), or if the decision was not made within 2000ms. Then, participants reported the confidence level of the decision on a scale of 0–6. Reaction time, decision accuracy and confidence were our key dependent variables. Kinematic data of the handle position during the task were also measured and recorded. Each participant went through 24 trials of each combination of the condition (i.e., 2 motion directions x 5 coherence x 2 effort levels). One block consisted of 60 trials and there were 8 blocks. The order of the trials was pseudo-randomized within each block of each participant.

Pilot experiment

An earlier version of the experiment was treated as a pilot experiment, in order to estimate the sample size (see above). In this pilot experiment, 15 participants (5 females, mean age = 22.4) volunteered, but 1 was excluded from the analysis, due to the high proportion of non-responding trials. Three different force levels (0N, 2.5N, and 5N) were set, with 200 trials for each force level. The results of this pilot experiment are available in S1 Fig in S1 File.

Data analysis

Prior to the parametric analysis, we confirmed that the data in each condition did not violate the normality assumption using the Kolmogorov-Smirnov test (p>0.05 for all cases). For the main analysis, Generalised linear mixture model (GLMM) was applied to the reaction time, decision accuracy and the confidence data. For all of the GLMM models applied here, each participant was considered as a random intercept. Metacognitive accuracy was calculated as in the previous literature [23]. In short, metacognitive accuracy is quantified by the interrelationship of the confidence rating and the accuracy of visual judgments using the type II receiver operating characteristic (ROC) curve, which characterized the probability of being correct for a given level of confidence [20, 24, 25]. To construct the ROC curve, p (confidence = i | correct) and p (confidence = i | incorrect) were calculated for all i, and were then transformed into cumulative probabilities and plotted against each other. The area underlying the ROC curve quantified the metacognitive accuracy. Metacognitive accuracy was compared between different force conditions using paired t-test. We also calculated the metacognitive accuracy for each low (3.2 and 6.4%) and high (12.8, 25.6 and 51.2%) coherent motions, to check if there is any difference in the effort linked modulation pattern for different uncertainty levels.

Inclusion and exclusion criteria as well as policy for dealing with missing data were pre-registered at OSF https://osf.io/tykcg. In detail, any trial with the decision taking more than 2 seconds, and any trial with the handle movement exceeding over 5cm was excluded from the analysis. Any participant’s data with the proportion of the excluded trials exceeding 15% of the whole trial was discarded from the analysis.

Prediction

In this study, we test our pre-registered hypothesis. First, we predict that the Metacognitive accuracy will be affected by the physical effort delivered during the deliberation period of the decision-making. Metacognitive accuracy will be reduced by physical effort, compared to when no physical effort is required. Second, we predict that the preceding effort will not influence the objective decision accuracy, or the time it takes for the decision (reaction time).

Results

Data exclusion

On average, 3.4% of the trials in the non-force condition and 6.7% of the trials in the force condition were excluded from the analysis. None of the participants exceeded the pre-defined exclusion criterion, therefore, were not excluded from the analysis.

Kinematic data

Since participants were asked to maintain the position of the handle during the task, large deviation of handle position from the starting position would indicate the failure to produce the required amount of force. By analyzing the kinematic data, we confirmed that participants were indeed producing force in the force condition compared to the non-force condition.

In 98.5% of all of the trials, deviation was in the range within 50mm, which was within the pre-defined inclusion criterion (i.e. 1.5% of the data was excluded from the further analysis for this reason). For the rest of the trials, the amount of deviation was at the negligible level. During the force trials, the deviation was on average 17.5mm (s.d. 10.6mm) towards the direction of the force, and 8.9mm (s.d. 6.5mm) towards the right. For non-force trials, it was -2.0 mm and 1.3 mm, respectively.

We also calculated the within-trial fluctuation of the handle position (i.e. variance of the handle movement), which should be proportionally scaled by the amount of force output due to the signal dependent noise [26]. Indeed, the variance was significantly higher for the force condition compared to the non-force condition (paired-t test; t(19) = 2.56, p = 0.019). Taken together, these data indicate that participants were indeed producing a significant amount of force during the force condition compared to the non-force condition (S2 Fig in S1 File).

Decision accuracy, decision confidence and the reaction time

Decision accuracy (Fig 2a), confidence data (Fig 2b) and reaction times (Fig 2c) were compared between two different force conditions under different uncertainty levels. First, all three variables were modulated by the difficulty of decision (i.e. motion coherence levels) (p<0.001, GLMM, see S1 Table in S1 File for details of the statistical values). As expected, this shows that the participant’s response is manipulated by the different coherence level (i.e. difficulty level) of the visual stimulus.

Fig 2. Comparison of behavioural data in different force conditions.

Fig 2

a: Accuracy of participants did not change by force conditions (red = force 0N and blue = force 6N). The curve is the weibull cumulative distribution fitted on the averaged data points of each condition (dots). b: Same as a but for confidence. c: Same as a but for RT. The difference here is significant. Error bars indicate the standard error of means (s.e.m) across all trials.

Second, decision accuracy and the confidence ratings did not differ between the two force conditions (p> 0.1, GLMM (S1 Table in S1 File)), which followed our pre-registered prediction. However, the reaction time became slightly faster for the force condition compared to the non-force condition (average RT; 1028ms for the force condition, 1047 for the non-force condition, p<0.001, GLMM (S1 Table in S1 File)). Note that, since faster reaction time during the force trials did not change the accuracy. This implies that our participants did not use the effort instrumentally, for example to prolong stimulus presentation actively in order to sample more evidence before responding.

Metacognitive accuracy

The metacognitive accuracy [20], i.e. how the confidence is correctly associated with the accuracy of decision, was calculated and compared between different force levels. We found no significant difference in metacognitive accuracy between different conditions (t(19) = 0.22, p = 0.83; Fig 3). Thus, this result did not support our pre-registered prediction. Furthermore, we also confirmed that such non-significant result holds even when the metacognitive accuracy was calculated independently for high and low motion coherence levels (S3 Fig in S1 File).

Fig 3. Metacognitive accuracy based on force conditions.

Fig 3

a: ROC curve of subjects in 0N (red) and 6N (blue) conditions. b: The area under the curve of ROC. Each dot and line represent different participant. Error bars indicate the standard error of means (s.e.m) across participants for both the hit rates and false alarm rates.

Discussion

In this study, we tested if exerting physical effort during the deliberation period of the perceptual decision-making influences the subsequent decision process. We proposed three different hypotheses that made contrasting predictions about this question. Based on a pilot experiment, in the pre-registration we predicted that the physical effort will reduce metacognitive accuracy, without affecting decision accuracy, reaction time or confidence. However, the hypothesis was not supported by the data. In our experimental setup, we found no evidence that the physical effort during the deliberation period could influence the subsequent metacognitive process. We report this negative result together with the pre-registered methods to help avoid file-drawer problems.

One reason for the discrepancy between the pilot and the main study results may be the few differences in the design elements that we changed in the main study (see S1 File). In the pilot experiment, we examined three force levels (0N, 2.5N, and 5N), whereas in the main experiment, there were two force levels (0N and 6N). Accordingly, the number of trials per condition also differed between two experiments. These choices were made to increase the power and sensitivity of the experimental paradigm and there is no principled reason to believe that the choices in the main study were problematic. In any case, if such trivial differences were indeed the cause of the failure to replicate the pilot experiment, this would indicate that the effect of physical effort on metacognition is not robust, at least in the given setup of the experiment.

As with the nature of any negative result, our findings do not necessarily discredit the hypothesis. Physical effort during the deliberation period may or may not influence the perceptual decision in general. We examine several issues worth considering for future studies. First, the pragmatic choices made for experimental design may be responsible for the negative results. The amount of force could have been too low to induce a sufficient amount of effort to influence the participants’ decision. In the current setup, we had to make sure that the amount of force is well within the range which the participants could muster and maintain holding the handle. This was because the slip of the handle from the participants’ hand could cause the handle to crash into the surrounding devices. This restriction of force level may have made the result unstable. Stronger force level may be necessary to overcome this problem, if using passive resistance to the force as a way to manipulate the effort.

Another point to consider is that the force level we used did not affect the accuracy, indicating that the effort level remained non-instrumental to the task. Increasing the force may change this balance of instrumental vs. incidental in our task, which is an interesting question to pursue in the future studies.

Another pragmatic issue was the way the force was produced in the experiment may have made a difference. We chose passive resistance to the force to manipulate the physical effort, because this allowed us to precisely control the required force level. However, an alternative way would have been to make the participants perform an approach movement, actively producing the force to a set level. One may argue that active effort may impact information seeking more directly, thus, making the effort more instrumental. Moreover, there could have been an effector dependency of effort on decision making. Previous studies have manipulated the effort on effector directly involved in the decision, such as by adding physical perturbation to that effector [2, 3] or by electrically/chemically perturbing the neural substrate controlling that effector (eye-movement) [6, 27]. However, in our study, the effector that resisted the force (right-hand) and the effector expressing the decision by the button press (left-hand) differed. Even though the force was terminated after the decision, the relationship between the decision and the force was indirect. Such effector dependency may have played a role in our experiment.

Having discussed the potential reasons for interpreting our results as false negatives, it is also important to discuss the plausible interpretation of them as true negatives too. The three main hypotheses that we raised in the introduction have been summarized in Table 1 together with their predictions for the 4 outcome measures of accuracy, reaction time, confidence and metacognitive precision. Our pilot data most closely favored hypothesis 3 [20] i.e., that the effort influences the metacognitive precision without affecting the perceptual decision itself. Indeed, in our main experiment, physical force did not change the parameters of perceptual decisions per se.

Table 1. An overview of effort hypotheses based on different behavioural features.

In Table 1 ↓ shows decrease of the behavioural feature with respect to increasing the effort, ↑ shows increase and × shows no change;—indicates no prediction.

Hypothesis/Behavioural prediction Accuracy Reaction time Confidence Metacognitive precision
Hypothesis 1 ↓ ↓ ↓ -
Hypothesis 2 × × ↑ -
Hypothesis 3 × × × ↓
Our Data (Pilot) × × × ↓
Our Data (Replication) × ↓ × ×

Surprisingly, we also found a significant decrease in RTs under the high effort condition without any change in the metacognitive sensitivity, choice accuracy or confidence. This isolated decrease in reaction time without any impact on accuracy and confidence points to an intriguing possibility. Under zero force condition, this finding implies, participants were not doing their optimal best but were, so to speak, complacent. When pushed by the force requirement, they stepped up and performed faster while at the same level of accuracy and confidence. Note that faster reaction times meant participants collected less perceptual evidence. This finding is not easy to square with previous works on speed-accuracy tradeoff (see [28–30]). More recent studies that have argued for a relationship between confidence and reaction time are similarly troubled by this finding. For example, Kiani and colleagues [31] showed that artificially prolonging RT, for example by inserting a delay between perceptual stimulus presentation and choice, decreases confidence. Taking their findings at face value, one would have predicted that in our experiment, faster reaction times under high force should have increased confidence, which is not borne by the data. Thus, we believe that our findings point to either a different mechanism not captured in our hypotheses, or more likely, a mixture of different mechanisms. An interesting avenue for future research is to explore how paradigms like ours could help find ways to persuade and motivate people to step up and do better.

Whether force could have acted as a motivating factor raises another important possible reason why our effort manipulation did not succeed as predicted. In a series of innovative experiments addressing the effort paradox (see Introduction), Clay and colleagues [32], gave their participants a challenging training task (i.e., a working memory test) requiring substantial cognitive effort. In the treatment condition, participants were rewarded proportionally to how much effort they put into the training task (and not by success or failure in doing the memory task). In the control condition, participants received a fixed reward. After this initial training, both groups completed a second, transfer task (math puzzles) for a fixed reward. The results showed that the participants in the treatment group were later more eager (vs. control) to exert mental effort: the treatment group voluntarily chose more difficult problems to solve. Effort-contingent rewards during training led to demand seeking in the transfer task. This result shows that in the transfer task, where effort is certainly not instrumental, the previous experience of effort-reward contingency can affect behavior. These findings raise interesting questions. For example, on a general level, are the findings specific to cognitive, mental effort or do they generalise to physical effort too? With respect to our experiment, our participants received a fixed reward compensation that was independent of their performance. Therefore, our experiment may be considered similar to Clay and colleagues’ control condition. Future experiments that examine the role of effort in decision making would do well to consider and perhaps include the role of reward explicitly in their experimental design, and to investigate the interaction between instrumental and incidental effort.

Supporting information

S1 File

(PDF)

Acknowledgments

Authors are grateful to Drs. Masaya Hirashima and Kisho Ogasa for their help in setting up the experiment, and to Mari Koshimizu for helping the data collection.

Data Availability

Content of the pre-registration is available at https://osf.io/tykcg. The associated data is accessible at https://osf.io/be4u5/.

Funding Statement

NH is supported by Japan Society for the Promotion of Science (Kakenhi: 20H00107, 21H00314) and Japan Science and Technology Agency (ERATO: JPMJER1801). BB and JE are supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (819040 - acronym: rid-O). BB is supported by the NOMIS foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1.Cisek P. and Kalaska J. F., ‘Neural Mechanisms for Interacting with a World Full of Action Choices’, Annu. Rev. Neurosci., vol. 33, no. 1, pp. 269–298, Jun. 2010, doi: 10.1146/annurev.neuro.051508.135409 [DOI] [PubMed] [Google Scholar]
  • 2.Hagura N., Haggard P., and Diedrichsen J., ‘Perceptual decisions are biased by the cost to act’, eLife, vol. 6, p. e18422, Feb. 2017, doi: 10.7554/eLife.18422 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Marcos E., Cos I., Girard B., and Verschure P. F. M. J., ‘Motor Cost Influences Perceptual Decisions’, PLOS ONE, vol. 10, no. 12, p. e0144841, Dec. 2015, doi: 10.1371/journal.pone.0144841 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Turner W., Angdias R., Feuerriegel D., Chong T. T.-J., Hester R., and Bode S., ‘Perceptual decision confidence is sensitive to forgone physical effort expenditure’, Cognition, vol. 207, p. 104525, Feb. 2021, doi: 10.1016/j.cognition.2020.104525 [DOI] [PubMed] [Google Scholar]
  • 5.Lee D. G. and Daunizeau J., ‘Trading mental effort for confidence in the metacognitive control of value-based decision-making’, eLife, vol. 10, p. e63282, Apr. 2021, doi: 10.7554/eLife.63282 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Gold J. I. and Shadlen M. N., ‘Representation of a perceptual decision in developing oculomotor commands’, Nature, vol. 404, no. 6776, pp. 390–394, Mar. 2000, doi: 10.1038/35006062 [DOI] [PubMed] [Google Scholar]
  • 7.Inzlicht M., Shenhav A., and Olivola C. Y., ‘The Effort Paradox: Effort Is Both Costly and Valued’, Trends Cogn. Sci., vol. 22, no. 4, pp. 337–349, Apr. 2018, doi: 10.1016/j.tics.2018.01.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Levy D. J. and Glimcher P. W., ‘The root of all value: a neural common currency for choice’, Curr. Opin. Neurobiol., vol. 22, no. 6, pp. 1027–1038, Dec. 2012, doi: 10.1016/j.conb.2012.06.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Prevost C., Pessiglione M., Metereau E., Clery-Melin M.-L., and Dreher J.-C., ‘Separate Valuation Subsystems for Delay and Effort Decision Costs’, J. Neurosci., vol. 30, no. 42, pp. 14080–14090, Oct. 2010, doi: 10.1523/JNEUROSCI.2752-10.2010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Aridan N., Malecek N. J., Poldrack R. A., and Schonberg T., ‘Neural correlates of effort-based valuation with prospective choices’, NeuroImage, vol. 185, pp. 446–454, Jan. 2019, doi: 10.1016/j.neuroimage.2018.10.051 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Klein-Flugge M. C., Kennerley S. W., Friston K., and Bestmann S., ‘Neural Signatures of Value Comparison in Human Cingulate Cortex during Decisions Requiring an Effort-Reward Trade-off’, J. Neurosci., vol. 36, no. 39, pp. 10002–10015, Sep. 2016, doi: 10.1523/JNEUROSCI.0292-16.2016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Klein-Flügge M. C., Kennerley S. W., Saraiva A. C., Penny W. D., and Bestmann S., ‘Behavioral Modeling of Human Choices Reveals Dissociable Effects of Physical Effort and Temporal Delay on Reward Devaluation’, PLOS Comput. Biol., vol. 11, no. 3, p. e1004116, Mar. 2015, doi: 10.1371/journal.pcbi.1004116 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Shadmehr R., Huang H. J., and Ahmed A. A., ‘A Representation of Effort in Decision-Making and Motor Control’, Curr. Biol., vol. 26, no. 14, pp. 1929–1934, Jul. 2016, doi: 10.1016/j.cub.2016.05.065 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.de Spinoza B., Curley E. M., Hampshire, Stuart, Ethics. 1996. [Google Scholar]
  • 15.Bandura A., Caprara G. V., Barbaranelli C., Gerbino M., and Pastorelli C., ‘Role of Affective Self-Regulatory Efficacy in Diverse Spheres of Psychosocial Functioning’, Child Dev., vol. 74, no. 3, pp. 769–782, May 2003, doi: 10.1111/1467-8624.00567 [DOI] [PubMed] [Google Scholar]
  • 16.Ryan R. M. and Deci E. L., ‘Self-Regulation and the Problem of Human Autonomy: Does Psychology Need Choice, Self-Determination, and Will?’, J. Pers., vol. 74, no. 6, pp. 1557–1586, Dec. 2006, doi: 10.1111/j.1467-6494.2006.00420.x [DOI] [PubMed] [Google Scholar]
  • 17.Shapiro D. H. Jr., Schwartz C. E., and Astin J. A., ‘Controlling ourselves, controlling our world: Psychology’s role in understanding positive and negative consequences of seeking and gaining control.’, Am. Psychol., vol. 51, no. 12, pp. 1213–1230, 1996, doi: 10.1037/0003-066X.51.12.1213 [DOI] [PubMed] [Google Scholar]
  • 18.Leotti L. A. and Delgado M. R., ‘The Value of Exercising Control Over Monetary Gains and Losses’, Psychol. Sci., vol. 25, no. 2, pp. 596–604, Feb. 2014, doi: 10.1177/0956797613514589 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Leotti L. A. and Delgado M. R., ‘The Inherent Reward of Choice’, Psychol. Sci., vol. 22, no. 10, pp. 1310–1318, Oct. 2011, doi: 10.1177/0956797611417005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Fleming S. M., Maniscalco B., Ko Y., Amendi N., Ro T., and Lau H., ‘Action-Specific Disruption of Perceptual Confidence’, Psychol. Sci., vol. 26, no. 1, pp. 89–98, Jan. 2015, doi: 10.1177/0956797614557697 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Shadlen M. N. and Newsome W. T., ‘Neural Basis of a Perceptual Decision in the Parietal Cortex (Area LIP) of the Rhesus Monkey’, J. Neurophysiol., vol. 86, no. 4, pp. 1916–1936, Oct. 2001, doi: 10.1152/jn.2001.86.4.1916 [DOI] [PubMed] [Google Scholar]
  • 22.Britten K., Shadlen M., Newsome W., and Movshon J., ‘The analysis of visual motion: a comparison of neuronal and psychophysical performance’, J. Neurosci., vol. 12, no. 12, pp. 4745–4765, Dec. 1992, doi: 10.1523/JNEUROSCI.12-12-04745.1992 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Song C., Kanai R., Fleming S. M., Weil R. S., Schwarzkopf D. S., and Rees G., ‘Relating inter-individual differences in metacognitive performance on different perceptual tasks’, Conscious. Cogn., vol. 20, no. 4, pp. 1787–1792, Dec. 2011, doi: 10.1016/j.concog.2010.12.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Galvin S. J., Podd J. V., Drga V., and Whitmore J., ‘Type 2 tasks in the theory of signal detectability: Discrimination between correct and incorrect decisions’, Psychon. Bull. Rev., vol. 10, no. 4, pp. 843–876, Dec. 2003, doi: 10.3758/bf03196546 [DOI] [PubMed] [Google Scholar]
  • 25.Kornbrot DE., ‘Signal detection theory, the approach of choice: model-based and distribution-free measures and evaluation’, Percept Psychophys., vol. 68, pp. 393–414, 2006, doi: 10.3758/bf03193685 [DOI] [PubMed] [Google Scholar]
  • 26.Harris C. M. and Wolpert D. M., ‘Signal-dependent noise determines motor planning’, Nature, vol. 394, no. 6695, pp. 780–784, Aug. 1998, doi: 10.1038/29528 [DOI] [PubMed] [Google Scholar]
  • 27.Jun E. J. et al., ‘Causal role for the primate superior colliculus in the computation of evidence for perceptual decisions’, Nat. Neurosci., vol. 24, no. 8, pp. 1121–1131, Aug. 2021, doi: 10.1038/s41593-021-00878-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Wald A. ‘Sequential Tests of Statistical Hypotheses.’, The Annals of Mathematical Statistics, vol. 16, no. 2, pp. 117–86, 1945. [Google Scholar]
  • 29.Stone M. ‘Models for choice-reaction time.’ Psychometrika 25, pp. 251–260 1960. doi: 10.1007/BF02289729 [DOI] [Google Scholar]
  • 30.Heitz RP. ‘The speed-accuracy tradeoff: history, physiology, methodology, and behavior.’ Front Neurosci. vol. 8, no.150. Jun. 2014, doi: 10.3389/fnins.2014.00150 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Kiani R, Corthell L, Shadlen MN. Choice certainty is informed by both evidence and decision time. Neuron. vol.84, no.8, pp.1329–1342. Dec. 2014, doi: 10.1016/j.neuron.2014.12.015 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Clay G., Mlynski C., Korb F. M., Goschke T., and Job V., ‘Rewarding cognitive effort increases the intrinsic value of mental labor’, Proc. Natl. Acad. Sci., vol. 119, no. 5, p. e2111785119, Feb. 2022, doi: 10.1073/pnas.2111785119 [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision Letter 0

Gennady S Cymbalyuk

19 Jul 2022

PONE-D-22-12208Does decision confidence reflect effort?PLOS ONE

Dear Dr. Hagura,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please, revise the manuscript to clarify the validity of the experimental paradigm and to address the reviewers' concerns about  statistical analysis. Please, revise to disambiguate the roles of 

two different types of effort : (i) effort that is instrumental to achieving the current goal, versus (ii) effort that is incidental to the task and their associated predictions/interpretations. 

Please submit your revised manuscript by Sep 02 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Gennady S. Cymbalyuk, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This paper describes the results of a pre-registered study on the impact of physical effort on decision deliberation. Authors report analyses of decision accuracy, response time, confidence, and metacognitive sensitivity (i.e. the similarity between decision accuracy and confidence). In contrast to authors prior assumptions, exerting effort while pondering decisions (here: standard perceptual decisions about the motion of moving dots) makes no difference for these variables, except response time (see below).

In brief, I have no critical concern with the study as it stands. Yes, reported results are negative results, but the authors’ conclusions are aligned with their data. Importantly, I see no major methodological flaw. Provided PLOS ONE accepts to publish negative results on pre-registered studies (which I think is important), then my conclusion is that this paper may be published almost as is.

Now, I think authors may improve the quality of their Introduction and Discussion sections. This is because it seems to me that they confuse two different types of effort (when exerted while performing some task): (i) effort that is instrumental to achieving the current goal, versus (ii) effort that is incidental to the task. For example, thinking hard to solve a reasoning problem or running fast to win a race are instrumental efforts that increase the likelihood of achieving the goal. In contrast, thinking hard while running a race or holding a hand grip while pondering a decision, are efforts that are incidental to the task. This distinction matters for understanding and/or predicting the effect of effort onto e.g., decision making (and decision-related variables). For example, one may expect that increasing instrumental effort may eventually increase decision accuracy and confidence (Lee and Daunizeau, 2021), whereas increasing incidental effort may decrease accuracy and confidence (e.g. because it may consume attentional or other resources that would be instrumental to the task). One may have different predictions, but I would still argue that one needs to distinguish instrumental and incidental types of effort. The issue here, is that authors’ reasoning seems to confuse these two types of effort, both when disclosing their hypotheses (Introduction) and when discussing their results. In my view, this renders their argument problematic at times. My advice is to systematically check and modify the Introduction and Discussion sections to disambiguate these two notions and their associated predictions/interpretations.

Jean Daunizeau.

Lee, D.G., and Daunizeau, J. (2021). Trading mental effort for confidence in the metacognitive control of value-based decision-making. ELife 10, e63282. https://doi.org/10.7554/eLife.63282.

Reviewer #2: In this study, Hagura et al., investigate the influence that making a physical effort during decision deliberation might have in decision accuracy and confidence. The interaction between decisions and actions is critical to fully understand decision making. However, I am not convinced that the experimental paradigm presented is enough to make any claim. Additional analysis and control are missing.

Main concerns:

- The study builds on the results of a pilot experiment in which metacognitive evaluation of perceptual decisions differed between force conditions. Supplementary Fig. 1 shows this data, but it seems that the test used to compare between the 3 force conditions was a t-test. When using a t-test to compare more than 2 groups Type I errors (ie. reject null hypothesis when it is true) accumulate. This could be the reason for finding significance in the metacognitive judgements in the pilot experiment but not in the main one. The appropriate tests to perform in this case are, for instance, an ANOVA or Kruskal-Wallis. A normality test should be performed to select the most appropriate one.

- It is not clear whether the strength needed to hold the joystick is effortful enough. In other words, there is no control on whether the motor cost associated with force and non-force conditions is significantly different. Only the mean and sd of the joystick’s deviation for each force condition is reported, but this does not ensure that the motor cost was of significance for subjects and this is critical to the experiment and the results. Although, this is stated in the discussion, without a control for that the results could be specific to the set-up, which would make them of low interest.

- Besides referencing a previous study, the ROC analysis of metacognitive reports should be explained in the text. Moreover, it might be useful to have the ROC analysis performed for each motion coherence separately. This might give some insight on whether there are differences, most likely, for low motion coherence conditions.

- Normality tests should be performed in all groups of data to select the most appropriate statistical test in each case.

- A link to the data is provided but it is not yet possible to access the data.

Minor comments:

- Age interval is missing.

- Some reference, for instance, Song et al., 2011 are referred but not listed.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jean Daunizeau

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2023 Feb 16;18(2):e0278617. doi: 10.1371/journal.pone.0278617.r002

Author response to Decision Letter 0


7 Sep 2022

Reviewer #1

Comment1: This paper describes the results of a pre-registered study on the impact of physical effort on decision deliberation. Authors report analyses of decision accuracy, response time, confidence, and metacognitive sensitivity (i.e. the similarity between decision

Accuracy and confidence). In contrast to authors prior assumptions, exerting effort while pondering decisions (here: standard perceptual decisions about the motion of moving dots) makes no difference for these variables, except response time (see below).

In brief, I have no critical concern with the study as it stands. Yes, reported results are negative results, but the authors’ conclusions are aligned with their data. Importantly, I see no major methodological flaw. Provided PLOS ONE accepts to publish negative results on pre-registered studies (which I think is important), then my conclusion is that this paper may be published almost as is.

Reply1: We appreciate the reviewer’s comment, acknowledging the value of reporting the negative result as an outcome of a pre-registered study.

Comment2: Now, I think authors may improve the quality of their Introduction and Discussion sections. This is because it seems to me that they confuse two different types of effort (when exerted while performing some task):

(i) effort that is instrumental to achieving the current goal, versus

(ii) effort that is incidental to the task.

For example, thinking hard to solve a reasoning problem or running fast to win a race are instrumental efforts that increase the likelihood of achieving the goal. In contrast, thinking hard while running a race or holding a hand grip while pondering a decision, are efforts that are incidental to the task. This distinction matters for understanding and/or predicting the effect of effort onto e.g., decision making (and decision-related variables).

For example, one may expect that increasing instrumental effort may eventually increase decision accuracy and confidence (Lee and Daunizeau, 2021), whereas increasing incidental effort may decrease accuracy and confidence (e.g. because it may consume attentional or other resources that would be instrumental to the task). One may have different predictions, but I would still argue that one needs to distinguish instrumental and incidental types of effort. The issue here, is that authors’ reasoning seems to confuse these two types of effort, both

When disclosing their hypotheses (Introduction) and when discussing their results. In my view, this renders their argument problematic at times. My advice is to systematically check and modify the Introduction and Discussion sections to disambiguate these two notions and their associated predictions/interpretations.

Reply2: We appreciate the reviewer for pointing this out. We agree that the two different types of effort, instrumental and incidental, have not been well distinguished in the manuscript. In the revised manuscript, we have set this distinction clearer.

We agree with the reviewer that, when the amount of effort is directly linked to the improvement of task performance (i.e. instrumental effort), it is straightforward to predict that higher effort expenditure will positively impact on the task performance and should lead to higher confidence. We now clarify that our key question relates to the unknown and interesting question; whether the task irrelevant, incidental effort can impact on the accuracy and the confidence of the decision. One example of such interactions is found in studies (also cited in our previously cited introduction) showing that (incidental) fluency can increase confidence.

Another possibility is the one pointed out by the reviewer: incidental effort may distract the participant from the task. The attentional account also makes additional predictions for higher reaction time and lower accuracy under high incidental effort, which are clearly and empirically testable. Previous studies have shown that task irrelevant incidental effort indeed interacts with the ongoing task performance, not by just simply distracting the task. For example, Hagura et al., (2017) showed that the participants tend to avoid the option associated with higher (incidental) force exertion when performing a perceptual decision task, without any change in the sensitivity of their task performance. Similarly, Turner et al. (2020) demonstrated that the amount of exerted force level, which is announced to the participants after having made a particular perceptual decision, can paradoxically increase the confidence for the same decision decision.

Going one step further from these studies, our aim in the current study was to test how the incidental effort required to commit to the task can influence the task performance and the task confidence. Commitment is instrumental for initiating the task, but not instrumental for strength of perceptual evidence and thereby not for task performance. Indeed, faster reaction time observed during the force condition was not reflected on the accuracy measure, indicating that the effort used in our experiment was non-instrumental, thus incidental, to the primary task.

In the revised introduction (lines 58-70) and the discussion section (lines 342-345, 371-386) we have now included new paragraphs to unpack this issue.

 

Reviewer #2:

Comment 1: In this study, Hagura et al., investigate the influence that making a physical effort during decision deliberation might have in decision accuracy and confidence. The interaction between decisions and actions is critical to fully understand decision making. However, I am not convinced that the experimental paradigm presented is enough to make any claim. Additional analysis and control are missing.

Reply1: We appreciate the detailed and constructive comments of the reviewer.

Comment 2: The study builds on the results of a pilot experiment in which metacognitive evaluation of perceptual decisions differed between force conditions. Supplementary Fig. 1 shows this data, but it seems that the test used to compare between the 3 force conditions was a t-test. When using a t-test to compare more than 2 groups Type I errors (ie. Reject null hypothesis when it is true) accumulate. This could be the reason for finding significance in the metacognitive judgements in the pilot experiment but not in the main one. The appropriate tests to perform in this case are, for instance, an ANOVA or Kruskal-Wallis. A normality test should be performed to select the most appropriate one.

Reply2: We thank the reviewer for the critical evaluation of the pilot data statistics. We have now checked the normality of the data using Kolmogorov-Smirnov test. Data in all of the conditions did not violate the normality assumptions (p=0.138, p=0.2, p=0.2, for force 0N, 2.5N and 5N respectively). We also applied the requested ANOVA to the data (F(2, 13) = 2.21, p = 0.12).

We agree with the reviewer that the analysis of the pilot experiment was done with a small sample size and relaxed a priori hypothesis about the direction of the effect. The purpose of the pilot experiment and data analysis was to explore the space of possible and likely hypotheses and narrow down this space with the discovery made by the pilot, which would eventually be formally tested in the main pre-registered experiment by setting the appropriate sample size for the predicted effect. Therefore, the analysis of the pilot experiment was exploratory in nature, aiming to maximise the sensitivity by relaxing the specificity. Given this aim, we believe that using paired t-test comparing the prominent two conditions in the pilot experiment was justified.

For this reason, we believe that describing the result of the ANOVA in the manuscript will not benefit the paper but instead cause some confusion (Please note that we have made the full data set, including the pilot data, available for readers). In the revised manuscript, we have now further specified the aim of the pilot experiment in the Method section (lines 170-172).

Comment 3: It is not clear whether the strength needed to hold the joystick is effortful enough. In other words, there is no control on whether the motor cost associated with force and non-force conditions is significantly different. Only the mean and sd of the joystick’s deviation for each force condition is reported, but this does not ensure that the motor cost was of significance for subjects and this is critical to the experiment and the results. Although, this is stated in the discussion, without a control for that the results could be specific to the set-up, which would make them of low interest.

Reply 3: We thank the reviewer for pointing this out, “whether the motor cost associated with force and non-force conditions is significantly different”. We should have explained this more clearly.

First, the participants were required to maintain the position of the manipulandum at the home position to initiate the task, and during the task. In the force condition, the manipulandum was programmed to continuously produce 6N force in the direction away from the participant. Therefore, if the participants did not produce 6N toward the direction of one’s body to resist the manipulandum movement, the participants’ hand would be passively moved towards the direction of the force from the home position, or the manipulandum would slip away from the participants’ hand (and collide with the wall). So, without applying and maintaining 6N force, the trial would not start in the first place.

In the no-force condition, the manipulandum did not produce any force. The participants did not have to resist any force. Therefore, in this setup, the amount of positional deviation of the manipulandum from the home position would directly indicate the failure to produce the required amount of force. The fact that the participants successfully maintained manipulandum position in the majority of trials (98.5%) indicates that the participants were indeed producing the force during the force condition.

Second, when the within-trial fluctuation of the manipulandum position during the task was compared between the force and no-force conditions, fluctuations were significantly more pronounced under the force condition than the no-force condition (p=0.019) (Figure S2). Since the variance of muscular activity is proportional to the size of its output (Signal Dependent Noise; Harris and Wolpert, 1997), this demonstrates that the participants were indeed producing more muscle activity (i.e. producing more effort) during the force condition.

Finally, in the force-condition, the reaction time generally decreased compared to the no-force condition, indicating that the force indeed gave impact to the participants’ behaviour, which made them to respond quickly to avoid the effort.

The evidence that (1) the manipulandum stayed at around the required position during the task even the force was applied, (2) within trial fluctuation of the manipulandum deviation was more prominent in the force condition, (3) participants tend to respond faster to avoid the force, clearly indicates that the motor cost was significantly higher in the force condition compared to the no-force condition.

Now, we have clarified this point in the revised manuscript. Please see lines 271-286, together with the new Figures S2 presented in the Supplementary materials.

Comment 4: Besides referencing a previous study, the ROC analysis of metacognitive reports should be explained in the text. Moreover, it might be useful to have the ROC analysis performed for each motion coherence separately. This might give some insight on whether there are differences, most likely, for low motion coherence conditions.

Reply 4: Now, the detail of the ROC value calculation has been added to the manuscript. Please see line 241-250 of the text.

Also, as the reviewer suggested, we have calculated the ROC analysis for different coherence levels. Note that ROC value calculation requires enough of both correct and incorrect trials, thus, for the high-coherence trials with extremely low probability of having incorrect trials, the value cannot be calculated. Therefore, instead of calculating for each coherence level, we divided the data into two groups, high coherence (12.8%, 25.6% 51.2%) and low coherence (3.2%, 6.4%) groups, and compared the ROC between conditions within each group. Consistent with the overall result, we found no difference between the two conditions (force vs. no-force) in each high and low coherence group (low; t(19)=0.69, p=0.50, high; t(19)=0.56, p=0.59). Now, this analysis has been added to the manuscript (lines 306-308) and in the Figure S3 of the Supplementary materials.

Comment 5: Normality tests should be performed in all groups of data to select the most appropriate statistical test in each case.

Reply 5: We have now checked if the data fulfils the normality assumption for the parametric test. Please see lines 236-238.

Comment 6: A link to the data is provided but it is not yet possible to access the data.

Reply 6: The data is now made available. Please see <https://osf.io/be4u5/>.

Comment 7: Age interval is missing. Some reference, for instance, Song et al., 2011 are referred but not listed.

Reply 7: Thanks for pointing them out. Now, they are fixed in the revised manuscript. See lines 159 and the reference section.

Attachment

Submitted filename: Response_to_Reviewers.docx

Decision Letter 1

Gennady S Cymbalyuk

21 Oct 2022

PONE-D-22-12208R1Does decision confidence reflect effort?PLOS ONE

Dear Dr. Hagura,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please, address the issues concerning validity of the statistical analysis raised by the reviewer.Please submit your revised manuscript by Dec 05 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Gennady S. Cymbalyuk, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: No

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: I appreciate the effort made by the authors to improve the manuscript. I think it has gain in clarity. Still, few remaining comments:

Reply 2: I understand that the pilot experiment was only performed as an exploratory step before performing the experiment of the main paper. While this is completely fine and I am happy that it is now clearly stated in the text, I am still not convinced of using the t-test to compare the data. Under the experimental conditions that the authors are using, it is not correct to use such test. Thus, I would suggest to include the data as it is (in Fig S1), but to not include the result of a t-test. The authors could anyway discuss the tendency that they observed. Indeed, it should be clearly stated in lines 173-175 that there is a tendency.

Comment: Throughout manuscript, p values are reported by not the specific statistical test used. For instance, lines number 291, 295 or 297.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2023 Feb 16;18(2):e0278617. doi: 10.1371/journal.pone.0278617.r004

Author response to Decision Letter 1


27 Oct 2022

Response to the Reviewers

Reviewer #2

Comment1: I appreciate the effort made by the authors to improve the manuscript. I think it has gain in clarity. Still, few remaining comments:

Reply 2: I understand that the pilot experiment was only performed as an exploratory step before performing the experiment of the main paper. While this is completely fine and I am happy that it is now clearly stated in the text, I am still not convinced of using the t-test to compare the data. Under the experimental conditions that the authors are using, it is not correct to use such test. Thus, I would suggest to include the data as it is (in Fig S1), but to not include the result of a t-test. The authors could anyway discuss the tendency that they observed. Indeed, it should be clearly stated in lines 173-175 that there is a tendency.

Reply1: Following reviewer’s suggestions, we have revised the Supplementary Materials. First, we omitted the description of t-test from the legend of Figure S1, and also, deleted the asterisk (indicator of significance) from Figure S1b.

Comment2: Throughout manuscript, p values are reported by not the specific statistical test used. For instance, lines number 291, 295 or 297.

Reply2: We have now added the name of the test together with the p-value, if not indicated. Full table of the statistical values is described in table S1.

Attachment

Submitted filename: Revision2_Response_to_Reviewers.docx

Decision Letter 2

Gennady S Cymbalyuk

21 Nov 2022

Does decision confidence reflect effort?

PONE-D-22-12208R2

Dear Dr. Hagura,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Gennady S. Cymbalyuk, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: The manuscript has significantly improved and all my comments have been addressed. I have no further concerns.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

**********

Acceptance letter

Gennady S Cymbalyuk

28 Nov 2022

PONE-D-22-12208R2

Does decision confidence reflect effort?

Dear Dr. Hagura:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Gennady S. Cymbalyuk

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 File

    (PDF)

    Attachment

    Submitted filename: Response_to_Reviewers.docx

    Attachment

    Submitted filename: Revision2_Response_to_Reviewers.docx

    Data Availability Statement

    Content of the pre-registration is available at https://osf.io/tykcg. The associated data is accessible at https://osf.io/be4u5/.


    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES