Effortfulness of Visual Working Memory: Gauged by Physical Exertion

Abstract

Active maintenance of information in working memory (WM) is an essential but effortful cognitive process. Yet, the effortful nature of WM remains poorly understood. Here, we constructed a model to evaluate how perceived effort of WM is directly compared to that of physical exertion. In Experiment 1, participants freely chose to either remember a certain number of colors in a visual WM task or hold a hand dynamometer to a required percentage of maximal voluntary contraction (%MVC) to obtain a fixed task credit upon successful task completion. We found that participants discounted WM-related effort in the same way as they discounted handgrip-related effort based on a computation of expected choice outcomes (hence utility) associated with different task loads. This rationality in an observer’s prospective choice in Experiment 1 was generalized to retrospective choice in Experiment 2 where participants reported which task was more effortful immediately after they had performed both tasks in a randomized order without any reward or feedback. Experiment 3 further probed this shared mechanism using a dual-task paradigm. As predicted by our model, we found that physical exertion could disrupt the performance in the concurrent WM task, proportional to the iso-effort relationship between WM and physical exertion when task loads were high for both tasks. Collectively, our findings converge on a shared computational principle connecting task load, perceived effort, and choice utility across physical and cognitive domains.

Working memory actively maintains a limited amount of information over a short delay at the service of other ongoing mental processes (Baddeley, 2012; Cowan, 2001). Given its central role in human cognition, information retained in WM constrains a wide range of mental functions, including attentional allocation (e.g., Kane et al., 2006), fluid intelligence (e.g., Conway et al., 2002), affective biases (e.g., Lynn et al., 2016; Xie et al., 2016), and social cognition (e.g., Xie et al., 2020). As this core mental faculty is limited in storage and processing capacity (Adam, Vogel, & Awh, 2017; Cowan, 2001; Donkin, Nosofsky, Gold, & Shiffrin, 2013; Luck & Vogel, 2013; Xie & Zhang, 2017a; Zhang & Luck, 2008), retaining a large chunk of information in WM – even when it is benign and simple – appears to be aversive (Westbrook, Kester, & Braver, 2013) and is often associated with more negative affect (Laybourn, Frenzel, Constant, & Liesefeld, 2022). As a result, when given a choice, people often would rather perform a more strenuous and less efficient physical task, instead of maintaining a goal in WM that would allow them to complete the physical task more efficiently (Rosenbaum et al., 2014). In an extreme case, people would even rather receive an electrical shock in exchange for skipping performing a WM task (Vogel, Savelson, Otto, & Roy, 2020). These phenomenological observations reinforce the idea that humans are “cognitive misers,” who would try to minimize cognitive efforts whenever possible (Fiske & Taylor, 1991). However, despite this phenomenological intuition, it remains elusive what constitutes the effortfulness of WM and how it affects people’s tendency to avoid the engagement of WM.

Perceived effort and choice preference

Conceptually, effort is closely tied to the cost it takes to consume a scarce resource needed to fulfill an operation (Kool & Botvinick, 2018). The subjective experience of effortfulness, namely perceived effort, reflects the awareness of such costs. For example, the perceived effort of physical exertion has been commonly assessed using self-report measures (e.g., Borg Rating of Perceived Effort Scale; Borg, 1973; Haile et al., 2015), which may capture the metabolic cost associated with physical exertion as well as the experience of fatigue and triteness (DeLuca, 2005). Past research has shown that these subjective measures of perceived effort can track physiological responses (e.g., heart rate and oxygen consumption) elicited by physical loads (Asfoura, Ayoub, Mital, & Bethea, 1983), suggesting some consistency between perceived effort and task loads in the physical domain. Grounded in this association, recent research has further illustrated how perceived effort is linked with the utility of a physical action. For example, the computational term utility has been formalized as the ability to fulfill a task based on per unit effort committed (Morel, Ulbrich, & Gail, 2017). A high-utility choice, therefore, should lead to task fulfillment with a low effort when other choice parameters such as reward and time are held constant. This inverse relationship between perceived effort and utility has been demonstrated to follow a negative logarithmic relationship when participants were asked to directly compare between and choose from two physical actions (Morel et al., 2017). That is, U(E) = −log (E), where U refers to utility, and E refers to perceived effort that scales monotonically with task loads, namely E ∝ task load. According to the least effort principle (Zipf, 2012) and utility theories (Von Neumann & Morgenstern, 1944), human observers would contrast the utilities of two physical actions based on their perceived efforts and prefer the one that requires lower effort.

In contrast to the better-defined relationships among task load, effort, utility, and people’s choice behavior in the physical domain (Morel et al., 2017), similar characterizations of these variables remain missing in the cognitive domain. For example, although WM task load (e.g., memory set size) scales with perceived cognitive effort captured by subjective reports and neurophysiological responses (e.g., Kahneman & Beatty, 1966; Kardan et al., 2020), it is unclear whether cognitive effort is converted into a common-scale choice utility in the same way as physical effort (see some discussions in Feghhi & Rosenbaum, 2019). This uncertainty adds to the elusiveness in people’s behavioral tendency to avoid cognitive effort in exchange for physical exertion (Radel, Brisswalter, & Perrey, 2017; Rosenbaum et al., 2014; Rosenbaum, Sturgill, & Feghhi, 2022). In other words, how do people compare the perceived effort of cognitive and physical tasks to render a behavioral preference for one task over another?

Prior research and challenges

Intuitively, the comparison between physical and cognitive efforts can be inferred from past research on effort discounting. In these studies, effort discounting functions in the cognitive and physical domains are often quantified separately with respect to monetary rewards (e.g., Białaszek et al., 2017; Hartmann et al., 2013; Ostaszewski et al., 2013; Suzuki et al., 2021; Westbrook & Braver, 2015). The perceived effort for a given level of task load is converted to a subjective value in monetary units (e.g., Ostaszewski et al., 2013). Based on these subjective values, one can then infer an observer’s choice when cognitive and physical tasks are compared. This indirect approach, however, is faced with some challenges for the following reasons.

First, as effort discounting is often estimated separately for cognitive and physical tasks, it diverges from the more naturalistic scenario when human observers are often presented with and choose from concurrently available cognitive and physical options (e.g., Rosenbaum et al., 2022). Second, although monetary reward is a powerful tool to estimate the subjective value of perceived effort, it introduces an additional demand for reward processing (Suzuki et al., 2021). Considering that people have different sensitivity to monetary rewards (Capa & Bouquet, 2018), it remains unclear whether the effort discounting principle associated with rewards can be directly translated to the scenario when rewards are the same for different task options or when tasks are compared without any reward at all (Morel et al., 2017). Third, because of the temporal separation and the indirect comparison via monetary values, it is also unclear whether cognitive and physical efforts are indeed converted into the same internal representation to allow their direct comparison. Computationally, if a cognitive task follows the same effort-utility function as that of a physical task, perceived efforts across modalities are likely to be supported by a shared mechanism that converts task difficulty into the same underlying mental representation (Potts et al., 2018). In contrast, because cognitive and physical tasks usually involve separate neurobiological systems, cognitive and physical tasks may draw from separate pools of resources, such that engaging one may not directly affect another (Feghhi & Rosenbaum, 2019).

Addressing these challenges requires answering two primary questions concerning the relationship between cognitive and physical efforts. First, it is important to clarify how people’s choice or preference arises when cognitive and physical tasks are directly pitted against each other. In practice, this comparison may occur before or after an observer has carried out a task. For example, based on anticipated effort, observers can prospectively choose to carry out either a cognitive or a physical task to attain an overall task goal (e.g., completing a study for course credit, Rosenbaum et al., 2014), knowing that choosing one would naturally preclude the other. Observers can also retrospectively decide which task is more effortful based on the level of experienced effort immediately after they have performed both tasks (Morel et al., 2017). Hypothetically, people’s choices in both cases would be influenced by the perceived effort associated with task loads when other choice parameters such as reward and time are kept the same. Perceived effort, as a latent variable, therefore, can bridge task load and utility to allow direct comparison between two very different tasks in different modalities. However, it remains unclear whether this computation is mediated by the same or different effort-utility functions across cognitive and physical domains.

Second, building upon the knowledge concerning the relationship among task loads, effort, and utility, it is important to probe the direct competition between cognitive and physical tasks when they are concurrently performed. If effort-utility relationships associated with difference tasks are indeed converted into the same internal representation, engaging one task to a certain level should readily impact the performance of another task (Feghhi & Rosenbaum, 2019; also see a similar rationale outlined in van Moorselaar et al., 2018 for a different cognitive phenomenon). Critically, because effort may be conceptualized as a finite resource that supports task fulfillment (Kool & Botvinick, 2018), the trade-off in dual-task performance should be predicted by the change in perceived effort associated with the manipulation of concurrent task loads across modalities. Thus far, evidence supporting this core prediction is scarce.

The current study

To investigate these issues and to gauge the perceived effort of WM in comparison to that of physical exertion, we examine how remembering different numbers of items in a visual WM task is compared to holding a hand dynameter at a certain percentage of the maximal voluntary contraction (%MVC). Here, isometric muscle contraction of the handgrip is considered, in which the hand muscle is activated but held at a constant length to hold a large object (Cain & Stevens, 1971). This power grip procedure is expected to require less executive control as compared with precision grip (Ehrsson et al., 2000; Guillery, Mouraux, & Thonnard, 2013; Guillery, Mouraux, Thonnard, & Legrain, 2017; Kobayashi-Cuya et al., 2018), and hence presumably minimizing the overlap in the processes required for WM and physical exertion. Based on these setups, we designed three experiments to investigate the two primary questions outlined above.

First, we examined how the anticipated and experienced components of perceived effort influence people’s choice when cognitive and physical tasks are directly compared. We began with the scenario where people were asked to choose to perform either a WM task at a given set size or a handgrip task at a given % MVC on a given trial in order to earn a fixed task credit upon successful task completion (Experiment 1). As completing a task successfully regardless of choice would lead to the same amount of reward (task credit) on each trial, we expected that participants’ choice behavior would somewhat depend on the anticipated effort and accuracy associated with the presented task options (Feghhi & Rosenbaum, 2019; Johnson & Payne, 1985). Here, as the anticipated effort and accuracy both scale with task loads, it provides an opportunity to model choice behavior that can be verified by choice outcomes. That is, a rational observer should pick the task with higher expected accuracy to maximize the reward, hence a choice option that has a higher utility (Von Neumann & Morgenstern, 1944). As we established this conjecture, Experiment 2 further removed the reward component and assessed how the experienced effort of the WM task would be compared to that of the handgrip task after an observer has carried out both tasks on each trial. Across experiments, by modeling the iso-effort pattern across modalities, we find that perceived efforts of WM and handgrip tasks, either before or immediately after an observer has carried out a task, can be captured by the same effort-utility function, suggesting a shared effort-utility representation across cognitive and physical domains. Based on this finding, one should expect a predictable trade-off in task performance when cognitive and physical tasks directly compete with one another (Potts et al., 2018).

Second, to test this core prediction, we then examined whether and how increasing the level of physical exertion could proportionally reduce observers’ performance in a concurrent visual WM task in Experiment 3. As the effort one can devote to a task is finite (Kool & Botvinick, 2018), participants’ performance in a dual-task paradigm should be predicted by the sum of perceived effort associated with task loads (Shenhav et al., 2017). For example, a high physical load at a large WM set size (e.g., set size 6) could lead to joint effort beyond 100% and subsequently reduce the effective number of items that can be held in visual WM. This prediction is in line with the observation that large WM set sizes often impose a higher level of competition in dual-task paradigms (Pashler, 1994; Tombu & Jolicoeur, 2003). Critically, if this trade-off is related to the competition related to a shared representation of effort (Shenhav et al., 2017), the reduced number of remembered WM items when both task loads are high (e.g., remembering 6 colors higher an increased physical load) should be predicted by the iso-effort relationship between WM and physical exertion. Alternatively, if concurrent physical exertion primarily distracts the ongoing WM task without competing for shared effort-related processes (Feghhi & Rosenbaum, 2019), it is expected that the iso-effort relationship between WM and physical exertion would be irrelevant for reduced WM task performance under concurrent physical loads. This prediction is in line with the strict, complete lapse model, in which adding a parameter capturing the momentary attentional disengagement irrespective of task demands can substantially improve model estimates of WM capacity (Rouder et al., 2008; Rouder, Morey, Morey, & Cowan, 2011). Conceptually, these potential lapses may occur at any time during an experiment (Adam, Mance, Fukuda, & Vogel, 2015; deBettencourt, Keene, Awh, & Vogel, 2019) and are different from failures in attentional control (Adam et al., 2015; Fukuda, Woodman, & Vogel, 2015). If concurrent physical load results in an overall reduction in WM task performance across set sizes, the data would be more in line with the strict, complete lapse model (Rouder et al., 2008, 2011). In this case, our modeling framework may not predict the magnitude of this overall drop in WM task performance across and within participants. To preview, results from Experiment 3 are more in line with the former prediction, in which increasing physical load during WM encoding and retention has proportionally reduced observers’ performance in a concurrent visual WM task at a large set size as predicted by the best-fit model in Experiments 1 and 2.

Experiment 1

Based on the intrinsic association between task loads and perceived effort (Kurzban, Duckworth, Kable, & Myers, 2013; Shenhav et al., 2017; Suzuki et al., 2021), we first investigate how isometric physical exertion and WM are directly compared to one another based on the anticipated effort associated with task loads. On each trial, participants chose either to perform a whole-report color WM task (Adam et al., 2015) with a given memory load (1, 2, 3, 4, or 6) or to hold a cylindrical hand dynamometer to a certain level of force exertion (20%, 30%, 45%, 65%, or 90% MVC). Successful completion of the chosen task on each trial within a fixed trial duration would lead to a fixed task credit that was cumulative over the course of the study and was converted into a monetary reward at the end of the study. We modeled participants’ choice behavior using a Bayesian hierarchical ideal-observer model based on a factorial combination of different utility-effort functions across cognitive and physical domains. A major assumption of this model is that observers are rational and would render a choice based on its utility or expected outcome even when qualitatively different tasks across modalities are directly compared. This assumption is not a given because participants can have a priori preferences regardless of task loads (e.g., avoidance of cognitive tasks, “cognitive miser,” Fiske & Taylor, 1991). However, if participants’ choice behavior indeed follows this assumption, their prospective choice in Experiment 1 should scale with the anticipated accuracy of the presented task options. To verify this, we found that participants’ likelihood of successful task completion and choice behavior systematically varied based on task loads across modalities in Experiment 1, suggesting that participants’ choices were not arbitrary. As we established this assumption, we intended to generalize this model to a scenario where the influence of explicit rewards is eliminated in people’s retrospective choices after they have experienced both the WM and handgrip tasks on each trial (see Experiment 2).

Methods

Participants

A total of 40 right-handed college students (20.91 ± 0.47 [mean ± s.e.m] years old, 24 females, 16 males, and none reporting as others) from the University of California, Riverside took part in the two-session experiment (~1 hour/session). As we intended to model people’s choice behaviors while incorporating potential biases or preferences, we set our sample size more than twice that in a previous study using similar analytical approaches (n = 16 to 17 participants within each experiment in Morel et al., 2017). All participants reported normal (or corrected-to-normal) vision and color vision. Participants provided informed consent at the beginning of the study based on the protocol approved by the local Institutional Review Board (IRB) and received a combination of course credits and monetary rewards for compensation.

Stimulus and Apparatus

Stimuli were presented using Psychophysics Toolbox (Brainard, 1997) in Matlab (MathWorks, Cambridge, MA) on a 60 Hz monitor, calibrated with an X-Rite I1Pro spectrophotometer, with a gray background (6.1 cd/m²) at a viewing distance of ~57 cm. For the visual WM task, 9 distinct colors were chosen based on previous studies (Rouder et al., 2008; Xie & Zhang, 2017a). In the study phase, a set of colors were randomly chosen from the 9 predefined colors and presented in squares subtending 2° × 2° in visual angle. These study items were randomly presented on an invisible circle with a radius of 6.5° visual angle from the center of the screen. For the handgrip task, a digital hand dynamometer (Vernier Software & Technology, Beaverton, OR) was used to collect participants’ hand force (Docx et al., 2015; Park, Ahn, & Zhang, 2021; Stanek & Richter, 2016).

Procedure

Participants completed two sessions, separated by 1–3 days, to minimize physical fatigue across sessions. In the first session, participants sequentially completed the measurement of MVC, the practice for the handgrip task, and the practice for the visual WM task. Afterward, they performed the first half of a decision-making task using the choice paradigm. In the second session, participants completed the second half of the decision task and provided survey data regarding their demographic information.

Measurement of MVC and Practice of the Handgrip Task.

Before detailed instructions to the study, participants were at first asked to grip a hand dynamometer with their left hand as hard as possible in three consecutive trials of 4 seconds. They placed the left hand on a table with the palm facing upward. No visual feedback was given in these calibration trials (Hartmann et al., 2013). The onset of measurement was indicated by a “Beep” sound delivered via a headset. The median force value of the last 2 seconds in each calibration trial was taken as an estimate of MVC for each trial. The final MVC was the average of these three estimates. In the current sample, participants’ mean MVC is 127.45 ± 8.63 Newton and male participants show a higher MVC as compared with female participants (173.78 ± 12.85 Newton vs. 96.55 ± 5.92 Newton, t₍₃₈₎ = 6.07, p < .0001). This pattern is consistent with the literature (Zaccagni et al., 2020) and is also present in Experiments 2 and 3, suggesting the reliability of the current MVC measure protocol across experiments (see Supplementary Table S1).

Following the measure of MVC, participants were asked to practice exerting their hand force to different levels of MVC. Specifically, at the beginning of each trial, a rectangular box (4° × 6° in visual angle) with a line indicating a certain percentage of individually calibrated MVC (e.g., 20%, 30%, 45%, 65%, or 90%) was shown at the center of the computer screen (similar to what is shown in the lower panel of Figure 1). Participants were instructed to use the left hand to hold the hand dynamometer at the required level for 4 seconds, during which a vertical gauge (red bar) proportional to the exerted force was displayed on the screen on a moment-by-moment basis. To be considered successfully completing the trial, participants would need to ensure that the red bar stayed beyond the indicated line for more than 66.7% of the time during the last 2 seconds of the 4-second measurement time window. Upon successful completion of a handgrip trial, a display of “You have successfully maintained the force!” with a “Cha-Ching” sound was presented for 1,000 ms. Otherwise, a display of “Not quite. Please try hander!” with a “Beep” sound was presented for 1,000 ms. Participants completed 10 trials with 2 trials for each MVC level in a random order.

Figure 1. The choice paradigm used in Experiment 1.

Participants were asked to choose to perform a color visual WM task or a handgrip task with different task loads (#Color or %MVC) pitted against each other. Upon successful task completion, participants would earn a point of credit regardless of the chosen task. Background color was converted from gray to white for visualization and printing efficiency.

Practice for the Whole-report Visual WM Task.

A whole-report visual WM task was used to ensure that chance performance, as a proxy for the required effort, would change as a function of memory set size (Adam et al., 2015). This whole-report task was similar to the typical change detection task (Luck & Vogel, 1997) with the exception that participants would need to recall each study item during the test phase. At the beginning of each trial, participants saw an array of study items (1, 2, 3, 4, or 6 color squares) for 500 ms. After a short delay of 1,000 ms, a test display was presented with a 3 by 3 grid of color squares (1° × 1° visual angle for each smaller color square) at the original locations of the studied items (similar to what is shown in the upper panel of Figure 1). Participants were instructed to click the mouse cursor on the color square corresponding to the study item at each location. Participants were encouraged to respond to as many items and as accurately as they could remember within the fixed 6-second response time window. If a participant had accurately recalled all studied colors, a display of “You have successfully recalled all colors!” with a “Cha-Ching” sound was shown for 1,000 ms after the response time window. Otherwise, a display of “You have missed a few colors!” with a “Beep” sound was shown for 1,000 ms. Participants completed 10 trials with 2 trials for each set size in a random order.

Choice Paradigm in Experiment 1.

Each trial begins with a visual illustration of a random combination of left-hand %MVC and WM study set size (see Figure 1). Participants were asked to choose between these two possible options and complete the chosen task successfully to earn a point of task credit. This credit would be the same regardless of which task they had chosen. All the points across successful trials would be accumulated then converted into monetary reward at the end of the study. Hence, participants were motivated to make the best decision to maximize the likelihood of earning a point of credit at the end of each trial. The chosen task (handgrip or visual WM) proceeded as that described in the practice phase. A feedback display indicating the number of points (either 0 or 1) participants had earned would appear for 3.5 seconds at the end of the trial, which also provided a short break to prevent physical or cognitive fatigue. Regardless of which task the participant opted to do, the total duration of each trial was fixed at 8 seconds, and participants were informed about this at the beginning of the experiment.

In each experimental block, the 5 levels of physical load and 5 WM set size were randomly pitted against each other twice across trials, yielding a total of 50 trials per block. In each session, participants completed 4 blocks of the experiment (i.e., a total of 200 trials per session). Participants were given a short break after every 10 trials to minimize fatigue. They continued the task at their own pace. Participants would possibly earn a total of 400 points across two sessions. Their compensation was staged as $20 for 380~400 points, $15 for 350~379 points, and $10 for points lower or equal to 349, which was not disclosed to the participants until the end of the study.

Data Analysis

Behavioral Performance.

It is practically difficult to estimate the behavioral performance separately for each task in Experiment 1, given that behavioral performance is contingent on a participant’s choice. For example, we may not have enough trials for a reliable measure of WM task performance (e.g., <5 trials) at high WM set sizes and low physical exertions. That said, participants’ overall likelihood of successful task completion regardless of the task chosen provides a key estimate to verify an assumption in our model regarding the intrinsic relationship between task performance (and potentially perceived effort) and task load (Cain & Stevens, 1971; Kardan et al., 2020). Conceptually, participants’ likelihood of successful task completion across task loads should reflect the respective task difficulty levels and potentially their perceived effort. To verify this as a manipulation check, we therefore examined how participants’ likelihood of successfully completing a chosen task would change as a function of %MVC and memory set size (#Color) based on two-factor (i.e., %MVC and #Color) repeated-measures analysis of variance (ANOVA). As a complementary analysis, we have also visualized participants’ task performance separately for the handgrip and WM tasks conditioned on the task they have chosen in Supplementary Figure S1. Because this conditional task performance is a biased estimate of overall task performance, we therefore refrain from further statistical inference on these results. Yet, the overall data profile appears consistent with the predicted relationship between task performance and task load, which will be further addressed in Experiment 2.

Modeling Choice Behavior.

We modeled participants’ choices based on a Bayesian hierarchical ideal-observer model (Morel et al., 2017) using the probabilistic programming language, Stan (Carpenter et al., 2017). In this model (Figure 2), human observers are assumed to make an ideal decision based on the utility of an option, which is inversely related to the level of perceived effort associated with the option. That is, an observer would choose a task that is perceived less effortful and hence has higher utility for task completion based on the amount of effort required. Consequently, the choice of the handgrip task over the WM task can be modeled as a Bernoulli distribution in which the associated probability of choosing the handgrip task as the less effortful option, P(Handgrip | %MVC, #Color), is a function of the direct comparison of utilities between the handgrip task and the WM task at each given level of task load,

$$P(\text{Handgrip | \%MVC, \#Color })=\mathrm{\Phi }\left(\frac{U\left(E_i(\%MVC)\right)-U\left(E_i(\#\mathit{\text{Color}})\right)}{{\gamma }_i}\right)$$

Figure 2. An ideal-observer modeling framework to capture participants’ choice behavior in deciding between a cognitive task and a physical task.

Participants’ choice is modeled as a probabilistic outcome through an internal comparison of task utility based on the perceived effort related to task loads. Here, the notation in this figure is simplified for visualization purpose. α’ and β’ reflect different values as compared with α and β. Please see Methods in Experiment 1 for formal notation.

Here, Φ is the cumulative density function (CDF) of the standard normal distribution. E_i(%MVC) is the perceived effortfulness of a handgrip action at a given level of %MVC for subject i, and similarly, E_i(#Color) is the perceived effort of the WM task with a given number of colors to be remembered (#Color) for subject i. To allow direct comparison, raw %MVC (20%/30%/45%/65%/90%) and raw #Color (1/2/3/4/6) were z-scored separately and then shifted to a common starting point of 0 as the lowest value. To capture the flexible pattern of perceived effort as a function of the task demand, perceived effort is modeled as a power-law function of %MVC or #Color,

$$E_i(\%MVC)=\%MV{C}^{{\alpha }_i^{MVC}}+{\beta }_i^{MVC}$$

$$E_i(\#\mathit{\text{Color}})=\#{\mathit{\text{Color}}}^{{\alpha }_i^{\mathit{\text{Color}}}}+{\beta }_i^{\mathit{\text{Color}}}$$

As perceived effort is expected to increase as a function of task loads, we only considered positive-bounded exponents for α_i^MVC and α_i^Color. These exponents, along with the effort sensitivity parameter γ_i for each subject i, were drawn from truncated normal distributions with only positive values. The unbounded effort offsets β_i^MVC and β_i^Color were sample from normal distributions. Each of these distributions have their separate means and standard deviations,

$${\alpha }_i^{\text{MVC}}~N\left({\mu }_{\alpha }^{\text{MVC}},{\sigma }_{\alpha }^{\text{MVC}}\right),\text{positive values only}$$

$${\alpha }_i^{\text{Color }}~N\left({\mu }_{\alpha }^{\text{Color }},{\sigma }_{\alpha }^{\text{Color }}\right),\text{positive values only}$$

$${\gamma }_i~N\left({\mu }_{\gamma },{\sigma }_{\gamma }\right),\text{positive values only}$$

$${\beta }_i{}^{\text{MVC}}~N\left({\mu }_{\beta }^{\text{MVC}},{\sigma }_{\beta }^{\text{MVC}}\right)$$

$${\beta }_i^{\text{Color }}~N\left({{\mu }_{\beta }}^{\text{Color}},{{\sigma }_{\beta }}^{\text{Color}}\right)$$

Relatively wide prior distributions were chosen to not constrain the model unnecessarily. Specifically, μ_α^MVC, μ_α^Color, μ_β^MVC, and μ_β^Color were sampled from N(1, 10). The priors of the positive scale parameters, namely σ_α^MVC, σ_β^MVC, σ_α^Color, σ_β^Color, μ_γ, and σ_γ, follow a truncated half-Cauchy distribution (location = 0; scale = 20). Posterior distributions were sampled 10,000 times across 4 Markov chains after a warmup of 2,500 samples in each chain.

U is the utility as a function of perceived effort E, which may or may not be the same across cognitive and physical domains. We therefore varied the associated effort-utility function for the handgrip and WM tasks, and then compared the model fits using the Watanabe–Akaike information criterion (WAIC, Watanabe, 2010). Based on the previous study (Morel et al., 2017), there are four competing models of interests:

$$U(E)=-\text{log}(E) \text{Negative logarithmic effort model}$$

$$U\left(E\right)=-E \text{Negative effort model}$$

$$U(E)=\frac{1}{E} \text{Inverse effort model}$$

$$U(E)=\frac{1}{\text{log}(E)} \text{Hyperbolic logarithmic effort model}$$

When two levels of physical load are pitted against one another, the negative logarithmic effort model can best capture participants’ choice patterns (Morel et al., 2017). To test whether the same principle also applies when a physical task is directly compared with cognitive task, we performed a factorial model comparison analysis and evaluated which one of the 16 possible combinations of effort-utility functions (4 in cognitive domain by 4 in the physical domain) could best capture the current cross-modality choice data.

Data Availability

Raw trial-level data and scripts for this experiment and all the subsequent experiments are available online via the Open Science Framework, https://osf.io/5sdgr/.

Results

Behavioral Performance

Although participants were motivated to maximize their performance to earn a point regardless of which task they had chosen, as the task load increased, participants’ overall likelihood of successful task completion monotonically decreased (main effect of %MVC, F_{(4, 156)} = 155.21, p < .001, η²_p = .78; main effect of #Color, F_{(4, 156)} = 84.26, p < .001, η²_p = .68; interaction effect: F_{(16, 624)} = 52.59, p < .001, η²_p = .57; Figure 3A). These results confirmed a key assumption in our model regarding the intrinsic relationship between overall task performance (and potentially perceived effort) and task load (Cain & Stevens, 1971; Kardan et al., 2020; also see Supplementary Figure S1A for results based on separate tasks).

Figure 3. Participants’ performance, choice behavior, and best-fit model in Experiment 1.

(A) Participants’ rates of successful trial completion as a function of task loads across cognitive and physical domains. See Supplementary Figure S1A for further task performance results separated by different task loads. (B) Participants’ tendency to choose the handgrip task as an easier or less effortful task across different levels of %MVC in the handgrip task and #Color in the WM task. The lines represent the predictions of the best-fitting model (Model 1), based on a factorial model comparison of all possible models summarized in (C). (D) Across all available models, Model 1 has yielded the smallest WAIC, suggesting that Model 1 is the preferred model that can best account for the data observed in Experiment 1. Each dot in (A) & (B) represents the mean estimate from one experimental condition, with the error bar representing the standard error of the mean across participants.

Modeling Choice Behavior

We then examined how perceived efforts in the physical and cognitive domains are linked with an internal calculation of utility that ultimately affects one’s choice behavior. As shown in Figure 3B, participants’ choice behavior systematically varied based on the given %MVC and #Color across trials. In general, participants preferred the handgrip task over the WM task when the required MVC level was low and when WM set size was high. Based on the proposed modeling framework, we modeled this behavior by comparing the physical and cognitive tasks on a utility metric that is inversely related to the level of perceived effort in performing a chosen task (Morel et al., 2017). We assumed that the higher the task demand of a chosen task is, the more effortful it would be and hence the lower utility of this task for successful task completion. A rational and ideal observer would then favor the choice associated with a higher utility.

We modeled this effort-discounting relationship in the physical and cognitive domains using the various discounting functions (e.g., negative linear, hyperbolic, negative logarithmic, and hyperbolic logarithmic models) and performed a factorial model comparison via Bayesian hierarchical modeling using WAIC as a criterion metric (summarized in Figure 3C). We found that the best fitting model assumes the same negative logarithmic relationship for the effort-utility function in both the physical and cognitive domains (Model 1, Figure 3B). This model has yielded the smallest WAIC value (Model 1, WAIC = −2386.3), which is smaller than the second-best fitting model of these data (Model 2, WAIC = −2373.8) by12.5 arbitrary unis. Furthermore, calculation of Akaike weights (Wagenmakers & Farrell, 2004) of these data suggests that Model 1 is 99.8% likely to be preferred over all the other available models for data observed in Experiment 1 (Figure 3D).

Discussion

Overall, data from Experiment 1 suggest that the same computational principle may underlie the relationship between anticipated effort and task loads across cognitive and physical domains. This computational principle follows a negative logarithmic function relating both cognitive and physical efforts to decision-related utility, that is, how useful it is to pay effort to complete a certain task. We have demonstrated this relationship based on a new choice paradigm, where participants were directly asked to maximize their anticipated reward regardless of task choice within a fixed trial duration (hence minimizing temporal discounting, Green, Fristoe, & Myerson, 1994). These results provide key evidence that participants’ choice behavior in the current paradigm systematically varied based on task loads and potentially perceived effort, instead of being driven by arbitrary response biases (e.g., always prefer the physical task over the cognitive task). This rationality suggests that people are not always “cognitive misers”, who would have an a priori preference for physical tasks over cognitive tasks (Fiske & Taylor, 1991; Rosenbaum et al., 2014; Vogel et al., 2020). Instead, our data suggest that when cognitive and physical tasks are directly compared, people’s choices may be driven by a deliberate computation of perceived effort associated with the utility of a task option.

Experiment 2

Grounded in the deliberate computation of perceived effort and choice utility, Experiment 2 aims to test the generalizability of the effort-utility model from Experiment 1 in two ways. First, we expected to extend prospective choice in Experiment 1 to retrospective choice in Experiment 2. Second, Experiment 2 further eliminated the reward and feedback components of the task to explicitly minimize the influences of reward expectation on choice behavior. In brief, on each trial, participants performed both the color WM task and the handgrip task in random order at a randomly chosen level of task loads within a fixed trial duration and immediately were asked to report “which task was more effortful” without any feedback. To minimize the involvement of performance-contingent rewards, participants only received course credit for their participation regardless of their task performance. This procedure tested retrospective choice preference and minimized the chances that participants would make a choice primarily based on trial duration (Morel et al., 2017), temporal discounting (Green et al., 1994), response accuracy (Westbrook & Braver, 2015), or reward history (Muranishi et al., 2011).

Methods

Participants

A different group of 20 right-handed college students (22.11 ± 0.95 years old, 15 females, 5 males, and none reporting as others) was recruited for the two-session Experiment 2 (~1 h per session) with course credit as the only form of compensation. As we have established the assumption that the participants would render a choice based on task loads instead of response biases in Experiment 1, we reduced the sample size to 20 in Experiment 2. This sample remains greater than the sample size used in a previous similar study (Morel et al., 2017), and would have ~80% statistical power to detect a medium-to-large load effect (Cohen’s f > 0.25 in repeated-measured ANOVA) in participants’ WM or handgrip task performance across 5 levels of task loads at an alpha level of .05 (Faul, Erdfelder, Buchner, & Lang, 2009). Here, the load effect on task performance is expected to be obvious based on data from Experiment 1 and a medium-to-large effect size would therefore be a conservative estimate. Furthermore, the primary analysis for the choice data (i.e., Bayesian hierarchical modeling) aggregates all information across trials and participants during model fitting, and consequently reducing parameters’ sensitivity to noise within each participant and to smaller sample sizes (He, Wang, Huang, Wang, & Guan, 2021; McNeish & Stapleton, 2016; Van de Schoot, Broere, Perryck, & Loey, 2015). All participants reported normal (or corrected-to-normal) vision and color vision and provided informed consent was given before the study began.

Stimulus and Apparatus

All stimuli, laboratory settings, and apparatus were identical to those in Experiment 1.

Procedure

The procedure was identical to that in Experiment 1, except that the choice paradigm was modified in several ways as described below.

Choice Paradigm in Experiment 2.

First, a major modification in Experiment 2 was that participants performed both the handgrip task and the whole-report WM task before they decided which task was more effortful on each trial (Figure 4). Specifically, at the beginning of each trial, participants were asked to complete a handgrip task with a certain level of physical load and a whole-report color visual WM task with a certain memory set size. The order of these two tasks was random across trials. The total duration of each trial in the handgrip task and the WM task was again fixed at 8 seconds. Following these tasks, participants were asked to report which task was more effortful. Second, to encourage participants to make the decision based on perceived effort, instead of anticipated accuracy or reward, no feedback on the task performance was given on each trial. The participants were encouraged to perform both tasks as best as they could, and then made the effortfulness evaluation as faithfully as possible. To check the load-dependent assumption concerning effort-related choice preference in our model (Figure 1), we verified that participants’ choice behavior indeed systematically varied based on task loads even when they had successfully performed both tasks on the same trials (see Supplementary Figure S2). Third, while most of the task settings remained the same as those in Experiment 1, the visual gauge for the exerted grip force in the handgrip task was presented only for the first 2 seconds. Participants were told to retain the physical exertion until a “Relax” sign was presented on the screen when the time was up (4 seconds). This was to minimize the potential fine-grained adjustment of physical exertion around the required %MVC based on the visual gauge (and hence precision grip). Lastly, to accommodate the increased trial length, participants completed 100 trials (2 blocks) in Session 1 and 150 trials (3 blocks) in Session 2, yielding a total of 10 trials per %MVC and WM set size combination across two sessions. Although the number of trials per condition was reduced, it was still on par with a previous study (Morel et al., 2017).

Figure 4. The choice paradigm used in Experiment 2.

Participants performed both the whole-report WM task and the handgrip task in random order without any reward or feedback before they were asked to report which task was more effortful. Background color was converted from gray to white for visualization and printing efficiency.

Data Analysis

Behavioral Performance.

Given that the participants performed both the handgrip task and WM task in random order on every trial, we can better evaluate how task load influences participants’ performance separately in each modality, which was practically impossible in Experiment 1. Hence, we analyzed the choice behavior data separated for the handgrip task and the visual WM task as a function of task loads (%MVC, Figure 5A and #Color, Figure 5B) based on repeated-measures ANOVA. For the visual WM task, to improve interpretability, we directly used the number of remembered items in the whole-report paradigm as an outcome measure for the WM task (Adam et al., 2015).

Figure 5. Participants’ performance, choice behavior, and best-fit model in Experiment 2.

(A) Participants’ performance in the handgrip task as a function of the required %MVC. (B) Participants’ performance in the whole-report visual WM task as a function of set size. The number of retained WM items is calculated as the actual number of successfully recalled items. See Supplementary Figure S1B for further task performance results separated by different task loads. (C) Participants’ tendency to choose the handgrip task as being the less effortful one across different levels of %MVC in the handgrip task and #Color in the WM task. This visualization ensures the consistency of the y-axis across Experiments 1 and 2. The lines represent the predictions of the best-fitting model (Model 1). (D) Across all available models, Model 1 has yielded the smallest WAIC, suggesting that Model 1 is the preferred model that can best account for the data observed in Experiment 2. Each dot in (A) ~ (C) represents the estimate from one experimental condition, with the error bar representing the standard error of the mean across participants.

Modeling Choice Behavior.

We modeled participants’ choice responses using the same Bayesian hierarchical ideal-observer modeling framework outlined in Experiment 1. Based on the iso-effort curve predicted by the best-fitting model, we quantified the perceived equivalent WM items for each %MVC. We operationalized this estimate by calculating the predicted number of to-be-remembered WM items when the likelihood of a participant choosing the handgrip task over the color WM task was at 50% (Figure 6A). This additional analysis allows us to gauge how WM set sizes (#Color) equate to %MVC in terms of perceived effort.

Figure 6. Iso-effort patterns.

(A) Example choice preference at a given level of %MVC across WM set sizes when WM and handgrip tasks are directly compared (data from the 45% MVC condition of Experiment 1). The equivalent WM set size for a given level of physical load is operationalized as the x-axis value where the y-axis value is predicted to be 50% in a participant’s choice profile (see red dashed lines) based on the best-fit model from Experiments 1 and 2. In other words, at this equivalent WM set size, participants would be equally likely to choose a handgrip task or a WM task when these task loads were directly compared in perceived effort. (B) Across Experiments 1 and 2, we find that the relationship between %MVC and the equivalent WM items follows a linear pattern after a certain level of set size (2 to 3 items, x-axis intercept). Error bars represent the standard error of the mean across participants.

Results

Behavioral Performance

In Experiment 2, because both tasks were performed independently with a randomized order, the interaction in behavioral performance between the two tasks were expected to be attenuated or inconsequential (also see Supplementary Figure S1B for details). Indeed, in the handgrip task (Figure 5A), participants’ percentage of successful hand force maintenance markedly reduce from 89% to 38% as the %MVC increased from 20% to 90% (main effect of physical load: F_{(4, 76)} = 53.69, p < .001, η²_p = .74), irrespective of the set sizes of a temporally adjacent but not concurrent WM task (main effect of WM set size: F_{(4, 76)} = 0.35, p = 0.85, η²_p = .018; interaction between physical load and WM set size: F_{(16, 304)} = 1.35, p = 0.17, η²_p = .066). The biggest drop occurred in the transition between 45% MVC to 65% MVC (t₍₁₉₎ = 7.91, p < .0001, Cohen’s d = 1.77), but there was no significant difference in success rate between 65% MVC to 90% MVC (t₍₁₉₎ = 1.28, p = 0.22, Cohen’s d = 0.29). In the whole-report color WM task (Figure 5B), we replicated a previously observed pattern (Adam et al., 2015), in that the number of items participants could successfully recall increased as a function of the number of to-be-remembered WM items (main effect of WM set size: F_{(4, 76)} = 123.61, p < .001, η²_p = .87), irrespective of the physical load (%MVC) of a temporally adjacent but not concurrent handgrip task (main effect of physical load: F_{(4, 76)} = 0.46, p = 0.77, η²_p = .023; interaction between physical load and WM set size: F_{(16, 304)} = 1.57, p = 0.075, η²_p = .076). Moreover, the performance increase seemed to asymptote at set size 4 and 6 (# of recalled items at set size 4 vs. 6: 2.84 vs. 2.80, t₍₁₉₎ = 0.38, p = 0.71, Cohen’s d = 0.09). Overall, these results suggest that participants perform both tasks reasonably well even without explicit reward or feedback and that their performance is reliably influenced by task loads.

Modeling Choice Behavior

We next examined how perceived efforts in the physical and cognitive domains are directly compared based on the modeling framework outlined in Experiment 1 (Figure 2). Unlike Experiment 1, participants’ evaluation on the effortfulness of the tasks in Experiment 2 presumably is driven by the experiential effort instead of anticipated effort. Despite this difference, participants’ choice behavior systematically varied based on the given %MVC and #Color (Figure 5C), which can also be best accounted for Model 1 as compared with all the other models. In particular, Model 1 has yielded the smallest WAIC value (−532.6), which is smaller than the second-best fitting model (Model 11, WAIC = −522.7) by 9.9 arbitrary units (Figure 5D). Similar to Experiment 1, Akaike weights of these data suggested that Model 1 was 99.3% likely to be the preferred model.

In addition to the consistency in the best-fitting model across Experiments 1 and 2, the modeling outcomes of participants’ choice behavior present some noticeable patterns. First, the same set of models (Models 1, 2, 5, 6, 11) emerges as better-fitting models across experiments, suggesting the generalizability of our current modeling framework in capturing underlying decision-making principles that are robust to procedural differences (e.g., prospective vs. retrospective choices). Second, not all discounting functions can capture the effort-utility relationship in the physical and cognitive domains. For example, among the set of worse-fitting models, most of them contain the model that assumes a hyperbolic logarithmic effort-utility profile (i.e., U = 1/log(E); Models 4, 8, 12, 16, 13, 14, 15). This is in line with the broader literature that has suggested certain shapes of the discounting functions (e.g., steep vs. shallow discounting) may better capture the effort-utility relationship than others (also see a discussion in Morel et al., 2017), an observation also seen in other discounting phenomena (e.g., temporal discounting, Patt et al., 2021). Third, within the models that can mostly generate reasonably good fits (U = −log(E), −E, or 1/E), models with the same effort-utility discounting function across both physical and cognitive domains (i.e., Models 1, 6, and 11) tend to yield better fits as compared with the models with a certain combination of different effort-utility discounting functions (i.e., Models 5, 9, 10, 2, 3, and 7). For example, the average WAIC of better-fit models with the same effort-utility discounting function (Models 1, 6, and 11) is markedly smaller than that of models with different effort-utility discounting functions across modalities (Models 5, 9, 10, 2, 3, and 7) in both experiments (average ΔWAIC between these two classes of models in Experiment 1: −687.8 and Experiment 2: −149.9, where a negative ΔWAIC means better model fit). This can be generalized to the comparison of average WAIC between all the models along the diagonal relative to the models off-diagonal in the summary tables of Figures 3C and 5C even when results from a less reasonable model (i.e., U = 1/log(E)) are also included (average ΔWAIC between these two classes of models in Experiment 1: −360.8 and Experiment 2: −64.4).

Collectively, these results suggest that certain effort-utility functions may capture participants’ choice behavior, with the models that assume the same effort-utility function in the physical and cognitive domains providing better fits as compared with those assuming different effort-utility functions across modalities. Among these better-fitting models, Model 1 with the same negative logarithmic effort-utility relationship across domains provides the best fit for our data.

Equating Physical and Cognitive Efforts

Based on the iso-effort patterns predicted by the best-fitting model (see Figure 6A for an example), we found that the perceived equivalent WM items linearly scaled with %MVC – a pattern that is consistent across experiments (Figure 6B). Although there seems to be a horizontal shift in the estimated x-axis intercepts in different experiments, these intercept values were similar across experiments (2.19 vs. 2.28). We tested whether these mean estimates were statistically different from one another by fitting these iso-effort patterns based on data with shuffled experiment labels, which created an empirical null distribution with minimal statistical assumptions (Good, 2013). Across 10,000 iterations, we found that the observed x-axis intercept difference between experiments was not statistically different from the null values (bootstrapped p = 0.65). These x-axis intercepts are meaningful in this case as they reflect the theoretical prediction of the equivalent WM items at 0% MVC, namely the theoretical number of colors that people can remember with minimal effort. These values are slightly below the number of color squares that participants can hold in visual WM as estimated in the current study (Figure 5B) and in the literature (Adam et al., 2015, 2017; Luck & Vogel, 1997; Rouder et al., 2011; Xie et al., 2020), potentially reflecting conservative metacognitive estimation of one’s WM capacity. Any additional item remembered beyond this theoretical number would require some effort, which can be directly quantified as the slope of the iso-effort pattern. We found that the slope of the equivalent WM items and %MVC was numerically similar to one another across experiments (38% vs. 33%; bootstrapped p = 0.32). These consistent patterns suggested that physical and cognitive efforts could be linearly converted from one to another, such that perceived cognitive effort for every additional unit of information would correspond to a relatively constant amount of physical effort approximated by the level of isometric muscle constriction.

Discussion

Overall, data from Experiments 1 and 2 suggest that the same computational principle may bridge perceived effort and utility when cognitive and physical tasks are directly compared, regardless of whether a choice is made before or after task performance. These findings therefore provide some support for the hypothesis that maintains a shared mental representation of “effortfulness” across cognitive and physical domains (Kurzban, 2016; Potts et al., 2018; Shenhav et al., 2017). These findings provide evidence against the claim that cognitive and physical tasks would draw from fully separate pools of resources so that engaging one may not directly affect another (e.g., Feghhi & Rosenbaum, 2019), even though the lack of interaction could occur when the level of competition is low (e.g., when a task is very easy).

Furthermore, the linear pattern in perceived effort between physical and cognitive task loads (Figure 6B) predicts the extent to which engaging one task may directly affect another task. For example, when a visual WM task is performed under concurrent physical exertion, the number of items one can remember in WM should proportionally reduce, with its magnitude predicted by the perceived equivalent WM items related to the level of physical exertion. This core prediction provides a direct test for our model that could not be inferred based on evidence from the existing literature. For example, past research has primarily examined the subsequent effects of physical exertion, such as fatigue (Féry, Ferry, Hofe, & Rieu, 1997), exercise (Lambourne & Tomporowski, 2010; Schmidt-Kassow et al., 2013; Tomporowski, 2003), and other complex physical activities (Hope, Lewinski, Dixon, Blocksidge, & Gabbert, 2012) on cognition. These subsequent effects may be driven by different mechanisms as compared with a dual-task paradigm when cognitive and physical tasks are simultaneously engaged (Park et al., 2021; Quak, Pecher, & Zeelenberg, 2014; van Dijck, Fias, & Andres, 2015). In addition, findings based on concurrent physical and WM tasks have yielded mixed findings that are hard to interpret (Quak et al., 2014; van Dijck et al., 2015), partly due to the lack of quantitative prediction on how physical exertion should impact WM. Here, based on the iso-effort pattern, we predict that increasing concurrent physical efforts during WM retention should lead to a proportional reduction of remembered WM items in a dual-task paradigm.

Experiment 3

To test this core prediction from our model, we asked participants to remember a certain number of color squares over a brief delay (i.e., 1, 2, 3, 4, or 6 colors), while simultaneously holding a hand dynameter with a certain level of grip force (i.e., 20% or 45% MVC). Conceptually, based on the iso-effort patterns outlined in Figure 6B, people can remember 2–3 items almost effortlessly (~ 0% MVC) and 4 items at a moderate level of effort (55%-60% MVC). With linear interpolation, remembering 6 colors in WM would hypothetically require more than 100% MVC in equivalent effort. This iso-effort pattern therefore can be used to predict the conditions in which the competition between cognitive and physical loads is likely to induce a trade-off in task performance. Specifically, if concurrent cognitive and physical tasks indeed tap into some shared effort-related computational processes (see Discussion in Experiment 3), increasing physical exertion from 20% to 45% MVC should lead to a pronounced reduction in concurrent WM task performance at a memory set size that already requires more than a moderate level of effort (e.g., set size > 4). For instance, participants’ WM task performance should reduce more at set size 6 as compared with smaller set sizes (e.g., <= 4). Critically, this reduction should be proportional to the equivalent WM items associated with the increase of a physical load from 20% to 45% MVC. This predictable reduction should hold both across participants (by looking at the magnitude of performance change) and within individuals (by looking at the repeated-measured correlation, Bakdash & Marusich, 2017; Xie, Lu Sing, Martinez-Flores, & Zhang, 2022). Alternatively, if effort-related mechanisms are independent across cognitive and physical domains, the iso-effort pattern should be irrelevant to the decrease in WM task performance under physical loads, although the increase in physical loads could still produce a drop in WM task performance across set sizes due to occasional lapses (Rouder et al., 2008, 2011).

Methods

Participants

A separate group of 20 right-handed college students (20.89 ± 0.55 years old, 16 females, 4 males, and none reporting as others) participated in this three-session experiment (~1/h per session) for course credit and monetary compensation. Based on our modeling results, we expected a sizable reduction in the number of remembered WM items when the concurrent physical load increased from 20% to 45% MVC. We therefore retained the sample size of 20 to detect an effect size of r ~= 0.5 with ~80% statistical power at an alpha level of .05 (Faul et al., 2009). All participants reported normal (or corrected-to-normal) vision and color vision and provided informed consent was given before the study began.

Stimulus and Apparatus

All stimuli, laboratory settings, and apparatus were identical to Experiments 1 and 2.

Procedure

This experiment contains 3 one-hour sessions on different days, each separated by 1–3 days. The structure of Session 1 was the same as the first session of Experiment 1, which included the measurement of MVC, practice trials for the handgrip task and the whole-report WM task, and the choice paradigm used in Experiment 1 with some modifications (see below). In Sessions 2 and 3, participants performed a typical color change detection task (Luck & Vogel, 1997), while trying to hold a hand dynamometer to a certain level of MVC (20% vs. 45%).

Choice Paradigm in Experiment 3.

As previous experiments have suggested that the choice paradigm used in Experiment 1 could provide good estimates of the equivalent WM items for different levels of physical loads with reasonable generalizability (Figure 6B), we adopted a similar protocol outlined in Experiment 1 (Figure 1), with the following exceptions. First, only three levels of hand force (i.e., 20%, 45%, and 90% MVC) were included in the choice paradigm along with 5 different levels of WM set sizes. This was to ensure sufficient variability in physical task loads (20% to 90% MVC), while prioritizing the measurement of equivalent WM items for 20% and 45% MVC physical loads. Second, participants completed 16 trials for each pair of %MVC and #Color comparison, yielding a total of 240 trials across 4 experimental blocks in the first experimental sessions.

Concurrent WM and handgrip task.

Participants completed a WM change detection task under concurrent physical loads in the second and third experimental sessions. The key features of this task remained consistent with those in the previous experiments with the following exceptions. First, to introduce dual-task demands, on each trial, participants were instructed to grasp the hand dynamometer with the left hand to a required level of MVC (20% or 45%, marked on a visual gauge bar for the first 4 seconds) and then maintain above the required level throughout the concurrent visual WM until a test display appeared (Figure 7A). Second, we changed the whole-report WM task to visual change detection task (Luck & Vogel, 1997; Phillips, 1974) to reduce the task demand on motor responses of the WM task response and focus on the direct competition between WM retention and physical exertion. Specifically, during the visual WM task, a fixation circle was first presented on the screen for 800 ms immediately after the offset of the grip exertion gauge. Afterwards, a study array of either 1, 2, 3, 4, or 6 squares (2° × 2° in visual angle) containing distinctive colors was presented for 500 ms, which was followed by a 1,000-ms retention interval with only the fixation circle on the screen and then a test array containing one color square and five placeholders. On half of the trials, the test color was the same as that in the study array at the same location (no-change); on the other half of the trials, the test color was a new color that was not shown in the study array (change). Participants used their right hand to press different buttons on a gamepad (Logitech Precision, Newark, CA, USA), to indicate whether the test has a changed color or not within a maximum of 2,200-ms response time window. Immediately following the response, feedback was given for 1,000 ms. To motivate participants’ engagement in this task, during the feedback period, participants were given 1 task credit point if both the handgrip task (based on the same criteria outlined in previous experiments) and the visual WM task were performed accurately.

Figure 7. Iso-effort patterns from Model 1 predict changes in visual WM task performance under concurrent physical exertion in Experiment 3.

(A) Participants performed a visual WM task under concurrent physical exertion with a low or high physical load (20% vs. 45% MVC). (B) Based on data from a separate experimental session, we estimated the equivalent WM items for 20% and 45% MVC using a similar choice paradigm and the same best-fit model (Model 1) outlined in Experiment 1. (C) Participants remembered a lower number of WM items under high physical load than under low physical loads, only at set size of 6. (D) Across participants, the extent to which physical load could reduce the number of remembered WM items (Cowan’s K difference between 45% and 20% MVC conditions) can be predicted by the shift in equivalent WM items across physical loads. (E) Within each participant, we obtain similar results based on repeated-measures correlation without taking the difference in task measurements between conditions. Each dot in (B) and (C) represents a mean estimate with its error bar representing the standard error of the mean estimate across participants. Each dot in (D) and (E) represents data from one participant. The solid lines represent linear fit across (D) or within (E) participants. The dotted lines in (D) capture 95% confidence intervals of the between-subject linear fit.

All experimental factors (%MVC and WM set size) were randomly intermixed within each experimental block of 20 trials. In addition to a random intertrial interval of ~3 seconds and sufficient time between blocks, we inserted a forced short break (1 min) after every 10 trials within each block to reduce physical strain. Participants completed 10 experimental blocks in each session, yielding a total of 400 trials across two experimental sessions. Participants were informed at the beginning of the study that their task credit points would be converted into a cash bonus at the end of the study. Across the three experimental sessions, participants could potentially earn a total of 640 points (240 points in Session 1 and 400 points in Sessions 2 and 3). The final compensation was staged as $20 for 600~640 points, $15 for 550~309 points, and $10 for the rest, which was not disclosed until the end of the study.

Data Analysis

Participants’ responses in the choice paradigm were analyzed using the same modeling procedure as detailed in Experiment 1 based on Model 1. With this model, we quantified the equivalent WM items for 20% MVC and 45% MVC (see Figure 6A for an example). The difference in this measure between the 20% MVC and 45% MVC conditions should predict the reduction in the number of remembered WM items in the dual-task paradigm. The empirical number of remembered stimuli in the current change detection task (Rouder et al., 2011) was estimated as Cowan’s K (set size × [hit rate – false alarm rate]) on trials with successfully maintained grip exertion to assess the effects of concurrent handgrip on WM performance. These trials capture 97% ± 2% and 86% ± 4% of the total trials for the 20% MVC and 45% MVC conditions, respectively. In particular, for the 97% ± 2% successful trials under the 20% MVC condition, participants’ handgrip exertion was above 20% MVC and also below 45% MVC, ensuring sufficient separation in the grip exertion between these two conditions. We compared the Cowan’s K estimates across experimental conditions based on within-subject statistics (e.g., repeated-measured ANOVA, pair-sample t-test). Furthermore, to evaluate the relationship between equivalent WM items at 20% to 45% MVC and the change in WM performance under concurrent physical exertion, we correlated these measures based on both between-subject correlation and repeated-measures correlation (r_rm, Bakdash & Marusich, 2017; Xie et al., 2022). These two measures are complementary to each other, in that repeated-measures correlation may capture the predicted association shared across participants with greater statistical power without the need to rely on different scores, while between-subject correlation serves as collaborative evidence in the current data with improved generalizability (Rosenthal & Rosnow, 2008b; Xie, Cappiello, Meng, Rosenthal, & Zhang, 2018).

Results

Estimating Equivalent WM Items for physical loads at 20% and 45% MVC

Consistent with the findings from the previous experiments, we found that the model assuming the same negative logarithmic relationship between perceived effort and task utility across modalities (Model 1) provided good fit to participants’ choice behavior in Session 1 (Figure 7B). Based on this model, we then estimated the equivalent WM items for physical loads at 20% and 45% MVC, which were 2.66 ± 0.12 and 3.53 ± 0.10 on average, respectively. These estimates were on part with the findings from the previous experiments (see Figure 6B).

WM Change Detection under Concurrent Physical Exertion

Next, we examined the experimental data from Sessions 2 and 3 to test the prediction that concurrent physical exertion proportionally reduces the number of remembered WM items. As shown in Figure 7C, the number of remembered WM items on trials with successful grip exertion increased as a function of set sizes (F_{(4, 76)} = 124.43, p < .001, η²_p = .79), but did not change significantly across physical load (F_{(1, 19)} = 1.88, p = .19, η²_p = .002). There was a significant interaction effect between WM set size and physical load (F_{(4, 76)} = 3.20, p = .018, η²_p = .011). In line with our predictions, this interaction effect was mostly driven by a significant reduction in the number of remembered item under a higher physical load at set size 6 (i.e., Cowan’s K under 20% vs. 45% MVC: 3.38 ± 0.18 and 2.93 ± 0.17, t₍₁₉₎ = 2.35, p = 0.030, Cohen’s d = 0.53), but not at lower set sizes (p’s > .10).

Of primary interest, we tested the core prediction that the reduction in WM task performance at set size 6 under concurrent physical exertion reflects a direct trade-off in the integrated perceived effort across physical and cognitive domains. As such, the magnitude of this reduction in retained WM items from 20% MVC to 45% MVC should be correlated with the shift of equivalent WM items across these two physical loads. In line with this prediction, we found that the increase in the iso-effort WM load from 20% to 45% MVC in the choice task was significantly correlated with the reduction in WM performance in the dual task at set size 6 across participants (r = −.59, 95% confidence interval: [−.82, −.21], p = .0058, Figure 7D). This association is not limited to the difference scores, as repeated-measures correlation within each participant also demonstrated a consistent finding for WM task measures taken at different levels of physical load (r_rm = −.62 [−.83, −.24], p = .0028, Figure 7E). Collectively, these findings suggest that the model-predicted iso-effort pattern across cognitive and physical tasks can capture both within- and across-subject variations in WM task performance under different physical loads.

Discussion

Our findings suggest that the influence of physical exertion on concurrent visual WM task performance is more prominent for a large WM set size (e.g., set size 6). Critically, this influence follows a resource-rational principle (Kool & Botvinick, 2018; Lieder & Griffiths, 2019; Shenhav et al., 2017), such that engagement in one task can readily lead to a predictable reduction in another task when both task loads are high. This novel observation suggests that concurrent physical exertion competes for a shared computational process required for the retention of multiple WM items.

In particular, as information with a large set size tends to compete for access to WM and interfere with items that were already encoded into WM (Fukuda, Mance, & Vogel, 2015; Konstantinou, Beal, King, & Lavie, 2014; Oberauer & Lin, 2017), the current findings may be attributed to a detrimental effect of physical exertion on information control in WM. According to this view, the locus of competition in the current dual-task paradigm could emerge from the computational parallel in the control component across physical and cognitive tasks (Prinz, 1997; Zwickel, Grosjean, & Prinz, 2009). Although the current power grip manipulation involves less executive control as compared with precision grip (Ehrsson et al., 2000; Guillery et al., 2013, 2017; Kobayashi-Cuya et al., 2018), successful maintenance of the grip exertion may lead to an opportunity cost to the information control in the WM task at large set sizes (Kurzban et al., 2013). As a result, when the number of WM items increases from smaller to larger set sizes (e.g., set size 6), the direct comparison between physical load and cognitive task performance becomes more obvious as supported by a significant physical load by WM set size interaction effect in Experiment 3. Recent evidence showing that concurrent physical exertion could impact control-related task performance in visual search and visual WM tasks with an increased number of distractors appears to be in line with this interpretation (Azer, Xie, Park, & Zhang, 2022; Park et al., 2021). In light of these findings, shared effort-related representations and processes across modalities may be rooted in the mechanisms involved in cognitive control (Kool & Botvinick, 2018; Shenhav et al., 2017).

Some alternative interpretations could not fully account for the current findings. First, as the handgrip task is concurrently performed during WM encoding and retention, our observations in Experiment 3 are unlikely driven by different task demands involved in WM retrieval. Retrieval-related task demand (i.e., the number of recall responses) could be associated with the perceived effort involved in the whole-report paradigm used in the previous experiments (Adam et al., 2015). Here, Experiment 3 resolves this issue by imposing the dual-task demand prior to WM response, rendering retrieval-related processes irrelevant in the direct competition between the WM and handgrip tasks. Furthermore, the use of a single-probe change detection paradigm also ensures that the task features related to making a response to the test array (change or no-change judgment) remains matched across WM set sizes.

Second, findings from Experiment 3 also may not be accounted for by a generic effect of physiological arousal on cognition. Previously, physical exertion has been used as a way to induce physiological arousal (e.g., Schmidt-Kassow et al., 2013; Tomporowski, 2003), which is a critical factor in the interaction between physical action and cognition (Beilock & Carr, 2001; Beilock & Carr, 2005; DeCaro, Thomas, Albert, & Beilock, 2011; Lambourne & Tomporowski, 2010; Tomporowski, 2003). However, these effects are usually more diffused, such that it should lead to a significant main effect of physical load on WM task performance across set sizes. The finding that a higher physical load selectively affects WM task performance at a larger set size (e.g., 6 items) but not at smaller set sizes (e.g., <=4 items) suggests that physical exertion may pose additional computational processes such as interference resistance involved in large WM set sizes (Lee & Grafton, 2015; Park et al., 2021; Zwickel et al., 2009). These additional computational processes may be one of the sources underlying increased perceived effort across modalities (Kool & Botvinick, 2018). Critically, our model can predict the extent to which physical exertion could impact the concurrent WM task performance both across and within subjects (Figures 7D & 7E). Hence, the current dual-task findings in Experiment 3 are more likely to be driven by a competition for the shared computation processes involved in both tasks instead of generic physiological arousal.

General Discussion

This study combines computational and experimental evidence to articulate how perceived effort of temporary information retention in WM is compared to that of transient physical exertion. We find that human observers discount perceived effort associated with WM in the same way as they would discount perceived effort involved in physical exertion when both tasks are directly compared in a choice paradigm (Experiments 1 and 2). As such, physical exertion can lead to a reduction in the number of remembered items in WM in a dual-task paradigm when perceived effort is high for both physical and cognitive tasks (Experiment 3). Although it is almost effortless to remember 2–3 colors in visual WM, remembering any additional color would produce more effort equivalent to ~35% MVC of grip exertion. This is not trivial, considering the amount of physical effort in daily activities. For instance, it takes 20–30% MVC of grip force to hold the steering wheel while driving (Eksioglu & Kizilaslan, 2008) and 5–10% MVC of manual resistant force when using a computer mouse (Visser, De Looze, De Graaff, & Van Dieën, 2004). Consequently, loading WM with a large amount of information, can easily max out the amount of effort that one can tolerate. As a result, people would rather perform a physical task within their tolerated level of effort instead of engaging in a highly demanding cognitive task. These findings, therefore, provide a formal account for how perceived efforts are directly compared across tasks in different modalities to give rise to people’ choice behavior in favoring one task over another.

These computational and experimental findings have several novel implications concerning the relationship between action and cognition. Theoretically, our computational model provides a unified framework to capture the relationships among task load, perceived effort, and task utility across physical and cognitive domains. Through factorial model comparison, we find that the same computational principle can link perceived effort and task utility across modalities. This suggests a shared mental (and potentially neural) representation may underlie “effort” and its associated internal operations (Braver et al., 2014; Kool & Botvinick, 2018). This interpretation is in line with the hypothesis that task difficulty in different domains may be converted into a common currency to allow direct comparison (Potts et al., 2018). However, the nature of this shared representation remains unclear. Previous research has proposed various possibilities, such as metabolic processes (e.g., glucose consumption; Kennedy & Scholey, 2000), time perception (Potts et al., 2018), and the tendency to avoid task failure (error avoidance, Feghhi & Rosenbaum, 2021), etc. Despite this uncertainty, our findings suggest that this shared representation may be computational in nature (Kool & Botvinick, 2018; Shenhav et al., 2017). Our interpretation is that the opportunity cost associated with control-related processes may underlie how perceived effort is calculated and compared across domains (Kurzban et al., 2013; Shenhav et al., 2017). Supporting this conjecture, we find that concurrent physical exertion impacts visual WM task performance only at a large set size, where the task demand on cognitive control is high. Yet, this prediction entails further investigation, such as testing the effect of concurrent physical exert on distractor inhibition in visual WM (Azer et al., 2022).

Empirically, our findings invite reconsideration of the claim that cognitive and physical tasks may draw from separate pools of resources, such that engaging one may not directly impact another (see some discussions in Feghhi & Rosenbaum, 2019). In fact, a submaximal level of grip force can readily reduce attentional vigilance (Button, Behm, Holmes, & Mackinnon, 2004), compromise memory (Tomporowski, Albrecht, & Pendleton, 2017), and even discourage altruistic behaviors (Lockwood et al., 2017). It is striking that simple physical exertion – as simple as isometric muscle contraction (Cain & Stevens, 1971) – could already have such profound mental costs. Our data suggest that this mental cost in part emerges from the direct competition between physical exertion and WM, although the exact locus of competition remains to be specified. Considering the core role of WM in human cognition, future research should articulate how seemingly effortless or even automatic physical exertion (Ehrsson et al., 2000; Eksioglu & Kizilaslan, 2008; Kobayashi-Cuya et al., 2018) can impact our moment-by-moment cognition (Park et al., 2021).

These theoretical and empirical contributions of our findings could not be accounted for by various alternative interpretations, summarized as follows. First, our modeling results could not be explained by temporal discounting, considering that trial duration is strictly controlled in these experiments. Second, our modeling results are unlikely driven by the reward-related processing, as our findings are consistent across Experiments 1 and 2 regardless of whether or not a reward is involved. Third, our dual-task evidence could not be attributed to retrieval demands or physiological arousal in Experiment 3 (see Discussion in Experiment 3). Furthermore, the model identified from Experiments 1 and 2 can adequately predict the experimental outcomes both across and within participants in Experiment 3, which validates a core prediction of our proposed model.

That said, several caveats of the current study entail further investigation. First, it is unclear whether concurrent physical exertion competes with WM primarily for information control or WM storage capacity. This distinction has been previously discussed (Shenhav et al., 2017), because physical and WM tasks could draw on the same pool of limited WM capacity (Kafry & Kahneman, 1977; Navon & Miller, 2002; Tombu & Jolicoeur, 2003) such that physical exertion can reduce the total amount of information one can simultaneously retain in WM. We find this explanation unlikely, given a recent finding that a concurrent physical load mainly reduces control efficiency in visual WM without reducing its overall storage capacity (Azer et al., 2022). Second, our current model only captures one form of cognitive and physical tasks. Future research should generalize these observations to other tasks (e.g., verbal WM and other forms of physical activities), especially under more naturalistic settings to address some applied problems related to ergonomics (Berguer, Chen, & Smith, 2003; Mehta, 2016).

Conclusion

By modeling how visual WM and physical exertion are compared to each other in terms of perceived effort, our data reveal a shared computational principle underlying effort and utility across cognitive and physical tasks. Grounded in the iso-effort relationship, we find that concurrent physical exertion can lead to a predictable reduction in visual WM task performance when perceived effort is high for both tasks. Collectively, these findings add to our understanding of the relationship between cognitive and physical efforts and provide a parsimonious account for how efforts are directly compared across modalities.

Constraints on Generality

Although the current study has attempted to generalize our modeling results across task procedures (e.g., prospective choice to retrospective choice) and experimental paradigms (e.g., whole-report to single-probe change detection), our study involves the use of well-controlled simple stimuli within college students in their early 20s. We expect that our findings can be generalized to populations with a similar demographic profile using similar experimental setups. Yet, considering that aging may lead to changes in both physical and cognitive domains and that stimulus properties could also lead to differences in cognitive engagement (e.g., Are familiar or memorable stimuli less effortful to remember? Xie, Bainbridge, Inati, Baker, & Zaghloul, 2020; Xie & Zhang, 2017b, 2017c, 2018), it is important for future research to test across different age groups (e.g., older participants, Azer et al., 2022) using more naturalistic task stimuli. Furthermore, using a within-subject design, we have no reason to believe that these results would depend on certain characteristics of the participants. That said, how experimental contexts such as participants’ expectations (Rosenthal & Rosnow, 2008a) and emotional states (e.g., Xie, Lu Sing, et al., 2022; Xie, Ye, & Zhang, 2022; Xie & Zhang, 2016, 2018b) may modulate the experimental outcomes remains untested and therefore entails future investigation.

Public Significance Statement.

This study suggests that people would discount cognitive effort in a similar way as they would discount physical effort. It, therefore, highlights a potential shared mechanism underlying the direct competition between physical and cognitive activities in everyday life.