Motor cortex retains and reorients neural dynamics during motor imagery

Abstract

The most prominent characteristic of motor cortex is its activation during movement execution, but it is also active when we simply imagine movements in the absence of actual motor output. Despite decades of behavioral and imaging studies, it is unknown how the specific activity patterns and temporal dynamics within motor cortex during covert motor imagery relate to those during motor execution. Here we recorded intracortical activity from the motor cortex of two people who retain some residual wrist function following incomplete spinal cord injury as they performed both actual and imagined isometric wrist extensions. We found that we could decompose the population activity into three orthogonal subspaces, where one was similarly active during both action and imagery, and the others were only active during a single task type—action or imagery. Although they inhabited orthogonal neural dimensions, the action-unique and imagery-unique subspaces contained a strikingly similar set of dynamical features. Our results suggest that during motor imagery, motor cortex maintains the same overall population dynamics as during execution by reorienting the components related to motor output and/or feedback into a unique, output-null imagery subspace.

Introduction

As people prepare to execute a skilled action, they often pause beforehand to mentally rehearse and visualize it. For example, a tennis player might imagine hitting an upcoming serve or a pianist might imagine playing a difficult sequence prior to performance. This type of covert motor imagery is constrained to the same performance limits as exist for overt execution. One study showed that the speed with which people were able to imagine performing a sequence of finger movements was limited to their actual overt performance¹. Motor imagery is similarly impacted by neurologic impairment; a lesion to motor cortex leads to an equal slowing of both executed and imagined movements². Conversely, imagery-based practice can improve actual motor function, in some instances offering a comparable performance benefit as standard overt training^3–6. This tight coupling between imagery and actual motor function suggests similar central mechanisms, so that information and experience gleaned from one modality can usefully inform the other.

Primary motor cortex, known mainly for its role in the generation of volitional movement, is also active during covert motor imagery^7–11. In fact, many movement-related brain areas are also active during the mental rehearsal of imagined movements. Premotor and supplementary motor cortices^12,13, anterior cingulate areas¹³ and parietal areas^13,14 all display modulated activity during covert motor imagery. Despite the clear link between imagery and action, we know little about how cortical population activity differs between the two. Previous studies in both monkeys¹⁵ and humans^10,11 show evidence that motor cortex activity is somewhat consistent across volitional states but exhibits clear differences as well. In particular, the study in monkeys by Jiang et al. found that a portion of motor cortex activity could be partitioned into distinct subspaces containing unique responses during either overt hand control of an on-screen cursor or passive observation of cursor movements. This segmentation of activity into orthogonal subspaces appears to be a common motif within motor cortex, the most established example being that of movement preparation^16,17. Preparation is certainly distinct from motor imagery; movement preparation involves “readying” for an imminent overt action, while motor imagery is the covert rehearsal of a complete action. However, during both imagery and preparation, the motor system faces a similar objective: to engage in movement-related processing while avoiding the activation of descending control pathways. In the case of movement preparation, the cortical implementation appears to be well explained by the coordination of a small number of population activity patterns^16,17. Within this framework, activity in motor and/or premotor cortices activate orthogonal neural subspaces during preparation and execution. This orthogonality allows preparatory activity to evolve while avoiding the dimensions that would engage descending pathways and cause overt movement.

Given that imagery, by definition, does not involve overt movement, it seems reasonable to assume that imagery activity in motor cortex, like preparatory activity, somehow avoids dimensions responsible for downstream control. One possibility is that imagery exists only in a subset of the neural dimensions active during overt action. Another possibility is that imagery exists in dimensions completely orthogonal to those for action. However, both of these possibilities are unlikely given the results from Jiang et al.¹⁵, which show strong evidence for a large degree of overlap across volitional states, plus additional subspaces specific to overt (arm movement) and covert (cursor observation) volitional states. Related work by Vargas-Irwin et al.¹¹ and Rastogi et al.¹⁰ also show that despite broad similarities between attempted and imagined movements in the motor cortex of paralyzed humans, different volitional states are still highly discriminable. However, the population analyses used in these studies did not identify the underlying geometry driving the separation in volitional state. The present study sets out to characterize the population-level organization across volitional states, and to identify both the dynamical features that are shared, as well as those that are unique to either overt motor action or covert motor imagery.

Here we use an isometric wrist extension task to examine the relationship between imagery and action in motor cortex. Two participants with tetraplegia due to spinal cord injury participated in the study. Despite having no hand or lower extremity function, both retained residual proximal arm and wrist extension control. We recorded from intracortical microelectrode arrays implanted in the hand and arm areas of motor cortex as they performed either real or imagined isometric wrist extensions to achieve low and high force targets. After reducing the recorded population activity to a low dimensional manifold, we found that it contained three distinct subspaces: (1) a shared space, in which responses were nearly identical during action and imagery, (2) an action-unique subspace that contained significant modulation only during actual force production, and (3) an imagery-unique subspace that contained significant modulation only during imagery. Strikingly, we found that the neural dynamics within the imagery-unique subspace during imagery closely resembled those observed in the action-unique subspace during execution. These unique subspace dynamics also contained elements that did not exist within the shared subspace. From this, we conclude that motor cortex maintains the same overall neural dynamics during imagery as during overt action. However, since the population activity must avoid output dimensions (and also lacks modulation along feedback-related dimensions) during imagery, cortex reorients output or feedback-related responses—contained within the action-unique subspace—into an orthogonal, imagery-unique (output-null) subspace. We propose that the retention of overall neural dynamics structure during imagery provides the motor system with a useful proxy for overt practice.

Results

Motor cortex is active during actual and imagined force

We asked participants to perform isometric wrist extensions within an immobile frame affixed with a force sensor to control the height of a line trace displayed on a monitor in front of them (Figure 1a). During “action” trials, they observed a horizontal bar indicating the required force (either low: ~5N or high: ~40N) and then were required to apply the appropriate force such that the line trace matched the vertical position of the bar, holding the target force for approximately four seconds. During “imagery” trials, the participants kept their hand within the force-sensing apparatus and were asked to imagine producing the same wrist extension forces without actually doing so. On imagery trials, the line trace automatically increased to the target force and then returned to zero. For each session, we collected alternating twelve-trial blocks of action and imagery, resulting in thirty-six total trials of action and thirty-six trials of imagery. We then removed trials with force profiles that deviated significantly from the average (see Methods: Experimental Setup), resulting in approximately 31 ± 4 action trials and 28 ± 4 imagery trials per session (six sessions for P2 and three sessions for P3).

Figure 1.

Motor cortex is active during both imagery and action. (a) Participants placed their hands on a board beneath a load cell and produced either real or imagined wrist extension forces. For all trials, they received visual feedback of either their actual produced force (action trials) or an automated proxy (imagery trials). (b) Locations of microelectrode arrays implanted in the motor cortices. C.S. = central sulcus. (c) Average low force (gray) and high force (black) traces for all sessions (P2: solid, P3: dashed) during action trials (top) and imagery trials (bottom). (d) Average population firing rate modulation in motor cortex (M1) during action trials (top) and imagery trials (bottom). Here “modulation” is calculated as the change in firing rate from trial start. Each trace corresponds to a single session. Lighter traces represent low force trials and darker traces represent high force trials. (e) Average activity for three example channels (P3) during high-force action (blue) and high-force imagery (red) trials. Shading represents 95% confidence bounds, calculated by bootstrapping with 1000 resamples. (f) Maximum modulation during low force (light) and high force (dark) action and imagery for all recorded channels and all sessions (left: P2, right: P3). Shaded bounds represent 95% confidence intervals, calculated by bootstrapping with 1000 resamples.

Both participants successfully achieved and maintained the requested force targets during action trials (Figure 1c, top and Supplementary Figures 1–2), and produced no appreciable force during imagery trials (Figure 1c, bottom and Supplementary Figures 1–2). Throughout the experimental sessions, we recorded activity from the hand and arm areas of motor cortex (Figure 1b). Despite the stark difference in force output between action and imagery trials, we observed modulation in the overall population-wide firing rates for both task types (Figure 1d, top vs. bottom; Supplementary Table 1). Individual channels displayed a wide variety of responses, including a mix of preferential activation during action or imagery (Figure 1e). Some channels displayed similar modulation during both action and imagery (e.g., Figure 1e, channel 183), while others appeared uniquely active during only one task type (e.g., Figure 1e, channels 188 and 19). For each channel, we calculated the maximum modulation during action and imagery by calculating the difference between the 5^th percentile and 95^th percentile firing rates for each task type (imagery or action) and force level. We found that for P2, the average low-force imagery modulation was 75 ± 6% that of low-force action modulation (bounds represent 95% confidence intervals obtained via bootstrapping with 10000 resamples). High-force imagery modulation was 57 ± 4% that of action (Figure 1f, left). For P3, low-force imagery modulation was 59 ± 5% that of action and high-force imagery modulation was 53 ± 6% that of action (Figure 1f, right).

Latent space contains distinct action and imagery subspaces

As a first step towards characterizing the differences between the neural representations of action and imagery, we asked whether a portion of the population activity could be separated into unique dimensions containing only action or imagery activity. For simple motor behaviors, the measured dimensionality of motor cortical activity is typically far lower than the number of recorded neurons^18–23. Thus, as an initial step, we reduced the activity from our recorded populations (176 channels for P2, 192 channels for P3) to a lower dimensional (~36D) latent space. To do this, we first performed principal components analysis (PCA) separately on action and imagery trials, keeping enough dimensions to explain >99% variance for each task. We then combined them into a single, unified space (see Methods: Dimensionality reduction for details). Within this combined space, we observed an incomplete overlap between the dimensions containing action and imagery variance. For example, by performing singular value decomposition (SVD) on action data from P2, we found that the leading eleven dimensions explained >99% of action variance, but only 66% of imagery variance (Figure 2a). Similarly, the leading twelve imagery dimensions explained >99% of imagery variance, but only 70% of action variance (Figure 2b). This discrepancy in the set of dimensions containing meaningful variance for each task suggests that the neural subspaces involved in the two tasks were not fully aligned. We summarized this partial alignment in Figure 2d using the subspace alignment index metric¹⁶, which ranges from 0 (when subspaces are orthogonal) to 1 (when subspaces are fully aligned). The action and imagery tasks showed only moderate alignment, well below what we would expect to be able to resolve if they were actually aligned given the trial-by-trial variability (Figure 2d; alignments calculated across 1000 shuffled datasets were uniformly larger than for the true dataset; p<0.001 two-sided t-test). These results suggest that some portion of the activity inhabited distinct orthogonal subspaces unique to both action and imagery.

Figure 2.

Population activity contains distinct action, imagery, and shared subspaces. (a) An example from P2 showing the percent of total action (blue) and imagery (red) variance explained by principal components computed on action trials for an example session (P2). (b) Same as in (a), but for principal components computed on imagery trials. (c) The same space as in (a) and (b) following an orthogonal rotation to isolate distinct action and imagery variances into unique subspaces. (d) Bar plots showing the alignment index between action and imagery, as well as a label-shuffled control. The alignment index reflects the proportion of variance for one condition that is captured by the leading principal components computed from the opposite condition. Bounds represent 95% confidence intervals through bootstrapping with 1000 resamples. For both participants, the action/imagery alignments were significantly lower than the shuffled label control, p<0.001, two-sided t-test (e) Percent action and imagery variance captured by the action-unique, imagery-unique and shared subspaces following the procedure in (c). (f) Estimates of the dimensionality of the full latent space, shared subspace, and action-unique subspace during action. (g) Estimates of the dimensionality of the full latent space, shared subspace, and imagery-unique subspace during imagery. Shaded vertical histograms in (f) and (g) represent the distribution of dimensionalities across 1000 threshold values from 0.5% to 2%.

Based on this observation, we developed a method to jointly identify the action-unique, imagery-unique, and shared (common) subspaces (see Methods: Subspace separation). The identified transformations from the latent space to the three subspaces were constrained to be fully orthogonal, such that the three subspaces fully spanned the original space (i.e. the percentages in Figure 2c sum to 100%). Thus, the final transformation simply provided a different view of the same underlying latent space such that activity clustered into discrete subspaces with unique task-related variance characteristics (Figure 2e). The action-unique subspace contained high variance responses only during the action task, the imagery-unique subspace contained high variance only during the imagery task, and the shared subspace contained both action and imagery variance. The subspace splitting procedure can be considered an extension of the alignment index concept¹⁶—the proportion of variance explained by the shared subspace is essentially equivalent to the alignment index value. In line with Jiang et al.¹⁵, the subspaces did not correspond to distinct subpopulations within the recordings (unimodal distribution of channel contributions to each subspace, P2: p=0.99, P3: p = 0.99, Hartigan’s dip test; Supplementary Figure 3). Rather, individual channels exhibited mixed selectivity, with the clear condition-specific subspace structure only appearing at the broader population level²⁴.

In addition to total variance explained, we also examined the dimensionality of each subspace (Figure 2f–g). To do this, we first performed a varimax rotation on the activity within each subspace and ranked the resulting components by variance. Compared to PCA, varimax provided a more severe “elbow” in the ranked variances, which in turn gave a more consistent estimate of dimensionality. We then counted the number of dimensions that accounted for more than one percent of the total task variance on all 1000 bootstrapped resamples across trials (see Methods: Feature extraction and subspace dimensionality estimation). We found that the estimated dimensionalities of the action and imagery subspaces were mostly larger than the dimensionality of the shared subspace for both P2 (action-unique vs. shared, p=0.122, imagery-unique vs. shared, p < 0.001; bootstrap with 1000 resamples) and P3 (action-unique vs. shared, p=0.002, imagery-unique vs. shared, p < 0.001; bootstrap with 1000 resamples). Additionally, the estimated dimensionalities of the full space during each task were significantly less than the sum of the shared and unique dimensionalities (P2 action: p < 0.001; P2 imagery: p = 0.001; P3 action: p < 0.001; P3 imagery: p = 0.013). This is because the unique subspaces are characterized by their cross-task variance properties. However, within a single task, there exists a more compact representation of the total population response.

Common dynamics within the shared subspace

The previous analysis of the neural latent space revealed a shared subspace containing activity for both tasks, and two separate subspaces that were differentially active depending on whether the participant exerted force or imagined exerting force. Next, we turn to an examination of how neural activity evolved in time within each of these subspaces.

Projecting trial-averaged activity from both tasks into the shared subspace revealed visually similar temporal profiles between action and imagery for each shared dimension (Figure 3b). To assess the extent of this correlation irrespective of chosen dimension, we performed a Monte Carlo method (see Methods: Monte Carlo sampling) in which we sampled 10000 random unit vectors from the shared subspace (Figure 3a, top). On each draw, we computed the correlation between the action and imagery activity along that dimension. Across all sampled dimensions, we found median correlations of 0.93 (P2; 95% CI: [0.75 0.97]) and 0.88 (P3; 95% CI: [0.64 0.94]), indicating that the shared subspace activity was universally well matched between the two task types.

Figure 3.

Temporal components of action and imagery are similar within the shared subspace and across unique subspaces. (a) Top: Correlations between action and imagery responses for 10,000 random dimensions selected from the shared subspace. Arrows indicate median correlations for each participant. Bottom: Correlations between aligned action-unique and imagery-unique components for 10,000 randomly selected dimensions. Arrows indicate median correlations for each participant. Gray distributions represent the maximal possible correlations if the corresponding aligned imagery dimension is removed on each random draw. (b) Activity within the shared subspace during action (left) and imagery (right). (c) Activity within the action-unique subspace during action (left) and imagery (right) (d) Activity within the imagery-unique subspace during action (left) and imagery (right), following an orthogonal transformation to align to the action-unique responses (see Methods: Action-Imagery subspace alignment). For b-d, thin traces correspond to individual sessions (participant P2), thick lines to cross-session averages. Light and dark traces represent low and high force conditions, respectively. Action-imagery correlation values in c-d correspond to the correlations between action-unique dimensions during action and imagery-unique dimensions during imagery.

Common dynamics across action- and imagery-unique subspaces

The comparison of temporal components between the action-unique and imagery-unique subspaces is less straightforward than for the shared subspace. By construction, the unique subspaces are orthogonal to each other and only contain meaningful activity during their respective tasks, making it futile to compare activity across tasks along a single dimension. Instead, we first found a rotation of the imagery subspace axes that aligned the multidimensional responses observed in the imagery-unique subspace during imagery to those in the action-unique subspace during action. (see Methods: Action-Imagery subspace alignment). Following this alignment procedure, we observed that the action and imagery spaces appeared to comprise a similar set of temporal components (Figure 3c,d), despite existing in orthogonal subspaces of the population space.

We quantified the overall similarity between the multidimensional action-unique and imagery-unique responses by calculating correlations between the action-unique activity on randomly chosen dimensions in the action-unique subspace and the imagery-unique activity along the corresponding (aligned) imagery-unique dimensions (Figure 3a, bottom). Across 10000 randomly sampled dimensions, we found median correlations of 0.93 (P2; 95% CI: [0.84 0.97]) and 0.86 (P3; 95% CI: [0.76 0.94]). To provide context for these values, we also found, for each randomly selected action-unique dimension, the maximally correlated imagery-unique dimension that did not incorporate the aligned dimension (Figure 3a, bottom; gray histograms). These secondary dimensions represent the “next best correlation”, indicating the degree of triviality in the original alignment. Temporal components reflecting nonspecific task timing, for example, could be fairly ubiquitous, and correlates might exist on multiple dimensions. However, we found that removing the aligned imagery-unique dimension consistently and significantly reduced the maximum possible correlation that could be achieved from all other dimensions (P2: p<0.001, P3: p<0.001, paired t-test on 10000 randomly chosen dimensions). As such, activity on aligned imagery dimensions correlated non-trivially with the corresponding action-unique dimension activity. This suggests that the same set of temporal components found in the action-unique subspace also exist during mental imagery, but within an orthogonal imagery-unique subspace.

Distinct unique subspace dynamics

Examining the components across subspaces in Figure 3 revealed that, in addition to the high correlation between action-unique and imagery-unique subspaces, there appeared to be some similarity between responses in the unique subspaces and those in the shared subspace. A strong similarity between these subspaces could mean that the unique subspaces simply recapitulate responses from the shared subspace. Thus, we set out to determine the degree of this dynamic similarity between the unique and shared subspaces.

To assess the degree to which temporal components of activity in the unique subspaces simply recapitulated those from the shared subspace, we used linear regression to reconstruct unique subspace activity from the shared subspace (Figure 4a). Similarly, we found a linear reconstruction of the shared subspace activity from the unique subspaces (Figure 4b). If the unique subspaces were merely reflections of the shared activity, we would expect the quality of these reconstructions to be fairly equivalent. Instead, we found that the reconstructions of unique subspaces were uniformly worse than reconstructions of shared subspace activity from the unique subspaces (Figure 4c). This argues against the possibility of the unique spaces being simple “readouts” of the shared subspace; rather, they contain novel dynamic components.

Figure 4.

Unique subspaces contain novel dynamical features (a) Average action-unique subspace components for P2 (left) and the corresponding best linear reconstruction of each component from the shared subspace responses (right). (b) Average shared subspace components for P2 (left) and the corresponding best linear reconstruction of each component from the action-unique subspace responses. (c) x-axis: the percentage of unique subspace variance than can be explained via linear combinations of shared subspace components. y-axis: the percentage of shared subspace variance that can be explained via linear combinations of unique subspace components. Each dot corresponds to an individual session. (d) left: action-unique dimensions ordered by increasing maximal correlation within the shared subspace during action. right: shared dimensions ordered by increasing maximal correlation within the action-unique subspace during action. Top plots show ranked correlations for both participants, and bottom plots show actual component traces for P2. For each subplot below, the colored traces on the left show the target subspace component and the gray traces on the right show the maximally correlated response from the opposite subspace. (e) As in (d) for the imagery-unique and shared subspaces during imagery. For the top sections of (d) and (e), shaded regions represent 95% confidence intervals obtained via bootstrapping with 1000 resamples. In the bottom sections of (d) and (e), thin traces correspond to individual sessions, thick lines to cross-session averages. Light and dark traces represent low and high force averages, respectively.

To examine the novel dynamics within the unique spaces a bit further, we performed a second analysis in which we identified specific components that exemplified this dynamic novelty (Methods: Assessing dynamic novelty). The result of this analysis showed that for each unique subspace, there existed at least one dimension that had no correlate within the shared subspace (Figure 4d,e). However, even the most novel response from the shared subspace could be fit relatively well from unique-subspace activity.

Unique subspace activity exhibits more complex dynamics

From the analysis in Figure 4, it appeared that the unique subspace components that were least correlated with the shared subspace tended to display a large force-dependent effect. We suspected that the discrepancy in dynamics between the shared and unique subspaces arose largely from differences in the force-related information (Figure 3 also suggests more pronounced and varied force-dependent effects within the unique subspaces than the shared). To test this, we first isolated the force-dependent response within each subspace by subtracting the mean response across both force levels. We then estimated the dimensionality of this resulting force-dependent response within each subspace (see Methods: Force-specific responses) and found that the dimensionality of the force-dependent activity within the shared subspace was lower than in the unique spaces (Figure 5a,b)—though the difference was statistically significant in only one out of the four cases based on bootstrapping with 1000 resamples (P2 action-unique vs. shared, p=0.064; P2 imagery-unique vs. shared, p = 0.17; P3 action-unique vs. shared, p = 0.11; P3 imagery-unique vs. shared, p < 0.001). The unique subspaces appeared to contain both transient and tonic force-dependent responses, whereas the shared space seemingly only exhibited a single tonic response throughout the entirety of the trial duration (Figure 5c,d).

Figure 5.

Action-unique and imagery-unique subspaces contain more complex force-specific responses (a) Estimates of force-related dimensionality in the shared and action-unique subspaces during action. (b) Estimates of force-related dimensionality in the shared and imagery-unique subspaces during imagery. (c) Leading force-dependent components (mean-subtracted) in the action-unique subspace (top) and shared subspace (bottom) during action (P2). (d) Leading force-dependent components (mean-subtracted) in the imagery-unique subspace (top) and shared subspace (bottom) during imagery (P2). Shaded vertical histograms in (a) and (b) represent the distribution of dimensionalities for 1000 threshold values from 0.5% to 2%.

Action-unique activity contains downstream motor commands

Based on the core property of the action-unique subspace—that it is active only during overt action—we hypothesized that it at least in part reflected communication of descending control signals. The second component displayed in Figure 3c, for example, resembles the recorded forces across the action and imagery conditions. To more directly test the relationship between each subspace and the executed force, we attempted to decode moment-by-moment force from both the action-unique and shared subspaces (see Methods: Force decoding). The example traces in Figure 6a show the recorded force (black) and predictions (on held-out data) from the action-unique (blue) and shared subspace (purple) decoders for eight consecutive action trials from P2 (session 5). Across all trials from all sessions, the action-unique subspace decoder outperformed the shared subspace decoder (p<0.001; bootstrap across trials with 1000 resamples), providing further evidence that the action-unique subspace is more closely linked to motor execution.

Figure 6.

Moment-by-moment force can be decoded more accurately from the action-unique subspace than from the shared subspace. (a) Eight example action trials (successive) from the fifth session by P2. Black traces correspond to the actual recorded wrist extension force. Blue and purple traces show cross-validated predictions from decoders (Wiener cascade) trained on action-unique and shared subspace responses, respectively. (b) Total R2 values for the action-unique and shared force predictions. Error bars represent the 95% confidence intervals computed across all trials/sessions.

Discussion

In this study, we examined the relationship between population activity in motor cortex during isometric force production and corresponding covert motor imagery. We found that the low-dimensional manifold activity comprised three orthogonal subspaces: a shared subspace, an action-unique subspace, and an imagery-unique subspace. Activity within the shared subspace accounted for approximately half of the total variance and was nearly identical during both action and imagery. Activity in the action and imagery subspaces, though constructed from completely orthogonal correlation patterns, also contained well-matched sets of temporal responses. Further, the action-unique and imagery-unique activity contained dynamically novel components over the shared space activity, and these novel components appear related to actual or imagined motor output (Supplementary Figure 4).

Because the action-unique subspace we identified only modulates during action (and not during motor imagery), we hypothesize that it is directly involved in generating motor output and possibly receiving sensory feedback. During imagery, output dimensions must be avoided, and there is no incoming somatosensory feedback. In theory, motor cortex could satisfy the constraint of avoiding output dimensions by restricting imagery activity to a lower-dimensional subspace, such that there existed only a shared subspace and an action-unique subspace. However, we instead found that imagery engaged an additional, separate subspace, containing temporal components equivalent to those in the action-unique subspace. This suggests that covertly imagining an action does not simply suppress output activity, but instead reorients into dimensions that do not generate muscle activity (Figure 7). There are multiple potential explanations for why the act of motor imagery should involve creating “dummy” output-related responses in motor cortex. One reason might be that during imagery, cortex is practicing to generate output commands, even though those commands do not actually make it downstream. Even for this one-dimensional task, the action-unique subspace contained approximately six dimensions (Figure 2f), which suggests an output-related space that is more complex than the eventual muscle activity²⁵. In addition to muscle-like responses, (Figure 3c, second component) the action-unique subspace also contains transient responses (Figure 3c, first component), which could reflect indirect control through subcortical areas. There is evidence that downstream motor structures are able to integrate brief, transient activity from cortex to generate sustained muscle output^26,27. The ability to practice producing this multidimensional control signal within a motor-output-null space before generating the actual output commands might explain why mental rehearsal improves subsequent performance on overt motor tasks^{3,4,6,28–30}.

Figure 7.

Hypothesized population-level architecture of motor cortex that enables both action and imagery.

A separate possible reason for the existence of output-like components during imagery is that they are necessary for maintaining the dynamical structure of the entire motor cortical ensemble. There is ample evidence motor cortex activity operates as a dynamical system^23,31,32. The multidimensional population response unfolds predictably from an initial neural state, often dictated by preparatory activity in premotor cortex¹⁶, presumably reflecting intrinsic motor cortical or broader synaptic connectivity. From our results, it appears that activity in the action-unique subspace is dynamically distinct from activity in the shared subspace, and thus is likely to be important for maintaining the dynamical structure of neural population activity (Figure 4). Simply suppressing action-unique activity entirely during imagery would lead to the loss of this dynamic structure in motor cortex. However, recapitulating those components in an orthogonal subspace suppresses output while preserving its dynamical properties, which might be important for stabilizing the behavior of the broader sensorimotor network across different volitional states.

The orthogonality between action and imagery subspaces presumably functions similarly to the orthogonality observed between movement preparation and movement execution^16,17. For both preparation and imagery, restricting the population activity to uniquely non-output dimensions prevents unwanted movement. However, the two processes are distinct in their dynamic relationship to the action. If enough time is allowed, preparatory activity in premotor areas appears to settle into a static neural state³³. From the dynamical systems perspective, this represents a set point, which dictates how the subsequent multidimensional response will unfold during movement execution³⁴. The process of motor imagery, on the other hand, is not an imminent preparation for movement, but rather rehearsal of the entire action. We did not observe strong preparatory responses in this experiment (perhaps due to array placement, Figure 1b), so we do not know how preparation for imagery might relate to preparation for action. Uncovering the full population-level organization of preparation-, imagery-, and action-related activity could help elucidate the processes by which cortex uses both overt and covert processes to improve motor skill.

The concept of covert motor imagery also invokes the related function of action observation. When people or animals observe others performing a motor action, it engages motor-related brain areas in a way similar to self-initiated movement^{15,35,36,36–42}. This correlation between observation and action exists even in the activity of individual cortical neurons^{12,35,42–44}, which supports the notion of a “mirror neuron” network through which the motor system can presumably learn new skills by observing others⁴⁵. It is tempting to assume that observation and imagery/rehearsal are equivalent processes, and that observing an action triggers a person (or animal) to imagine performing the action themselves. Since the vast majority of intracortical observation-based experiments are performed with monkeys, it is often impossible to resolve the degree to which the animal is actively involved in motor imagery. However, recent work in humans with tetraplegia found that observation and imagery are actually not equivalent^10,11, and that it is possible to distinguish those two volitional states from population-level activity in motor cortex. Our observation of orthogonal subspaces containing action-unique and imagery-unique activity mirrors results from Jiang et al.¹⁵, who observed separate subspaces containing action-unique and observation-unique activity. This suggests that although observation and imagery can be considered distinct volitional states^10,11, they might employ similar population-level mechanisms (i.e. orthogonal subspaces for output and non-output conditions). Because the task used in our study was isometric, we could not include a meaningful observation-only condition. In the future, including kinematic limb movements could help identify the degree of overlap (common versus unique subspaces) across a larger range of volitional conditions, including observation, imagery, action, and perhaps even replay during sleep⁴⁶.

While the task-dependent nature of the action and imagery subspaces provides clear insight into their functional roles (e.g. the action-unique subspace includes output-related activity), interpretation of the shared subspace is more difficult. Activity in this subspace was nearly identical for both tasks, suggesting that it represents some sort of higher-level, abstracted task objective. The separation of force levels within the shared subspace argues against the interpretation that it is highly nonspecific and reflects broad subject state processes like arousal or engagement^47,48. The shared responses also do not seem driven by visual feedback, as they consistently lead the executed force (and subsequent visual feedback; Supplementary Figures 5–6). Instead, it appears to contain information related to the specific task goal (i.e. force level). We speculate that activity in the shared subspace corresponds to goal-oriented or “task-intention” signals from higher order brain areas.

The shared subspace in particular may also be responsible for allowing the transfer of learning from covert practice (imagery) to actual motor performance. Work in monkeys found that activity within preparatory subspaces shared across volitional states can indeed help facilitate covert-to-overt learning of visuomotor rotations⁴⁹. Similarly, a recent study in humans showed that movement preparation alone (without overt execution) can drive motor adaptation⁵⁰. From an ethological perspective, the ability to imagine movements is only useful if it can meaningfully inform or assist overt motor control. Our results suggest that such transfer may be made possible in motor cortex by maintaining the full repertoire of population responses during imagery as is present during action. The responses related to motor output are reoriented into an orthogonal subspace, which allows the system to suppress actual motor output without changing the overarching dynamics. Further investigation of these faux-output responses—and their interactions with shared components—may give insight into how covert imagery can be used to drive skill learning or enhance rehabilitation following injury or disease.

Methods

Participants

Two participants (P2 and P3) took part in this study. The participants provided informed consent prior to performing any study-related procedures and were each compensated for the time committed to the ongoing clinical trial. The study was conducted under an Investigational Device Exemption from the Food and Drug Administration and approved by Institutional Review Board at the University of Pittsburgh (Pittsburgh, PA), registered at ClinicalTrials.gov (NCT01894802). The clinical trial is an early feasibility study with a primary outcome of evaluating the safety of an intracortical brain-computer interface for long-term neural recording and stimulation. The work presented here is a scientific effort to understand how neural activity during motor imagery, the basis of brain-computer interface devices, relates to that of overt movement. P2 is a 35-year-old man with tetraplegia caused by C5 motor/C6 sensory ASIA B spinal cord injury. P3 is a 30-year-old man with tetraplegia caused by incomplete C6/C7 ASIA B spinal cord injury. Both participants retain some residual upper arm and wrist control, but no hand function. The manual muscle test scores for wrist extension⁵¹ were 4- for both participants (full range of motion against gravity/ mild resistance). Using ASIA exam sensory testing of the right side⁵², P2 had normal sensation from C2-C4, altered sensation for C5, no sensation for C6, and altered sensation for C7-T1. P3 had normal sensation from C2-C6 and altered sensation from C7-T2.

Both participants had two microelectrode arrays (Blackrock Microsystems, Salt Lake City, UT) implanted in the hand and arm areas of motor cortex (P2: two 88-channel arrays, P3: two 96-channel arrays). They also had two 64-channel arrays implanted in somatosensory cortex⁵³, which were not used for this study. Data collection for P2 occurred approximately four years post-implant, and collection for P3 approximately one year post-implant.

Experimental Setup

For each experimental session, the participants placed their pronated right hands on a board on their lap. We then secured a load cell in a frame attached to the board, positioning it such that it made gentle contact with the top of the hand. At the start of each session, we asked participants to perform one maximal voluntary contraction (MVC). We then set initial low and high force targets based on the peak force observed during MVC (10% and 60%) and asked participants to practice by attempting each force level a few times. If the participants reported concern that the high force target was too high and he would be unable to perform the task without significant pain or fatigue, we lowered it until it reached a comfortable level. This resulted in average low forces of 5%, 8%, 4%, 4%, 4%, and 4% (P2) and 11%, 11%, and 7% (P3), and high forces of 50%, 66%, 50%, 47%, 48%, and 48% (P2) and 51%, 52%, and 38% (P3) MVC. Once the force targets were set, we began the experiment, alternating blocks (12 trials each for P2, 10 trials each for P3) of action and imagery. Participant P2 performed three blocks of action and three blocks of imagery for all sessions. Participant P3 performed four blocks of each for sessions one and two, and five blocks for session three. During both action and imagery blocks, the participants’ hands remained positioned within the force-sensing apparatus (Figure 1a). On each block, we randomly interleaved low and high forces. An audible cue (“gentle” or “firm”) cued the upcoming target, and then at the “go” time, a red bar appeared at the target force level. The participants then attempted to achieve and maintain the target force, using the line trace of exerted force as feedback. Each session always began with action to provide a reference for the subsequent imagery. We excluded trials that contained force traces that significantly deviated from the cued profile. On average, we excluded 6 ± 2 action trials and 8 ± 4 imagery trials (due to non-zero force output) per session.

Data acquisition

We collected neural data via digital NeuroPlex E headstages connected via fiber optic cable to two synced Neural Signal Processors (Blackrock Microsystems, Salt Lake City, UT). The neural signals were filtered using a 4^th order 250 Hz high-pass filter, logged as threshold crossings (−4.5 RMS) and subsequently binned at 50 Hz. These binned counts were then convolved offline with a Gaussian kernel (σ = 200 ms) to provide a smoothed estimate of firing rate.

Dimensionality reduction

We sought to reduce the dimensionality of the neural population recordings by projecting the activity into a low-dimensional space. Principal components analysis (PCA) is a common approach for reducing dimensionality, but simply applying PCA to the combined action+imagery dataset could bias later results; the leading dimensions would preferentially capture action-related variance, since the action task contained higher overall variance. To ensure that the low-dimensional space encompassed both action and imagery responses equally, we first performed PCA separately on the condition-averaged firing rates for each dataset, which resulted in two orthonormal weight matrices, ${\mathit{W}}_{\mathit{\text{action}}}\in {\mathbb{R}}^{N\times D_{\mathit{\text{action}}}}$ and ${\mathit{W}}_{\mathit{\text{imagery}}}\in {\mathbb{R}}^{N\times D_{\mathit{\text{imagery}}}}$. Here N is the number of channels (N = 176 for P2, N = 192 for P3) and $D_{\mathit{\text{action}}}$ and $D_{\mathit{\text{imagery}}}$ are the dimensions needed to capture at least 99% of the corresponding task-related variance. For all sessions of P2, $D_{\mathit{\text{action}}}=[17,16,16,15,15]$ and $D_{\mathit{\text{imagery}}}=\left[201916212220\right]$. For P3, $D_{\mathit{\text{action}}}=[19,18,17]$, $D_{\mathit{\text{imagery}}}=\left[312231\right]$. We then concatenated these two matrices into a new matrix ${\mathit{W}}_{\mathit{\text{action}}+\mathit{\text{imagery}}}\in {\mathbb{R}}^{N\times (D_{\mathit{\text{action}}}+D_{\mathit{\text{imagery}}})}$ and performed singular value decomposition,

$${\mathit{W}}_{\mathit{\text{action}}+\mathit{\text{imagery}}}=\mathit{U}\mathit{\Sigma }{\mathit{V}}^T$$ Equation 1

This procedure provided $\mathit{U}\in {\mathbb{R}}^{N\times (D_{\mathit{\text{action}}}+D_{\mathit{\text{imagery}}})}$, an orthonormal basis spanning both W_action and W_imagery. The resulting space overestimated the actual dimensionality of the combined action and imagery tasks, since there ended up being a great deal of overlap between the two (Figure 2). This approach represented the most conservative choice, allowing for mild dimensionality reduction (which aided in cross-session alignment and computational savings in future analyses) while ensuring that the space contained all (>99%) meaningful variance for both tasks. We projected all data from both tasks into this space by simply multiplying the firing rate estimates by U.

Alignment index shuffled control

To provide a comparison for the alignment index values computed between action and imagery, we performed a shuffled control analysis. For all trials of a given force condition (low or high), we randomly reassigned the action/imagery labels and recomputed the alignment index between the new scrambled “action” and “imagery” conditions. We repeated this 10,000 times for each dataset. Without exception, the shuffled control alignments were always higher than the action/imagery alignments (100%), suggesting that the amount of overlap between action and imagery was lower than what could be trivially explained by trial-by-trial variability.

Cross-session alignment

For each dataset, we performed principal component analysis to reduce the dimensionality from channel-space (176 channels for P2, 192 channels for P3) to a lower dimensional latent space (see Methods: Dimensionality reduction). However, there is evidence that such low-dimensional (manifold) representations of the population activity remain consistent for a single behavior²⁰. To combine datasets from different sessions, it is necessary to first align the low-dimensional spaces⁵⁴. We chose to align the low-dimensional latent activity across sessions for each participant using Generalized Procrustes Analysis (GPA)⁵⁵. GPA iterates to find a multidimensional response common to all datasets and returns the axis transformation (an orthonormal rotation with uniform scaling) necessary to align each dataset with that common response. We found that this approach successfully aligned the responses, achieving a high degree of correlation across sessions (see Supplementary Figure 7). We used the cross-session average response to perform the subspace separation as described below.

Subspace separation

We aimed to identify, if possible, subspaces within the population activity that contained wholly task-specific variance (only variance during action or imagery). To achieve this, we implemented an optimization method that extends the concept of the alignment index¹⁶ to identify orthogonal subspaces containing the “unaligned” responses (i.e. variance during one task that appears in the trailing principal components of the opposite task).

The intuition behind the approach is as follows. Given a dataset containing two tasks (e.g. A and B), we can perform PCA on the neural data from just one of the tasks, ${\mathit{Z}}_A\in {\mathbb{R}}^{M\times N}(M$ time points, $N$ latent dimensions) and identify the leading $D_{\mathit{\text{potentA}}}$ dimensions that capture the vast majority of task A variance (we chose 99% as the cutoff, but other reasonable choices provide nearly equivalent results; see Supplementary Figure 8). This also results in $D_{\mathit{\text{nullA}}}=N-D_{\mathit{\text{potentA}}}$ dimensions, which combined contain insignificant (e.g. <1%) variance for task A and can be considered task A null. However, we also can project activity from task B into this task A null space. If the null dimensions of task A contain a significant proportion of task B variance, we consider the activity within that subspace to be task B-unique. We can perform an equivalent procedure starting with PCA on task B activity to identify task A-unique activity.

The main challenge with this approach is that the null spaces from each task—i.e. $A_{\mathit{\text{null}}}$ (containing B-unique activity) and $B_{\mathit{\text{null}}}$ (containing A-unique activity)—are computed by performing PCA separately on data from different task conditions, and so will not be mathematically orthogonal (due to noise introduced by low-variance components). However, we know that they are in fact functionally orthogonal. This is because A-unique activity resides in the potent space of task A (since it contains significant variance during task A), but the null space of task B. Likewise, B-unique activity resides in the null space of task A, but the potent space of task B. Therefore, since A-unique exists in the potent space of A and B-unique exists in the null-space of A, they must be orthogonal. We were able to obtain orthogonal subspaces containing the unique responses through a simple optimization, as described below for our specific action/imagery case.

We begin with latent activity $\mathit{L}\in {\mathbb{R}}^{M\times D_{latent}}$, containing $M$ time points and $D_{latent}$ dimensions (see Methods: Dimensionality reduction). For convenience, we also split this activity into task-specific data matrices ${\mathit{L}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}\in {\mathbb{R}}^{M_{\mathit{\text{action}}}\times D_{latent}}\mathbf{ }(M_{\mathit{\text{action}}}$ time points from the action task) and ${\mathit{L}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}}\in {\mathbb{R}}^{M_{\mathit{\text{imagery}}}\times D_{latent}}\mathbf{ }(M_{\mathit{\text{imagery}}}$ time points from the action task).

We identified the imagery-null subspace ${\mathit{U}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{n}\mathit{u}\mathit{l}\mathit{l}}\in {\mathbb{R}}^{D_{latent}\times D_{\mathit{\text{imagery}}-\mathit{\text{null}}}}$ by performing PCA on ${\mathit{L}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}}$ and keeping only the $D_{\mathit{\text{imagery}}-\mathit{\text{null}}}$ trailing dimensions containing a total of <1% of ${\mathit{L}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}}$ variance. We then performed PCA on ${\mathit{L}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}{\mathit{U}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{n}\mathit{u}\mathit{l}\mathit{l}}$ to obtain the subspace ${\mathit{V}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n},\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{n}\mathit{u}\mathit{l}\mathit{l}}\in {\mathbb{R}}^{D_{\mathit{\text{imagery}}-\mathit{\text{null}}}\times D_{\mathit{\text{action}}-unique}}$ by keeping the leading $D_{\mathit{\text{action}}-unique}$ dimensions (discarding trailing dimensions that accounted for a total of <1% ${\mathit{L}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$ variance). We then multiplied ${\mathit{U}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{n}\mathit{u}\mathit{l}\mathit{l}}$ and ${\mathit{V}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n},\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{n}\mathit{u}\mathit{l}\mathit{l}}$ to obtain a single orthonormal subspace ${\mathit{Z}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}}\in {\mathbb{R}}^{D_{latent}\times D_{\mathit{\text{action}}-unique}}$, which contained meaningful variance during action and no meaningful variance during imagery. Projecting activity from both tasks, $\mathit{L}$, into this subspace gave data matrix ${\mathit{Y}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}}\in {\mathbb{R}}^{M\times D_{\mathit{\text{action}}-unique}}$, a representation of the action-unique responses. We then used the same method to obtain imagery-unique responses, ${\mathit{Y}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}}\in {\mathbb{R}}^{M\times D_{\mathit{\text{imagery}}-unique}}$.

We used gradient descent via the Manopt toolbox⁵⁶ to identify two orthonormal unique subspaces ${\mathit{Q}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}}\in {\mathbb{R}}^{D_{latent}\times D_{\mathit{\text{action}}-unique}}$ and ${{\mathit{Q}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}}\in \mathbb{R}}^{D_{latent}\times D_{\mathit{\text{imagery}}-unique}}$, which minimized the sum of squared residuals between $\left[{\mathit{Y}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e},\mathit{ }}{\mathit{ }\mathit{Y}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}}\right]$ and $[\mathit{L}{\bullet \mathit{Q}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}\mathbf{ }},\mathit{L}\bullet {\mathit{Q}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}\mathbf{ }}]$, subject to ${\mathit{Q}}_{\mathit{a}\mathit{c}\mathit{t}\mathit{i}\mathit{o}\mathit{n}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}\mathbf{ }}\perp {\mathit{Q}}_{\mathit{i}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\mathit{r}\mathit{y}-\mathit{u}\mathit{n}\mathit{i}\mathit{q}\mathit{u}\mathit{e}}$ (brackets indicate concatenation across dimensions). This optimization reconstructed the action-unique and imagery-unique responses, but ensured that they were contained within orthogonal subspaces.

The optimization above resulted only in subspaces containing task-unique responses. We defined the remaining subspace not spanned by the combination of these two unique subspaces to be the shared space, i.e.

$${{\mathit{Q}}_{\mathit{\text{shared}}}\in \mathbb{R}}^{D_{latent}\times (D_{latent}-D_{\mathit{\text{action}}-unique}-D_{\mathit{\text{imagery}}-unique})}$$

$${{\mathit{Q}}_{\mathit{\text{shared}}}\perp {[\mathit{Q}}_{\mathit{\text{action}}-\mathit{\text{unique}}},\mathit{Q}}_{\mathit{\text{imagery}}-\mathit{\text{unique}}}]$$

This shared subspace necessarily contains only dimensions for which meaningful variance exists during both tasks, or no meaningful variance for either task. Together, all of the subspaces $\mathit{Q}$ can be concatenated to form a single transformation ${\mathit{Q}\in \mathbb{R}}^{D_{latent}\times D_{latent}}$ that represents an orthonormal transformation of the original space. This approach to subspace identification contrasts with existing methods—e.g. demixed PCA⁵⁷—that identify subspaces containing cross-condition variance (Supplementary Figure 9).

Feature extraction and subspace dimensionality estimation

With the latent responses split into shared and unique subspaces, we then found a compact representation of the underlying temporal components within each subspace using a varimax rotation. That is, we performed a varimax rotation on the baseline-centered multidimensional common response (cross-session average) for each subspace. We defined “baseline” as the average response during a 200 ms window at the start of the trial. We then ranked the varimax-rotated version of the multidimensional response by variance. As an orthogonal transformation, this varimax procedure—like PCA—does not change the underlying nature of the multidimensional responses, but rather highlights the separable temporal features within each subspace. For example, the first, second, and fifth dimensions in the action subspace (Figure 3c) are readily interpretable as onset, sustained, and offset responses, respectively.

For dimensionality estimation, as in Figure 2f–g, we calculated the number of dimensions within each subspace that explained more than 1% of variance during the appropriate task (action or imagery) for all of 1000 bootstrapped resamples of the trials. That is, for each bootstrapping run, we re-calculated the condition means using the bootstrapped selection of trials (all sessions), aligned the within-subject averages, and identified unique and shared subspaces. We performed a varimax rotation on each subspace response (we found that the ranked variances following the varimax rotation displayed a greater discontinuity in slope (i.e. “elbow”) than did those resulting from PCA) and ranked the dimensions in decreasing order of variance. The reported dimensionality represents the number of dimensions for which the condition (action or imagery) variance exceeded 1% for all bootstrapped runs. However, true dimensionality is difficult to estimate, and may differ significantly from the values reported here. However, here we were mostly interested in a qualitative comparison of dimensionalities across subspaces, rather than actual values. Different cutoffs and dimensionality estimation approaches provided similar main results, namely that the unique subspaces consistently displayed higher dimensionality than did the shared subspace.

Action-Imagery subspace alignment

Just as the low-dimensional subspace responses had to be aligned to make cross-session comparisons, so too did the action and imagery subspaces. We wanted to compare the similarity of the multidimensional temporal responses between the action and imagery subspaces. However, we could not simply compare, e.g. the first dimension of the action subspace with the first dimension of the imagery subspace since the two subspaces existed in orthogonal subspaces (see Methods: Subspace separation).

To compare the imagery and action subspace responses, we found an orthonormal transformation, Z_im-act, of the imagery subspace that maximized the sum of the squared covariance between action responses in the action subspace and imagery responses in the imagery subspace.

$$\underset{Z_{\mathit{\text{im}}-\mathit{\text{act}}}}{\text{max}}\mathrm{T}\mathrm{r}\left(\right({\mathit{X}}_{\mathit{\text{action}}}^T{\mathit{X}}_{\mathit{\text{imagery}}}{\mathit{Z}}_{im-act}{)}^2)$$ Equation 2

We performed this optimization using the Manopt toolbox in Matlab⁵⁶. Unlike the more commonly used canonical correlation analysis for aligning temporal components^20,58, this rotation is orthonormal and represents a middle ground between maximizing correlation and returning leading components that explain a large amount of within-subspace variance.

Monte Carlo sampling

For the temporal response comparisons in Figure 3a we wanted to quantify the similarity of the multidimensional responses in a way that did not depend on the chosen coordinate frame. For example, we provide in Figure 3b–d the specific correlation values for the displayed dimensions, but an orthogonal rotation of the space—which preserves the actual multidimensional relationship—would result in a different set of correlations.

To provide a coordinate-frame-agnostic quantification of two multidimensional responses, we performed a Monte Carlo sampling-based procedure in which we calculated the correlation between responses on randomly selected dimensions.

For multidimensional data X₁ and X₂ (dimensionality d):

1. Generate a random d-dimensional unit vector u_rand.

2. Project X₁ and X₂ onto u_rand and calculate the correlation between the resulting projections

   $${\mathit{c}}_{\mathit{i}}=\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}({\mathit{X}}_1{\mathit{u}}_{rand},{\mathit{X}}_2{\mathit{u}}_{rand})$$ Equation 3

3. Repeat for 10,000 random vectors

The resulting distribution of correlations c provides an overall picture of the multidimensional correspondence between X₁ and X₂.

Action-Imagery correlation control distributions

The Monte Carlo sampling-based method provided a quantification of the overall correlation between responses in the action and imagery subspaces following the imagery-action alignment (Methods: Action-Imagery subspace alignment). However, we also wanted to include an additional reference distribution that would help provide context for the resulting distribution of correlations (Figure 3a, bottom). Ideally, we aimed to clarify whether the relatively high correlations observed between the action and imagery subspaces were unique to the specific alignment (indicating a true alignment of similar components), or if they simply reflected broad, nonspecific modulation throughout the multidimensional space as a result of e.g. task timing.

The core question that we addressed with our control procedure was: for any given dimension, what is the maximum possible correlation between action and imagery if we remove the aligned imagery dimension? To implement this, we performed an additional optimization on each draw of the Monte Carlo routine (Methods: Monte Carlo sampling).

1. Project imagery subspace activity X_imagery into the (d-1)-dimensional space orthogonal to u_rand (i.e. in the nullspace of u_rand) to obtain X_im-V-null.

   $$\underset{{\mathit{m}}_{\mathit{\text{im}}-V-\mathit{null}}}{\mathrm{maximize}}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}({\mathit{X}}_{im-V-\mathit{\text{null}}}{\mathit{m}}_{im-V-\mathit{\text{null}}},{\mathit{X}}_{\mathit{\text{action}}}{\mathit{u}}_{rand}{)}^2$$ Equation 4

2. Find unit vector m_im-V-null that maximizes the correlation between X_im-V-null and X_actionu_rand

3. Save resulting (positive) correlation

   $${\mathit{c}}_{\mathit{\text{null}},i}=\left|\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}({\mathit{X}}_{im-V-\mathit{\text{null}}}{\mathit{m}}_{im-V-\mathit{\text{null}}},{\mathit{X}}_{\mathit{\text{action}}}{\mathit{u}}_{rand})\right|$$ Equation 5

The distribution of c_null is therefore a strong control, since (unlike c) each element results from an independent optimization routine. However, even with that additional freedom, the values of c_null were consistently lower than those in c. This indicates that the temporal structure of action activity along any given action dimension is uniquely mirrored by imagery activity along the corresponding matched imagery dimension.

Assessing dynamic novelty

To determine the dynamic overlap between unique and shared subspaces (Figure 4), we compared the extent to which the responses within one subspace could be fit via linear combinations of the responses in the other subspace. For example, in Figure 4a, we performed a simple linear regression to find a best fit to the action-unique components from shared subspace responses during action. The condition-specific variance explained by those fits (action variance for action-shared fits, imagery variance for imagery-shared fits) are reported for each session in Figure 4c. We obtained 95% confidence intervals by bootstrapping over trials with 1000 resamples.

To provide a more in-depth analysis of the novel dynamics that seemed to exist within the action- and imagery-unique subspaces, we ranked the components within each unique subspace with increasing maximal correlation with the shared subspace. To do this, we first performed the following optimization:

$$\underset{u}{\text{max}}\frac{\sum (X_{shared}{\left({X_{shared}}^T{X}_{shared}\right)}^{-1}{X_{shared}}^T{X}_{act}\bullet u-X_{act}\bullet u{)}^2}{\sum (X_{act}\bullet u{)}^2}$$ Equation 8

Intuitively, this finds the dimension $u$ within the action-unique subspace activity $X_{act}$ (mean-centered) that results in the poorest linear fit from $X_{shared}$ to $X_{act}\bullet u$. The normal equation within the numerator identifies the optimal fit from $X_{shared}$ to each projected component $X_{act}\bullet u$, and the maximization finds $u$ for which that fit is least successful (lowest variance accounted for). We then projected the action-unique activity $X_{act}$ into the nullspace of $u$ and repeated. By doing this, we were able to assemble a full orthonormal transformation of the action-unique activity in which each successive transformed dimension reflected the next most novel dynamic response (poorest fit from $X_{shared}$). We performed this entire procedure for both unique subspaces as above, as well as for the shared subspace during action and imagery, optimizing with respect to the action-unique and imagery-unique subspaces, respectively.

Force-specific responses

To identify the force-specific responses within each subspace, we first subtracted the mean response across force conditions and then performed varimax on the resulting trajectories. We estimated the dimensionality in a similar way as for the full responses (see Feature extraction and subspace dimensionality estimation). However, instead of using a cutoff based on the percentage of force-specific variance explained, we instead used the same actual variance cutoff as identified from the full condition response. That is, for the action condition we calculated the number of force-specific action-unique and shared dimensions that explained at least 1% of total action variance.

Force decoding

To probe the relationship between the action-unique/shared subspaces and executed force, we employed a simple Wiener cascade decoding model⁵⁹. The Wiener cascade involves a simple linear model with a static nonlinearity, and has been used previously to decode force and EMG responses⁶⁰. The predictions were fully cross-validated using a leave-one-out approach for each session; i.e. on each trial, the model used to predict force was trained on all other trials from that session. We then concatenated all of the cross-validated predictions to report total R².

Data Availability

Given the potential sensitivity concerns, deidentified data from this study are posted on DABI, a repository for data related to the NIH Brain Research Through Advancing Neurotechnologies (BRAIN) Initiative. Data for this specific sub-project can be found at https://doi.org/10.18120/70gm-a975 and are available upon request. A portion of the data included in this paper (action conditions only) was used in a previous publication⁶¹.