Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks

Abstract

Task-trained recurrent neural networks (RNNs) are widely used in neuroscience and machine learning to model dynamical computations. To gain mechanistic insight into how neural systems solve tasks, prior work often reverse-engineers individual trained networks. However, different RNNs trained on the same task and achieving similar performance can exhibit strikingly different internal solutions, a phenomenon known as solution degeneracy. Here, we develop a unified framework to systematically quantify and control solution degeneracy across three levels: behavior, neural dynamics, and weight space. We apply this framework to 3,400 RNNs trained on four neuroscience-relevant tasks—flip-flop memory, sine wave generation, delayed discrimination, and path integration—while systematically varying task complexity, learning regime, network size, and regularization. We find that higher task complexity and stronger feature learning reduce degeneracy in neural dynamics but increase it in weight space, with mixed effects on behavior. In contrast, larger networks and structural regularization reduce degeneracy at all three levels. These findings empirically validate the Contravariance Principle and provide practical guidance for researchers seeking to tune the variability of RNN solutions, either to uncover shared neural mechanisms or to model the individual variability observed in biological systems. This work provides a principled framework for quantifying and controlling solution degeneracy in task-trained RNNs, offering new tools for building more interpretable and biologically grounded models of neural computation.

1. Introduction

Recurrent neural networks (RNNs) are widely used in machine learning and computational neuroscience to model dynamical processes. They are typically trained with standard nonconvex optimization methods and have proven useful as surrogate models for generating hypotheses about the neural mechanisms underlying task performance [1, 2, 3, 4, 5, 6]. Traditionally, the study of task-trained RNNs has focused on reverse-engineering a single trained model, implicitly assuming that networks trained on the same task would converge to similar solutions—even when initialized or trained differently. However, recent work has shown that this assumption does not hold universally, and the solution space of task-trained RNNs can be highly degenerate: networks may achieve the same level of training loss, yet differ in out-of-distribution (OOD) behavior, internal representations, neural dynamics, and connectivity [7, 8, 9, 10, 11, 12, 13]. For instance, [8] found that while trained RNNs may share certain topological features, their representational geometry can vary widely. Similarly, [7] showed that task-trained networks can develop qualitatively distinct neural dynamics and OOD generalization behaviors.

These findings raise fundamental questions about the solution space of task-trained RNNs: What factors govern the solution degeneracy across independently trained RNNs? When the solution space of task-trained RNNs is highly degenerate, to what extent can we trust conclusions drawn from a single model instance? While feedforward networks have been extensively studied in terms of how weight initialization and stochastic training (e.g., mini-batch gradients) lead to divergent solutions, RNNs still lack a systematic and unified understanding of the factors that govern solution degeneracy [14, 15, 16, 17, 18, 19, 20, 21, 22, 23]. Cao and Yamins [24] proposed the Contravariance Principle, which posits that as the computational objective (i.e., the task) becomes more complex, the solution space should become less dispersed—since fewer models can simultaneously satisfy the stricter constraints imposed by harder tasks. While this principle is intuitive and compelling, it has thus far remained largely theoretical and has not been directly validated through empirical studies.

In this paper, we introduce a unified framework for quantifying solution degeneracy at three levels: behavior, neural dynamics, and weight space (Figure 1). Leveraging this framework, we isolate four key factors that control solution degeneracy—task complexity, learning regime, network width, and structural regularization. We apply this framework in a large-scale experiment, training 50 independently initialized RNNs on each of four neuroscience-relevant tasks. By systematically varying task complexity, learning regime, network width, and regularization, we map how each factor shapes degeneracy across behavior, dynamics, and weights. We find that as task complexity increases—whether via more input–output channels, higher memory demand, or auxiliary objectives, or as networks undergo stronger feature learning—their neural dynamics become more consistent, while their weight configurations grow more variable. In contrast, increasing network size or imposing structural regularization during training reduces variability at both the dynamics and weight levels. At the behavioral level, each of these factors reliably modulates behavioral degeneracy; however, the relationship between behavioral and dynamical degeneracy is not always consistent.

Figure 1: Key factors shape degeneracy across behavior, dynamics, and weights.

Schematic of our framework for analyzing solution degeneracy in task-trained RNNs. We evaluate how task complexity, learning regime, network size, and structural regularization influence degeneracy at three levels: behavior (network outputs), neural dynamics (state trajectories), and weight space (connectivity).

Table 1 summarizes how task complexity, learning regime, network size, and regularization affect degeneracy across levels. In both machine learning and neuroscience, the desired level of degeneracy may vary depending on the specific research questions being investigated. This framework offers practical guidance for tailoring training to a given goal—whether encouraging consistency across models [25], or promoting diversity across learned solutions [26, 27, 28].

Table 1:

Summary of how each factor affects solution degeneracy. Arrows indicate the direction of change for each level as the factor increases. Contravariant factors shift dynamic and weight degeneracy in opposite direction; covariant factors shift them in the same directions.

Factor	Dynamics	Weights	Behavior
Higher Task complexity (contravariant)	↓	↑	↓
More Feature learning (contravariant)	↓	↑	↑
Larger Network size (covariant)	↓	↓	↓
Regularization (covariant)	↓	↓	↓

Our key contributions are as follows:

• A unified framework for analyzing solution degeneracy in task-trained RNNs across behavior, dynamics, and weights.

• A systematic sweep of four factors—task complexity, feature learning, network size, and regularization—and a summary of their effects across levels (Table 1), with practical guidance for tuning consistency vs. diversity [25, 26, 27, 28].

• A double dissociation: task complexity and feature learning yield contravariant effects on weights vs. dynamics, while network size and regularization yield covariant effects. Here, contravariant means that a factor decreases degeneracy at one level (e.g., dynamics) while increasing it at another (e.g., weights), whereas covariant means both levels change in the same direction.

2. Methods

2.1. Model architecture and training procedure

We use discrete-time nonlinear vanilla recurrent neural networks (RNNs), defined by the update rule: ${\mathbf{h}}_t=\tanh ({\mathbf{W}}_h{\mathbf{h}}_{t-1}+{\mathbf{W}}_x{\mathbf{x}}_t+\mathbf{b})$ where ${\mathbf{h}}_t\in {\mathbb{R}}^n$ is the hidden state, ${\mathbf{x}}_t\in {\mathbb{R}}^m$ is the input, ${\mathbf{W}}_h\in {\mathbb{R}}^{n\times n}$ and ${\mathbf{W}}_x\in {\mathbb{R}}^{n\times m}$ are the recurrent and input weight matrices, and $\mathbf{b}\in {\mathbb{R}}^n$ is a bias vector. A learned linear readout is applied to the hidden state to produce the model’s output at each time step. Networks are trained with Backpropagation Through Time (BPTT) [29], which unrolls the RNN over time to compute gradients at each step. All networks are trained using supervised learning with the Adam optimizer without weight decay. Learning rates are tuned per task (Appendix B). For each task, we train 50 RNNs with 128 hidden units. Weights are initialized from the uniform distribution $𝒰(-1∕\sqrt{n},1∕\sqrt{n})$ and hidden states are initialized to be zeros.

In all experiments, we train networks until them reach a near-asymptotic, task-specific mean-squred error (MSE) threshold on the training set (see Appendix B), after which we allow a patience period of 3 epochs and stop training to measure degeneracy. This early-stopping criterion ensures that networks trained on the same task achieve comparable final losses before any degeneracy analysis.

2.2. Task suite for diagnosing solution degeneracy

We selected a diverse set of four tasks designed to elicit distinct neural dynamics commonly studied in neuroscience. The N-Bit Flip-Flop task captures pattern recognition and memory retrieval processes, analogous to Hopfield-type attractor networks that store discrete binary patterns and retrieve them from partial cues [30, 31]. The Delayed Discrimination task models working memory maintenance in classic delayed-response paradigms [32, 33]. The Sine Wave Generation task represents pattern generation, analogous to Central Pattern Generators (CPGs) that produce self-sustaining rhythmic outputs underlying motor control [34], as well as oscillatory activity observed in motor cortex during movement [35]. Finally, the Path Integration task is inspired by hippocampal and entorhinal circuits that build a cognitive map of the environment to track position by integrating self-motion cues [36]. These tasks have also been used in prior benchmark suites for neuroscience-relevant RNN training [37, 38, 8], underscoring their broad relevance for studying diverse neural computations. Below, we briefly describe the task structure and the typical dynamics required to solve each one.

N-Bit Flip-Flop Task

Each RNN receives $N$ independent input channels taking values in {−1, 0, +1}, which switch with probability $p_{\text{switch}}$. The network has $N$ output channels that must retain the most recent nonzero input on their respective channels. The network dynamics form $2^N$ fixed points, corresponding to all binary combinations of ${\{-1,+1\}}^N$. The output range of this task is [−1, 1] and we apply an early-stopping training MSE threshold at 0.001.

Delayed Discrimination Task

The network receives two pulses of amplitudes $f_1$, $f_2\in [2,10]$, separated by a variable delay $t\in [5,20]$ time steps, and must output $\mathrm{sign}(f_2-f_1)$. In the $N$-channel variant, comparisons are made independently across channels. The network forms task-relevant fixed points to retain the amplitude of $f_1$ during the delay period. The output range of this task is [−1, 1] and we apply an early-stopping training MSE threshold at 0.01.

Sine Wave Generation

The network receives a static input specifying a target frequency $f\in [1,30]$ and must generate the corresponding sine wave $\sin (2\pi ft)$ over time. We define $N_{\mathrm{freq}}$ target frequencies, evenly spaced within the range [1, 30], and use them during training. In the $N$-channel variant, each input channel specifies a frequency, and the corresponding output channel generates a sine wave at that frequency. For each frequency, the network dynamics form and traverse a limit cycle that produces the corresponding sine wave. The output range of this task is [−1, 1] and we apply an early-stopping training MSE threshold at 0.05.

Path Integration Task

Starting from a random position in 2D, the network receives angular direction $\theta$ and speed $v$ at each time step and updates its position estimate. In the 3D variant, the network takes as input azimuth $\theta$, elevation $\varphi$, and speed $v$, and outputs updated ($x$, $y$, $z$) position. The network performs path integration by accumulating velocity vectors based on the input directions and speeds. After training, the network forms a map of the environment in its internal state space. The output range of this task is [−5, 5] and we apply an early-stopping training MSE threshold at 0.05.

In our task suite, trained RNNs develop distinct stable dynamical objects: fixed-point (N-Bit Flip Flop, Delayed Discrimination), limit cycle (Sine Wave Generation), and attractor manifold (Path Integration). In Appendix E, we extend our task suite to include a next-step prediction task on the Lorenz 96 chaotic attractors [39], where networks exhibit chaotic dynamical regime.

2.3. Multi-level framework for quantifying degeneracy

2.3.1. Behavioral degeneracy

We define a novel metric for behavioral degeneracy as the variability in network responses to out-ofdistribution (OOD) inputs. We quantify OOD performance as the mean squared error of all converged networks that achieved near-asymptotic training loss under a temporal generalization condition. For the Delayed Discrimination task, we doubled the delay period. For all other tasks, we doubled the length of the entire trial to assess generalization under extended temporal contexts. Behavioral degeneracy is defined as standard deviation of the OOD losses: ${\sigma }_{\mathrm{OOD}}=\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left({ℒ}_{\mathrm{OOD}}^{(i)}-{\overline{ℒ}}_{\mathrm{OOD}}\right)}^2}$, where ${\overline{ℒ}}_{\mathrm{OOD}}$ is the mean OOD loss. While we focus primarily on the temporal generalization condition for behavioral degeneracy since it directly probes RNNs’ sequence processing capacities and their ability to generalize across extended temporal horizons, the same metric can be readily applied to other OOD conditions, such as input noise or external perturbations. In the rest of the paper, we use the term behavioral degeneracy [temporal generalization] to explicitly indicate the OOD condition being tested.

2.3.2. Dynamical degeneracy

We use Dynamical Similarity Analysis (DSA) [40] to compare the neural dynamics of task-trained networks through pairwise analyses. While previous comparison methods mostly focus on geometry of the data [41, 42, 43, 44], RNNs implement computations through time-varying trajectories rather than static representations, and two RNNs exhibiting similar representational geometry can implement distinct dynamical computations, and vise versa. DSA compares the topological structure of the neural dynamics and has been shown to be more robust to noise and better at identifying behaviorally relevant differences than geometry-based comparison method [45]. For a pair of networks $X$ and $Y$, DSA projects their time series of activities to a higher-dimensional space and identifies a linear dynamic operator for each system via next-step prediction. The DSA distance between two systems is then computed by minimizing the Frobenius norm between the operators, up to an orthogonal transformation (rotation and reflection):

$$d_{\mathrm{DSA}}(A_x,A_y)=\underset{C\in O(n)}{\min }{\Vert A_x-C{A}_y{C}^{-1}\Vert }_F,$$

where $O(n)$ is the orthogonal group. We define dynamical degeneracy as the average DSA distance across all network pairs. Additional details on the DSA metric are provided in Appendix F. We note that scale of the DSA distance used to quantify dynamical degeneracy can depend on the choice of DSA hyperparameters. To ensure fair comparison across conditions, we keep all DSA hyperparameters fixed for RNNs trained on the same task. To assess if the neural dynamics across different trained networks are statistically different, we also establish a null distribution by comparing neural trajectories sampled from the same underlying network, see Appendix F.3 for details.

We focus on comparing neural dynamics because RNNs implement computations through time-evolving trajectories rather than static input representations. In addition, we assess representational degeneracy using Singular Vector Canonical Correlation Analysis (SVCCA) [41]. As shown in Appendix G, the four factors that influence dynamical degeneracy do not impose the same constraints on representational degeneracy.

2.3.3. Weight degeneracy

We quantify weight-level degeneracy via a permutation-invariant version of the Frobenius norm, defined as:

$$d_{\mathrm{PIF}}({\mathbf{W}}_1,{\mathbf{W}}_2)=\underset{\mathbf{P}\in 𝒫(n)}{\min }{\Vert {\mathbf{W}}_1-{\mathbf{P}}^{\mathsf{T}}{\mathbf{W}}_2\mathbf{P}\Vert }_F$$

where ${\mathbf{W}}_1$ and ${\mathbf{W}}_2$ are the recurrent weight matrices for a pair of RNNs, $𝒫(n)$ is the set of permutation matrices of size $n\times n$, and ${\Vert \cdot \Vert }_F$ denotes the Frobenius distance. See Appendix F.2 for additional details. For comparing $d_{PIF}$ computed on networks of different sizes, we normalize the above norm by the number of parameters in the weight matrix.

3. Results

3.1. Task complexity modulates degeneracy across levels

To investigate how task complexity influences dynamical degeneracy, we varied the number of independent input–output channels. This increased the representational load by forcing networks to solve multiple input-output mappings simultaneously. To visualize how neural dynamics vary across networks, we applied two-dimensional Multidimensional Scaling (MDS) to their pairwise distances. As task complexity increased, network dynamics became more similar, forming tighter clusters in the MDS space (Figure 3A). This contravariant relationship between task complexity and dynamical degeneracy was consistent across all tasks (Figure 3B). Higher task demands constrain the space of viable dynamical solutions, leading to greater consistency across independently trained networks.

Figure 3: Higher task complexity reduces dynamical and behavioral degeneracy, but increases weight degeneracy.

(A) Two-dimensional MDS embedding of network dynamics shows that independently trained networks converge to more similar trajectories as task complexity increases. (B) Dynamical, (C) weight, and (D) behavioral degeneracy [temporal generalization] across 50 networks as a function of task complexity. Shaded area indicates ±1 standard error.

At the behavioral level, networks trained on more complex tasks consistently showed lower variability in their responses to OOD test inputs (Figure 3D) in the temporal generalization condition. This finding suggests that increased task complexity, by reducing dynamical degeneracy, also leads to more consistent and less degenerate behavior on the temporal generalization condition across networks. Together, the results at the behavioral and dynamical levels support the Contravariance Principle, which posits an inverse relationship between task complexity and the dispersion of network solutions [24].

At the weight level, we found that pairwise distances between converged RNNs’ weight matrices increased consistently with task complexity (Figure 3C). This likely reflects increased dispersion of local minima in weight space for harder tasks. This interpretation is consistent with prior work on mode averaging and loss landscape geometry in feedforward networks, showing that harder tasks tend to yield increasingly isolated minima, separated by steeper barriers [46, 47, 48, 49, 50, 51, 52]. A complementary perspective comes from [53], who introduced the intrinsic dimension—the lowest-dimensional weight subspace that still contains a solution—which can serve as a proxy for task complexity. As task complexity increases, the intrinsic dimension of the weight space expands and each solution occupies a thinner slice of a higher-dimensional space, leading to minima that lie further apart. In Section 3.2, we propose an additional mechanism: an interaction between task complexity and the network’s learning regime that further amplifies weight-space degeneracy.

3.1.1. Additional axes of task complexity

In earlier experiments, we controlled task complexity by varying the number of independent input–output channels, effectively duplicating the task across dimensions. Here, we explore two alternative approaches: increasing the task’s memory demand and adding auxiliary objectives.

Changing memory demand.

Of the four tasks, only Delayed Discrimination requires extended memory, as its performance depends on maintaining the first stimulus across a variable delay. See Appendix D for a quantification of each task’s memory demand. We increased the memory load in Delayed Discrimination by lengthening the delay period. This manipulation reduced degeneracy at the dynamical and behavioral levels but increased it at the weight level, mirroring the effect of increasing task dimensionality (Figure 4A).

Figure 4: Increasing memory demand or adding auxiliary loss changes task complexity, which in turn modulates degeneracy.

In the Delayed Discrimination task, both manipulations reduce dynamical and behavioral degeneracy [temporal generalization] while increasing weight degeneracy. The auxiliary loss also induces additional line attractors in the network’s dynamics, as shown in (C).

Adding auxiliary loss.

We next examined how adding an auxiliary loss affects solution degeneracy in the Delayed Discrimination task. Specifically, the network outputs both the sign and the magnitude of the difference between two stimulus values ($f_2-f_1$), using separate output channels for each. This manipulation added a second output channel and increased memory demand by requiring the network to track the magnitude of the difference between incoming stimuli. Consistent with our hypothesis, this manipulation reduced dynamical and behavioral degeneracy [temporal generalization] while increasing weight degeneracy (Figure 4B). Crucially, the auxiliary loss induced additional line attractors in the network dynamics, further structuring internal trajectories and aligning neural responses across networks (Figure 4C). While the auxiliary loss increases both output dimensionality and temporal memory demand, we interpret its effect holistically as a structured increase in task complexity.

3.2. Feature learning

3.2.1. Task complexity scales feature learning

In deep learning theory, neural networks can either solve tasks using their random features at initialization, or adapt their weights and internal features to capture task specific structure [54, 55, 56, 57]. These are referred to as the lazy learning regime, where weights and internal features remain largely unchanged during training, and the rich learning, or feature learning regime, where networks reshape their hidden representations and weights to capture task-specific structure [54, 58, 59, 55]. As the complexity of a task grows, the initial random features no longer suffice to solve it, pushing the network beyond the lazy regime and into feature learning, where weights and internal representations adapt more substantially. [60, 61]. If more complex task variants, like those in Section 3.1, truly induce greater feature learning, then networks should adapt more from their initializations and traverse a greater distance in the weight space, resulting in more dispersed final weights.

We therefore hypothesize that the increased weight degeneracy observed in harder tasks reflects stronger feature learning within the network. To test this idea, we measured feature learning strength in networks trained on different task variants using two complementary metrics [62, 58]: Weight-change norm: ${\Vert {\mathbf{W}}_T-{\mathbf{W}}_0\Vert }_F$, where larger values indicate stronger feature learning. Kernel alignment (KA): The geometry of learning under gradient descent can be described by the neural tangent kernel (NTK), which captures how weight updates affect the network output. The NTK is defined by $K=\nabla w{\widehat{y}}^{\mathsf{T}}\nabla w\widehat{y}$ where $\widehat{y}$ denotes the network output. KA measures the directional change of the NTK before and after training: $\mathrm{KA}(K^{(T)},K^{(0)})=\frac{\mathrm{Tr}(K^{(T)}{K}^{(0)})}{{\Vert K^{(T)}\Vert }_F{\Vert K^{(0)}\Vert }_F}$. Lower KA indicates greater NTK rotation and thus stronger feature learning.

We find that more complex tasks consistently drive stronger feature learning and greater dispersion in weight space, as reflected by increasing weight-change norm and decreasing kernel alignment across all tasks (Figure 5).

Figure 5: More complex tasks drive stronger feature learning in RNNs.

Increased input–output dimensionality leads to higher weight-change norms $({\Vert \Delta W\Vert }_F)$ and lower kernel alignment (KA). Error bars indicate ±1 standard error.

3.2.2. Controlling feature learning reshapes degeneracy across levels

Our earlier results show that harder tasks induce stronger feature learning, which in turn shapes the dispersion of solutions in the weight space. To test whether feature learning causally affects degeneracy, we used a principled network parameterization known as maximum update parameterization ($\mu P$), which allows stable feature learning across network widths, even in the infinite-width limit [57, 54, 56, 55]. In this setup, a single hyperparameter ($\gamma$) controls the strength of feature learning: higher $\gamma$ values induce a richer feature-learning regime. Under this parameterization, the network update rule, initialization, and learning rate are scaled with respect to network width $N$. For the Adam optimizer, the output is scaled as $f(t)=\frac{1}{\gamma N}{W}_{\text{readout}}\varphi (h(t))$. The hidden state update is scaled as $h(t+1)-h(t)=\tau \left(-h(t)+\frac{1}{N}J\varphi (h(t))+Ux(t)\right)$, where $J_{ij}\sim 𝒩(0,N)$ are the recurrent weights and $\varphi$ is the tanh nonlinearity. The learning rate scales as $\eta =\gamma {\eta }_0$. A detailed explanation of $\mu P$ and its relationship to the standard parameterization is in Appendix K and L. For each task, we trained networks with multiple $\gamma$ values and confirmed that larger $\gamma$ consistently induces stronger feature learning, as evidenced by increased weight-change norm and decreased kernel alignment (Appendix M).

We observed that stronger feature learning reduced degeneracy at the dynamical level but increased it at the weight level. We see that when $\gamma$ is high, networks tend to learn similar task-specific features and converge to consistent dynamics and behavior. In contrast, lazy networks (with small $\gamma$) rely on their initial random features, leading to more divergent solutions across seeds—even though their weights move less overall (Figure 6). This finding aligns with prior work in feedforward networks, where feature learning was shown to reduce the variance of the neural tangent kernel across converged models [60]. At the behavioral level, however, increasing feature-learning strength leads networks to overfit the training distribution (Appendix J.2). We hypothesize that stronger feature learning exacerbates overfitting, increasing both average OOD loss and the variability of OOD behavior across models (Figure 6) [63, 64, 65, 66]. Although stronger feature learning increases behavioral degeneracy [temporal generalization], this may partially reflect overfitting to the training distribution, an effect we highlight in Appendix J.2. Clarifying the mechanistic link between dynamical and behavioral degeneracy [temporal generalization] remains an important direction for future work. In Appendix I, we demonstrate that the observed effects of feature learning on degeneracy both interpolates smoothly within the range of $\gamma$ values and extrapolates beyond the range reported in Figure 6.

Figure 6: Stronger feature learning reduces dynamical degeneracy but increases weight and behavioral degeneracy.

Panels show degeneracy at the dynamical, weight, and behavioral levels (top to bottom). Shaded area indicates ±1 standard error.

3.3. Larger networks yield more consistent solutions across levels

Prior work in machine learning and optimization shows that over-parameterization improves convergence by helping gradient methods escape saddle points [67, 68, 69, 70, 71, 72, 16]. We therefore hypothesized that larger RNNs would converge to more consistent solutions across seeds. However, increasing width also tends to push models towards the lazy regime, where feature learning is suppressed [73, 59, 54, 55, 56]. To disentangle these competing effects, we again use the $\mu P$ parameterization, which holds feature learning strength constant (via fixed $\gamma$) while scaling width. Although larger networks may yield more consistent solutions via self-averaging, this outcome is not guaranteed without controlling for feature learning. In standard RNNs, increasing width often induces lazier dynamics, which can paradoxically increase dynamical degeneracy rather than reduce it. The $\mu P$ setup enables us to isolate the size effect cleanly.

Across all tasks, larger networks consistently exhibit lower degeneracy at the weight, dynamical, and behavioral levels, producing more consistent solutions across random seeds (Figure 7). Our dense sweep over 12 intermediate network sizes from 32 to 512 on the 3-Bits Flip Flop task in Appendix I further confirms the observed effect of network width on degeneracy. This pattern aligns with findings in vision and language models, where wider networks converge to more similar internal representations [74, 75, 41, 76, 77, 65]. In recurrent networks, only a few studies have investigated this “convergence-with-scale” effect using representation-based metrics [74, 78]. Our results extend these findings by (1) focusing on neural computations across time (i.e., neural dynamics) rather than static representations, and (2) demonstrating convergence-with-scale across weight, dynamical, and behavioral levels in RNNs.

Figure 7: Larger networks reduce degeneracy across weight, dynamics, and behavior.

After controlling for feature learning strength ($\gamma =1$ held constant across network widths), wider RNNs yield more consistent solutions across all three levels of analysis. Panels show degeneracy at the dynamical, weight, and behavioral levels (top to bottom). Shaded area indicates ±1 standard error.

3.4. Structural regularization reduces solution degeneracy

Low-rank and sparsity constraints are widely used structural regularizers in neuroscience-inspired modeling and efficient machine learning [4, 79, 80, 81, 82]. A low-rank penalty compresses the weight matrices into a few dominant modes, while an ${\ell }_1$ penalty drives many parameters to zero and induces sparsity. In both cases, task-irrelevant features are pruned, nudging independently initialized networks toward more consistent solutions on the same task. To test this idea, we augmented the task loss with either a nuclear-norm penalty on the recurrent weights $ℒ={ℒ}_{\text{task}}+{\lambda }_{\text{rank}}{\sum }_{i=1}^r{\sigma }_i$, where ${\sigma }_i$ are the singular values of the recurrent matrix, or an ${\ell }_1$ sparsity penalty: $ℒ={ℒ}_{\text{task}}+{\lambda }_{{\ell }_1}{\sum }_i\mid w_i\mid$. We focused on the Delayed Discrimination task to control for baseline difficulty, and observe that both regularizers consistently reduced degeneracy across all levels. Similar effects hold in other tasks (Appendix O, Figure 8) and intermediate regularization strengths (Appendix I).

Figure 8: Low-rank and sparsity regularization reduce solution degeneracy across all levels.

On the Delayed Discrimination task, both regularizers lower degeneracy in dynamics, weights, and behavior. Shaded area indicates ±1 standard error.

4. Discussion

In this work, we introduced a unified framework for quantifying solution degeneracy in task-trained recurrent neural networks (RNNs) at three complementary levels: behavior, neural dynamics, and weights. We systematically varied four factors within our generalizable framework: (i) task complexity (via input–output dimensionality, memory demand, or auxiliary loss), (ii) feature learning strength, (iii) network size, and (iv) structural regularization. We then evaluated their effects on solution degeneracy across a diverse set of neuroscience-relevant tasks.

Two consistent patterns emerged from this analysis. First, increasing task complexity or boosting feature learning produced a contravariant effect: dynamical degeneracy decreased while weight degeneracy increased. Second, increasing network size or applying structural regularization reduced degeneracy at both the weight and dynamical levels—that is, a covariant effect. Here, covariant and contravariant refer to the relationship between weight and dynamic degeneracy—not whether degeneracy increases or decreases overall. For example, task complexity and feature learning reduce dynamical degeneracy but increase weight degeneracy, whereas network size and regularization reduce both.

We also observed that the relationship between dynamical and behavioral degeneracy depends on the varying factor. For instance, stronger feature learning leads to more consistent neural dynamics on the training task but greater variability in OOD generalization This suggests that tightly constrained dynamics on the training set do not guarantee more consistent behavior on OOD inputs. This highlights the need for further empirical and theoretical work on how generalization depends on the internal structure of task-trained networks [83, 84, 85]. This divergence highlights a key open question: how much of behavioral consistency generalizes beyond training-aligned dynamics, and what task or network factors drive this decoupling?

These knobs allow researchers to tune the level of degeneracy in task-trained RNNs to suit specific research questions or application needs. For example, researchers may want to suppress degeneracy to study common mechanisms underlying a neural computation. Conversely, to probe individual differences, they can increase degeneracy to expose solution diversity across independently trained networks [86, 87, 88, 89]. Our framework also supports ensemble-based modeling of brain data. By comparing dynamical and behavioral degeneracy across trained networks, it may be possible to match inter-individual variability in models to that observed in animals—helping capture the full distribution of task-solving strategies [90, 91, 92, 93].

Although our analyses use artificial networks, several of the mechanisms we uncover may translate directly to experimental neuroscience. For example, introducing an auxiliary sub-task during behavioral shaping—mirroring our auxiliary-loss manipulation—could constrain the solution space animals explore, thereby reducing behavioral degeneracy [94]. Finally, our contrasting findings motivate theoretical analysis—e.g., using linear RNNs—to understand why some factors induce contravariant versus covariant relationships across behavioral, dynamical, and weight-level degeneracy.

In summary, our work takes a first step toward addressing this classic puzzle in task-driven modeling: What factors shape the variability across independently trained networks? We present a unified framework for quantifying solution degeneracy in task-trained RNNs, identify the key factors that shape the solution landscape, and provide practical guidance for controlling degeneracy to match specific research goals in neuroscience and machine learning.

Limitations and future directions.

This work considers networks equivalent if they achieve similar training loss. Future work could extend the framework to tasks with multiple qualitatively distinct solutions, to examine whether specific factors bias the distribution of networks across those solutions. Another open question is the observed decoupling between dynamical and behavioral degeneracy: how much of behavioral consistency generalizes beyond training-aligned dynamics, and what task or network factors drive this divergence.

Figure 2: Our task suite spans memory, integration, pattern generation, and decision-making.

Task schematics and representative network trajectories projected onto the top principal components are shown in (A)–(B). The four tasks are: N-Bit Flip-Flop: The network must remember the last nonzero input on each of $N$ independent channels. Delayed Discrimination: The network compares the magnitude of two pulses, separated by a variable delay, and outputs their sign difference. Sine Wave Generation: A static input specifies a target frequency, and the network generates the corresponding sine wave over time. Path Integration: The network integrates velocity inputs to track position in a bounded 2D or 3D arena (schematic shows 2D case).