Skip to main content
NCBI home page
UKPMC Funders Author Manuscripts logoLink to UKPMC Funders Author Manuscripts
. Author manuscript; available in PMC: 2022 Nov 30.
Published in final edited form as: Neurobiol Learn Mem. 2019 Nov 6;166:107106. doi: 10.1016/j.nlm.2019.107106

Response and sample bridging in a primate short-term memory task

Hadas Sloin 1, Eran Stark 1,*
PMCID: PMC7613866  EMSID: EMS157592  PMID: 31705981

Abstract

Freely-moving rodents can solve short-term memory (STM) tasks using “response bridging” strategies, relying on motor patterns instead of mnemonic functions. This limits the interpretational power of results yielded by some STM tasks in rodents. To determine whether head-fixed monkeys can employ parallel non-mnemonic strategies, we measured eye position and velocity of two head-fixed monkeys performing a delayed response reaching and grasping task. We found that eye position during the delay period was correlated with reach direction. Moreover, reach direction as well as grasp object could be predicted from eye kinematics during the delay. Both eye velocity and eye position contributed to the prediction of reach direction. These results show that motor signals carry sufficient information to allow monkeys to solve STM tasks without using any mnemonic functions. Thus, the potential of animals to solve STM tasks using motor patterns is more diverse than previously recognized.

Keywords: Animal cognition, Delayed response, Mediating strategies, Working memory

1. Introduction

The psychological and neurobiological mechanisms underlying short-term memory (STM) have been studied extensively over the last hundred years (Atkinson & Shiffrin, 1968; Richardson, 2007). While the psychological processes and brain regions involved in STM have been studied mainly in human subjects (Doncic, Falleur-Fettig, & Skotheim, 2011; Richardson, 2007), the neurobiological basis of STM is typically studied in animal models. During the past few decades, non-human primates were used to study neuronal spiking mechanisms underlying STM (Funahashi, Bruce, & Goldman-Rakic, 1989; Fuster & Alexander, 1971; Fuster, Bauer, & Jervey, 1985; Goldman-Rakic, 1995; Kubota & Niki, 1971; Levy & Goldman-Rakic, 2000; Miller, Li, & Desimone, 1993; Romo, Brody, Hernández, & Lemus, 1999). In parallel, behaving rodents have been used to research the biological basis of STM, from the molecular-cellular to the brain-region level (Deadwyler, Bunn, & Hampson, 1996; Dunnett, 1985; Dunnett, Wareham, & Torres, 1990; Li et al., 2018; McQuail et al., 2016; Prusky, Douglas, Nelson, Shabanpoor, & Sutherland, 2004; Roozendaal, 2004; Sloan, Good, & Dunnett, 2006; Yuen et al., 2011).

Various behavioural tasks have been used to study STM in animals, all sharing a similar sample-delay-choice architecture (Dudchenko, 2004) (Fig. 1A). It has been recognized that rodents can solve many STM tasks without relying on STM per se, using “non-mnemonic” strategies (Chudasama & Muir, 1997; Dudchenko, 2004; Dudchenko & Sarter, 1992; Pontecorvo, Sahgal, & Steckler, 1996). Over a century ago, Hunter (1913) trained rats on a delayed match to position task. In each trial, one of three doors was briefly illuminated (sample). After a delay of 1–3 s, the rat was required to select the previously-illuminated door (choice). In most sessions, animals oriented their bodies toward the location of the sample door immediately after the stimulus was presented, maintaining this orientation during the delay phase. This behavior was termed “postural mediation” or “response bridging” (Pontecorvo et al., 1996). Later studies replicated this finding, reporting motor mediating behavior in many rodent STM tasks (Chudasama & Muir, 1997; Dudchenko, 2004; Dudchenko, Talpos, Young, & Baxter, 2013. Dudchenko & Sarter, 1992; Herremans, Hijzen, & Slangen, 1994; Pontecorvo et al., 1996; Sloan et al., 2006; van Hest & Steckler, 1996).

Fig. 1. Neuronal, mnemonic, and motor patterns during short term memory (STM) tasks.

Fig. 1

Open in a new tab

The neuronal mechanisms underlying STM are typically studied using tasks with a canonical sample-delay-choice architecture. The rationale for this design is that delay-phase neuronal activity underlies the STM component (A). However, the appearance of motor patterns that are predictive of the behavioural choice demands an expansion of this scheme. One possibility is that the neuronal activity underlies the STM trace, which in turn generates the motor patterns (B). A second option is that the neuronal activity only generates motor patterns, which bridge between the sample stimulus and correct performance, effectively bypassing the mnemonic requirement (C). Finally, the STM trace and the motor patterns may both be used to generate correct choice behavior (D).

Multiple neuronal mechanisms have been suggested to underlie STM in behaving animals (Constantinidis et al., 2018; Goldman-Rakic, 1995; Lundqvist, Herman, & Miller, 2018; Wang, 2001), including persistent activity (Fuster & Alexander, 1971; Goldman-Rakic, 1995; Romo et al., 1999), neuronal reverberations (Wang, 2001), and synchrony (Constantinidis, Franowicz, & Goldman-Rakic, 2001; Sakurai & Takahashi, 2006). It is generally assumed that neuronal activity during the delay underlies mnemonic processes (Fig. 1A). The same neuronal activity may also generate motor patterns that are predictive of the behavioural choice (Fig. 1B). In such a situation, the observation of motor patterns does not confound the interpretation of the neuronal activity as underlying the mnemonic component. Alternatively, the neuronal activity may only underlie motor patterns which in turn generate the correct response (Fig. 1C). In this case, the mnemonic requirement is bridged by the motor patterns, and any task-related neuronal mechanisms may be false-positive for STM. An intermediate possibility is that the motor patterns do not entirely replace memory but only decrease the mnemonic load. Then, memory-related neuronal mechanisms may be harder to detect, resulting in a false-negative scenario (Fig. 1D). Since a-priori all four options are possible, it is essential to evaluate whether predictive motor patterns are limited to freely-moving rodents or appear in other constellations or species.

Non-human primates are often trained on STM tasks similar to those employed in rodents, such as delayed response (Darusman et al., 2014; Fuster & Alexander, 1971; Takeda & Funahashi, 2004). Like rodents, which orient their body towards the goal location, monkeys may use self-motor patterns to bridge the delay and “maintain” the required choice response. Since most primate STM tasks have been carried out during head fixation, bridging by body orientation is unlikely. However, there are at least two possible types of motor patterns relevant to primate STM tasks, mediated by the limb or the eyes. In some tasks the limbs of the animal are restricted (Romo et al., 1999), and then eye movements may be used for bridging. In other tasks, eye fixation is required (Funahashi et al., 1989; Miller et al., 1993; Spaak, Watanabe, Funahashi, & Stokes, 2017), and then limb movements may be used for bridging. Even when both limbs and eyes are fixated, an animal may use other actuators such as the tail to bridge a delay. The possibility of bridging in head-fixed animals has yet to be addressed.

We found that monkey ocular position and velocity during the delay phase of a STM reach and grasp task can predict reach direction. This is the first evidence that head-fixed animals, and in particular non-human primates, can solve STM tasks using motor mediation. In addition, we show that monkey ocular position and velocity during the delay phase can also predict grasp object. This suggests that motor patterns can facilitate “sample bridging”, the mediation of non-spatial sample stimuli.

2. Materials and methods

The dataset used for this study was previously described in Stark and Abeles (2007), Stark, Asher, and Abeles (2007) and Stark, Globerson, Asher, and Abeles (2008).

2.1. Animals

The data used for the current work were collected from two monkeys (female Macaca fascicularis, monkey D, 2.5 kg; and monkey J, 3.2 kg) trained to perform a delayed response reach and grasp task using their right hand. The task and data collection procedures were described in detail elsewhere (Stark & Abeles, 2007; Stark et al., 2007, 2008) and are briefly summarized below. All animal handling procedures were in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals (1996), complied with Israeli law, were approved by the Ethics Committee of the Hebrew University, and supervised by a veterinarian.

2.2. Behavioral task

During recording sessions, the monkey was seated in a primate chair in an isolated booth with its head fixed and left arm restrained. During each session, six reach directions and two grasp types were used for a total of 12 reach-grasp combinations (Fig. 2A). A trial started when the monkey pressed a button (Ready event). The monkey was required to keep its hand on the central button of the touch pad, but was not required to attend to any visual stimulus or memorize any target. After 500–1000 ms (median, 750 ms), a single object was shown at one location for 200–400 ms (Sample on to Sample off; median, 300 ms), indicating the upcoming reach direction and grasp type (sample). After a delay of 1000–1500 ms (Sample off to Go signal; median, 1250 ms) during which the monkey was required to continue pressing the button and no visual stimulus was provided (delay), a non-informative Go signal prompted the monkey to reach, grasp, and hold the target object without visual feedback (choice). During the delay and movement, a half mirror lit only from above prevented the monkey from seeing its hand or the object. Correct trials were reinforced by a juice reward. A typical session lasted ~2 h during which monkeys completed 379 of these trials (median of 41 sessions; range of 209–585).

Fig. 2. Experimental procedures.

Fig. 2

Open in a new tab

Description of the behavioral task and an example of eye kinematics data. (A) Delayed response reach and grasp task. Top: schematic illustration of task goals; left, touch pad, used for presenting the Ready and Go signal (center), surrounded by goal directions; right, goal objects. In each session, six directions and two (out of three possible) objects were employed. Bottom: the timeline of the behavioral paradigm. In each trial, an object was briefly presented in one of six locations arranged in a virtual circle around the central button of a touchpad. The animal was required to keep its right hand on that button until the Go signal. The left hand was restrained, the head was fixed, and the animal could not see the goal or its hand during the delay and movement. (B) Eye movement data during a single trial. Vertical lines represent behavioral events (same color code as in panel A). The delay phase is highlighted in light blue. Eye position in the horizontal and vertical axes was recorded using an intra-red beam (top). From these data, the eye velocity vector (speed and direction) was calculated (second panel from top). Events in which a peak in the instantaneous speed exceeded one SD from the mean speed were defined as saccades (red lines in third panel from top). Vectors indicate the size and direction of these events (bottom panel).

One control and six task epochs were defined for analyses, all 400 ms long. During the Control epoch, starting 100 ms after Ready, the monkey did not know in which direction it would have to reach and what type of grip it would have to use. During the Sample epoch, starting 50 ms after Sample On, the identity and location of the goal objects were briefly visible, yet no movement was required. Following Sample off, a delay was initiated. The delay was divided into three epochs – Delay1, Delay2, and Delay3. Delay1 started 450 ms after Sample on, and thus at least 50 ms after the target object disappeared. Delay2 started 400 ms after the first delay epoch, and Delay3 started 400 ms prior to the Go signal. Delay2 and Delay3 overlapped on average by 100 ms; in such cases, the data in the overlapping periods were used for both epochs. The Choice1 epoch started 150 ms before the hand left the touch pad, and Choice2 started 100 ms after a Correct grasp.

2.3. Eye movement recordings

Eye movements were recorded using an infra-red beam system tracking movements of the right eye (Oculometer, Dr. Bouis, Karlsruhe, Germany). The horizontal and vertical signals from this system were sampled at 400 Hz and low-pass filtered (40 Hz).

2.4. Classification based on a Gaussian mixture model

The classification method used here is identical to that used in Stark & Abeles, 2007. For each epoch separately, a quadratic Gaussian classifier was trained. Ocular data were organized in an n × 4 matrix X↔, the columns of which were the mean values of each of the four movement parameters (horizontal eye position, vertical eye position, horizontal eye velocity, and vertical eye velocity) during that epoch, and each row represented a single trial. Goals were organized in a vector of class labels (goals) y→ ; the elements of y were integers, k ∈ {1, 2, ..,K} (for directions, K = 6; for objects, K = 2). The classification problem can be then formulated as follows: “Given a new observation x→ (row of X↔), what is the most likely class y (element of y→)?”. The framework of discriminant analysis proceeds as follows. Bayes’ theorem indicates that:

p(y∣x→)=p(x→∣y)p(y)p(x→) (1)

The log is then taken; since p(x→) does not depend on y, it is an additive element identical for all y’s and can be ignored. We thus obtain the general expression for a discriminant score:

d(y∣x→)=logp(x→∣y)+logp(y) (2)

which depends on the class-specific prior p(y) and the conditional probability p(x→∣y). To estimate p(x→∣y), some assumptions on its p.d.f. must be made. Here, we assume that on any single trial, the ocular data are drawn from a class-specific N-dimensional (N = 4) Gaussian distribution. The parameters of that distribution, namely the mean μ→y and the covariance matrix Σ↔y, are both specific to the class (goal) in that trial. Under these assumptions, p(x→∣y) can be written as

p(x→∣y)=12πN|Σ↔y|exp(−12(x→−μ→y)TΣ↔y−1(x→−μ→y)) (3)

Plugging Eq. (3) in Eq. (2) yields the discriminant score for the multi-dimensional Gaussian case,

d(y∣x→)=−12(x→−μ→y)TΣ↔y−1(x→−μ→y)−12log|Σ↔y|+logp(y) (4)

Thus, the discriminant score consists of a data-dependent (ocular activity) part (the first element); and a class-dependent, activity independent part (the last two elements). Given a new sample x→ (trial), the score d(y∣x→) is computed for every possible value of y (classes/ goals). Then, the maximum a-posteriori (MAP) is selected as the classification result for that trial.

In this framework, classification proceeds in three steps. First, the full dataset (n trials) is divided into a training set and a test set. Second, a Gaussian mixture model (GMM) is estimated from the training set; in the case of goal directions, K = 6 so there are six 4D Gaussian distributions. Third, each trial from the test set is assigned K discriminant scores according to Eq. (4), and the MAP is assigned as the classification result for that trial. The process is repeated for all trials in the test set. A confusion matrix is then built (Fig. 5A), tabulating the number of times a trial with label k was classified as 1, 2, ..., K. Since the diagonal of that matrix contains the number of correctly-classified trials, the ratio between the sum of the on-diagonal elements and the total number of trials n is the prediction accuracy A.

Fig. 5. Eye kinematics can predict behavioral choice.

Fig. 5

Open in a new tab

Prediction of behavioral choice from eye movement data. (A) An example of correct and incorrect classifications for decoding goal direction based on eye position and velocity during Delayl (same dataset as used in Fig. 3A). Direction labels correspond to those indicated in Fig. 4. Data include 77–81 trials per goal direction. The diagonal elements of the confusion matrix depict correctly-classified trials. (B) Distribution of accuracy values, generated under the null hypothesis that goal direction does not influence eye kinematics (same dataset as in panel A). (C) Classification accuracy (fraction of correctly classified trials) based on eye kinematics, for each of the seven task epochs. Dashed lines show chance levels. Values were calculated separately for every session (shown by individual dots) and then averaged over sessions. Error bars indicate SEM. (D) Fraction of sessions in which eye kinematics predicted goal direction (left) or object (right) above chance. Significance was evaluated using the permutation test described in panel B. Dashed lines show uncorrected alpha level (0.05), error bars represent SEM (41 sessions), numbers above each bar represent p-values (Binomial test; ***p ≪ 0.001; **p < 0.005), and significant p-values are marked in red.

2.5. Cross-validation and significance of classification results

The procedure described above is statistically sound if three assumptions are met: (1) normality: the data conform to a Gaussian model; (2) stationarity: distributions yielding the training and test data are the same; and (3) data size: the training dataset is sufficiently large to precisely estimate the distributions. In practice, these assumptions rarely hold exactly, and (2) and (3) may even contradict. While the process of dividing the data into training and test sets can be randomized, the size of the training set is always limited. To minimize estimation bias and simultaneously maximize accuracy, we use a leave-one-out (n-fold) cross-validation procedure. Specifically, we build the GMM based on a training set that includes all available data except one trial (i.e. n-1 trials), and use the last trial as the test set. The classification result on that trial is noted. The process is then repeated for every single trial being the test set. The confusion matrix shown in Fig. 5A was built using this procedure.

To evaluate the statistical significance of a given classification result, we use a permutation test, in which we shuffle the labels of the trials (the elements of y→) and repeat the entire n-fold cross-validated procedure. Under the null hypothesis that the class (goal) does not change the multi-dimensional Gaussian distributions (i.e. that p(x→∣y) does not depend on y), classification accuracy A should not change following the permutation of labels. To determine whether this null should be accepted or not, we repeat the shuffling procedure many times (M = 5000), obtaining a distribution of the accuracy under H0 (see example in Fig. 5B). Then, the probability of accepting the null is estimated by the number of cases in which the shuffled accuracy Ashuffled exceeds the observed accuracy A:

prob(H0)=#(Ashuffled≥A)+1M+1 (5)

2.6. Statistical analyses

The dataset contained 41 sessions from two monkeys. Non-parametric tests were used for statistical analyses: Mann-Whitney U test for comparing two unpaired samples, Wilcoxon’s signed-rank test for paired samples or for comparing the median of a sample to zero, and resampling tests for comparing prediction accuracy to chance level prediction. A significance level of alpha = 0.05 was used in all tests, and the Bonferroni method was used to correct for multiple comparisons.

3. Results

3.1. Delay phase eye kinematics differs between goals

Two monkeys performed a STM reach and grasp task (Fig. 2A). In each trial, a sample stimulus was presented at one of six possible directions. The stimulus was one of three objects, each requiring a different type of grasp. Sample presentation was followed by a 1000–1500 ms delay, during which the monkey could not see its hand or the sample stimulus. Following a Go signal, the animal was required to reach at the correct direction and grasp the previously-presented object. In the following, reach direction and object are referred to as goal direction and object.

We examined whether this task could be solved using non-mne-monic motor mediation (“bridging”). Since the subjects (monkeys) were head-fixed and their left arms were restrained, body position was fixed and thus body posture could not be used to bridge the delay. We therefore evaluated whether they could use ocular signals (eye position and velocity) to encode goal direction or object. During each trial, eye position in the horizontal and vertical axes was measured using an infra-red eye tracker (Fig. 2B). From these data, the eye velocity vector (speed and direction) was calculated. Initially, events in which a peak in the speed exceeded a threshold of one SD were defined as saccades (red lines in Fig. 2B). However, smaller (e.g. micro-saccades) or slower (e.g. smooth pursuit) eye movements may contain information (Collewijn & Kowler, 2008; Lisberger, 2015). Thus, to account for various types of eye movements, we employed a total of four eye movement parameters: position in the horizontal axis, position in the vertical axis, velocity in the horizontal axis, and velocity in the vertical axis.

To determine if oculomotor bridging may be employed in the delayed response primate task, we first calculated the characteristic eye position and velocity in all trials in which the same goal direction and object were used. Traces were aligned to the Sample off event and averaged (Fig. 3A). The time-dependent horizontal eye position for right or left goal trials during an exemplary session revealed that the animal oriented its eyes towards the goal direction during the delay (Fig. 3A). Towards the end of the delay, the horizontal eye position returned to the center, consistent with anticipation of the Go signal. Comparison between the goal direction-averaged eye positions revealed a difference spanning the end of the Sample epoch, the entire Delay1 epoch, and the bulk of the Delay2 epoch (from 140 ms before until 705 ms after Sample off; Bonferroni-corrected Mann-Whitney U test, p ≪ 0.001; Fig. 3B). Similar patterns were observed in 36/41 sessions (93%; Binomial test, p ≪ 0.001; Fig. 3C). These differences were not due to visualization, since the sample stimulus was not visible to the subject following Sample off.

Fig. 3. Eye position during the delay phase corresponds with goal direction.

Fig. 3

Open in a new tab

Comparison between eye positions during trials associated with left vs. right goals. (A) Horizontal eye position during trials corresponding to the left or right goal direction, averaged over 38 right and 39 left trials from the same session. All trials correspond to the same goal object. Data are aligned to the Sample off event. Vertical lines represent behavioral events; the delay phase is highlighted in light blue. Each goal direction was associated with a distinct eye trajectory. (B) Comparison of average eye position between left and right goal trials from the same exemplary session. Surprise values are log-transformed p-values (Mann-Whitney U test) and horizontal dashed line indicates the Bonferroni-corrected threshold (alpha level, 0.05). (C) Comparison of eye position between left and right goal trials, averaged over all sessions (n = 41). Horizontal scale is identical in all panels.

To determine whether additional goal stimuli corresponded with unique eye kinematics, we examined horizontal and vertical eye position during the Delay1 epoch for all goals in an exemplary session. We found that each goal direction (averaged over goal objects) was typically associated with a distinct position, mirroring the spatial arrangement of the goal directions (Fig. 4). Furthermore, each goal object (averaged over goal directions) corresponded to a distinct eye position.

Fig. 4. Delay phase eye position differs between goals.

Fig. 4

Open in a new tab

Statistics of horizontal and vertical eye positions, for each goal direction and object during the Delay1 epoch. Data are from 473 trials from a single session (same dataset as used in Fig. 3A). The grey “ + “ sign represents the origin (neutral eye position). Each ellipse represents data covariance: the center indicates the mean, the long and short axes indicate the SDs, and the angle represents the correlation between the two parameters (horizontal and vertical position).

3.2. Delay phase eye kinematics can predict behavioral choice

To objectively evaluate the potential of eye position and velocity to encode goal directions and objects in all available sessions, we used a decoding approach. Separately in each session, for each of the seven behavioral epochs, we trained two separate classifiers: one for predicting goal directions and another for goal objects (total 14 different classifiers per session). Each classifier was trained on all trials except one, and tested on the last trial; this process was repeated for every single trial and the results were averaged over all trials (n-fold cross-validation). An example of correct and incorrect classifications for decoding goal direction based on eye position and velocity during the first delay epoch (Delay1) can be observed in Fig. 5A. In that session, goal direction was classified correctly in 50.7% of the trials, chance level being 16.7%. To determine whether this result is consistently above chance, we generated a distribution of classification accuracy values under the null hypothesis that goal direction does not influence eye kinematics by shuffling goal labels (permutation test). The probability of rejecting the null was estimated by the tail of the distribution (p < 0.001 in that case; Fig. 5B).

For each of the seven epochs and for every goal type (direction or object), classification performance was quantified by accuracy (fraction of correctly-classified trials) and then averaged over all sessions (n = 41 from two subjects). During the Control epoch, goal direction was predicted correctly in 16.8% of the trials (chance level is 16.7%). This is compared to 49%, 34% and 31% during Delay1, Delay2, and Delay3 epochs (Fig. 5C, left). Eye kinematics predicted goal direction above chance in 95–98% of the sessions during the delay phase epochs (p ≪ 0.001 for all epochs except Control, Binomial test; Fig. 5D, left). Thus, eye kinematics could predict goal direction throughout the delay phase, during which no goal was visible and no action was required.

A similar analysis carried out for the goal objects yielded correct prediction in 54% and 53% of the trials during the Delay1 and Delay2 epochs (chance being 50%; Fig. 5C, right). Eye kinematics predicted goal object above chance in 39% and 17% of the sessions during these epochs (Delay1: p ≪ 0.001; Delay2: p = 0.004, Binomial test), but at chance level during the Control and Delay3 epochs (p = 0.61 and p = 0.15; Binomial test; Fig. 5D, right). Hence, eye kinematics could predict goal object during the first two epochs of the delay.

3.3. Both eye position and velocity contribute to goal direction decoding

To determine whether eye position, velocity, or both contributed to goal direction decoding accuracy, we trained two new sets of classifiers: one that used only eye position information (horizontal and vertical), the other using only eye velocity. To determine the contribution of eye position beyond that provided by eye velocity, we compared the results yielded by the classifiers based on the full data to those of the classifiers based on the velocity data alone.

For the Control epoch, the improvement index (Afull – Avelocity)/ (Afull + Avelocity) was negligible (– 0.0005; p = 0.63, Wilcoxon’s signed rank test comparing median to zero). However, for all other epochs, eye position made a considerable contribution (Fig. 6A). For instance, during the Delay1 epoch, the improvement index was 0.27 (p ≪ 0.001 for all six non-control epochs). An equivalent metric was used to quantify the non-redundant contribution of eye velocity: (Afull – Aposition)/(Afull + Aposition). This analysis indicated that the eye velocity made a considerable contribution during the Sample (p ≪ 0.001) and Delay1 (p = 0.004) epochs (Fig. 6B). However, even during these epochs, effect sizes were smaller than those observed for position: for instance, during Delay1, the improvement index was 0.27 for position but only 0.035 for velocity (paired signed rank test, comparing velocity and position effects: p ≪ 0.001 for all epochs except Control: p = 0.61). Together, these results show that while eye position dominates, both eye position and velocity contain independent information useful for goal direction classification.

Fig. 6. Eye position and velocity both contribute to goal direction decoding.

Fig. 6

Open in a new tab

Improvement of goal direction classification accuracy when employing a classifier based on both eye position and velocity, compared to a classifier based on only one of these parameters. (A) Improvement index for position. Negative contribution values indicate that relying on the tested parameter decreases classification accuracy. (B) Improvement index for velocity. In both panels, error bars represent SEM over 41 sessions, numbers above each bar represent p-values, and significant values are marked in red (Wilcoxon’s signed-rank test comparing to zero; ***p ≪ 0.001; **p < 0.005; *p < 0.05).

4. Discussion

We showed that in head-fixed monkeys, delay phase oculomotor signals contain enough information to correctly solve an STM task. Eye position during the delay was correlated with goal direction. Moreover, eye position and velocity during the delay could predict goal directions and objects. Both eye position and eye velocity contributed to the prediction of goal direction. These results suggest that oculomotor signals can be used by monkeys to mediate the delay phase of a STM task without relying on STM per se.

Previous studies have shown that while performing STM tasks, freely-moving rodents often demonstrate mediating behaviors that can assist correct performance without using any mnemonic functions (“response bridging”; Chudasama & Muir, 1997; P. Dudchenko & Sarter, 1992; Hunter, 1913; Pontecorvo et al., 1996). The present results show that motor mediation in STM tasks is more diverse than previously recognized, in terms of (1) animal species (rodent, primate); (2) subject degrees of freedom (freely-moving, head-fixed); (3) type of motor mediation (postural, kinematic); and (4) mediated content (response-related, abstract).

4.1. Species, degrees of freedom, and type of mediation

First, the current results show that motor mediating behaviors are not exclusive to rodents and can occur in non-human primates. Second, the monkeys were head-fixed, their left arm was restrained, and they were required to press a button using their right hand throughout the delay. This indicates that bridging behaviors are not exclusive to freely-moving animals and can be observed when an animal is head-fixed. Third, the main contribution to goal classification accuracy was made by eye position. However, eye velocity – the temporal derivative of position – made an independent contribution to classification. This indicates that a static “lookup table” can be extended by dynamical aspects of the motor apparatus, increasing the coding potential of motor mediation.

4.2. Sample bridging

Bridging behavior mediating the goal response was described previously: in delayed non-match and match to position tasks, rodents may bridge the delay by orienting their body to the goal position, as if preparing to respond once the delay ends (Chudasama & Muir, 1997; Dudchenko & Sarter, 1992; Pontecorvo et al., 1996). Similarly, in the current study, the monkeys oriented their eyes towards the goal direction, perhaps in preparation for reaching to the same direction. However, eye position is not directly connected to the choice response. While rodent body orientation is performed by the same actuators creating the correct response (Chudasama & Muir, 1997), the reach and grasp task only necessitated forelimb movements and no specific oculomotor output was required. Furthermore, animals could not see their limbs or the goals during the delay and choice epochs. Therefore, the coding of goal information by oculomotor signals represents a more abstract form of bridging, in which the bridged content is not necessarily the beginning of the response itself but rather some representation of it. Abstraction of information is even more prominent when considering that the delay phase oculomotor signals could also predict the goal objects. Goal objects are independent of location: any object could appear in any direction. Thus, the fact that delay phase oculomotor signals could be used to predict goal object suggests a higher level of abstraction, connecting an object with ocular location in space or/and movement.

The abstraction of goal stimuli observed here raises the possibility that any stimulus, including perhaps location-independent stimuli such as odor or texture, could be mediated by associating a stimulus with a motor pattern. These considerations suggest a newly described of bridging behavior, in which a motor pattern is linked with a non-spatial stimulus not directly associated with a specific response. We term this potential phenomenon Sample bridging.

In sample bridging, the memory trace of a sample can be replaced with associations engraved in long-term memory. In such a scenario, at the end of the delay the subject does not have to recall the sample stimulus from STM, but rather to “look up” the stimulus associated with a motor pattern. It is not known whether animals employ sample bridging intentionally. Such behaviors may develop spontaneously, based on a strong association between certain motor patterns and stimuli, created during the learning phase of an STM task. In other words, in response bridging, the motor pattern mediates the response required at the choice stage (Fig. 7, left). In sample bridging, however, it is not the goal response that is being mediated, but the sample stimulus itself (Fig. 7, right). In delayed response tasks, such as used here, the sample and choice stimuli are the same. Therefore, it is not clear whether the sample or the response is bridged. Yet if sample bridging exists, it implicates STM tasks which are immune to response bridging (Pontecorvo et al., 1996), such as delayed match and non-match to sample and delayed comparison tasks (Dudchenko, 2004; Dudchenko et al., 2013; Fassihi, Akrami, Esmaeili, & Diamond, 2014).

Fig. 7. Sample and response bridging in behaving animals.

Fig. 7

Open in a new tab

Cartoon examples of response and sample bridging in animals performing STM tasks. Left: Response bridging. A delayed nonmatch to position task can be bridged by moving a limb or a finger used for lever pressing (sample lever in red, choice lever in white). Right: Sample bridging. A delayed-non match to sample task can be bridged by associating each sample with a unique configuration of body parts not directly related to the task, such as the eyes or the tail.

4.3. Approaches to minimize bridging

Together, previous and the current results show that motor patterns during the delay contain useful information which enables predicting a correct choice. This indicates that animals can solve STM tasks using non-mnemonic strategies. It does not, however, mean that animals actually do solve tasks in that manner (Fig. 1C). Mediating behaviors could be performed in parallel to STM (Fig. 1D) and/or may be an externalized manifestation of the STM maintenance process (Fig. 1B). Nevertheless, the fact that STM tasks can be solved using non-mne-monic strategies raises a concern in terms of the experimental design and interpretation of results yielded by STM tasks. For this reason, it is imperative to design behavioral tasks which minimize the possibility of bridging.

As pointed out by Pontecorvo et al. (1996), the use of response bridging can be completely prevented by employing tasks in which the correct response cannot be predicted before the choice phase. Tasks fulfilling this requirement include (1) multiple-choice delayed nonmatch to position (Kim et al., 2015; Spellman et al., 2015); (2) sensory delayed non-match and match to sample (Fuster & Alexander, 1971; Liu et al., 2014; Moore, Schettler, Killiany, Rosene, & Moss, 2012; Otto & Eichenbaum, 1992); and (3) delayed comparison (Akrami, Kopec, Diamond, & Brody, 2018; Fassihi et al., 2014; Hernández, Salinas, García, & Romo, 1997; Romo et al., 1999). However, since the inability of a subject to predict a correct response is not sufficient to prevent sample bridging, such tasks do not fully prevent the potential use of non-mnemonic strategies.

A second approach, often employed in studies with head-fixed animals, is the use of bodily restriction. Restriction reduces the diversity of possible motor mediation strategies but cannot prevent them. Fixation can prevent motor mediation by the fixated actuator. For instance, ocular fixation during the delay (e.g. as used by Funahashi et al., 1989) suffices to prevent oculomotor mediation. Likewise, application of muscle relaxants to the ocular muscles would prevent ocular bridging. However, unless fully paralyzed, an animal can still perform motor mediation with any unrestricted actuators such as fingers, legs, or tail movements.

Full protection from sample and response bridging might prove considerably difficult. However, we suggest addressing this issue by making bridging behavior less “economic” than the usage of mnemonic functions. A transition from bridging to mnemonic strategies was previously shown by Chudasama and Muir (1997), who reported that rats which performed bridging at short delays, abandoned bridging as the delay elongated. We thus suggest two additional approaches to increase the difficulty of bridging behavior.

The first approach is motor interference. If the animal is involved in motor behavior, bodily mediation becomes harder. The more distracting the motor behavior, the lower the chances of motor bridging. To efficiently prevent bridging, motor interference must be prominent. For example, in the delayed non-match to position task developed by Dunnett (1985), rats were required to poke a centralized port during the delay to prevent postural mediation. However, later studies showed this is insufficient for preventing postural mediation (Chudasama & Muir, 1997; Dudchenko & Sarter, 1992; Herremans et al., 1994; van Hest & Steckler, 1996). We thus suggest the employment of more engaging motor distractors. Freely-moving animals may be required to run on a treadmill with obstacles (Asante et al., 2010). Head-fixed animals could be required to engage in a whole-body motor distractor (Sofroniew, Cohen, Lee, & Svoboda, 2014).

A complementary approach for minimizing the use of non-mne-monic strategies is to use an enlarged stimulus pool. In the delayed response task employed here, subjects could associate one goal object with leftward eye position and another with rightward position. However, if the stimulus pool would have consisted of many different non-spatial goals, each additional stimulus would have required the animal to employ another motor pattern. The larger the stimulus pool, the higher the diversity of the motor patterns that must be developed for efficient bridging, reducing the chances that the animal will adopt a bridging strategy. As a radical example, Moore et al. (2012) trained monkeys on a delayed non-match to sample task with 600 different objects. The rationale is that the stimulus pool should be very large, akin to the dictionary of English nouns used in human STM free recall and word span tasks (Dudukovic, 2017; Richardson, 2007).

Acknowledgements

This work was supported by an ERC Starting Grant #679253. We thank Alexander Taranavsky, Amir Levi, Lidor Spivak, Ronni Cohen, Roni Gattegno, Shir Sivroni, Shirly Someck and Yaniv Sela for constructive comments.

Footnotes

Declaration of Competing Interest

The authors declared that there is no conflict of interest.

References

  1. Akrami A, Kopec CD, Diamond ME, Brody CD. Posterior parietal cortex represents sensory history and mediates its effects on behaviour. Nature. 2018;554(7692):368–372. doi: 10.1038/nature25510. [DOI] [PubMed] [Google Scholar]
  2. Asante CO, Chu A, Fisher M, Benson L, Beg A, Scheiffele P, Martin J. Cortical control of adaptive locomotion in wild-type mice and mutant mice lacking the ephrin-Eph effector protein alpha2-chimaerin. Journal of Neurophysiology. 2010;104(6):3189–3202. doi: 10.1152/jn.00671.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Atkinson RC, Shiffrin RM. Human memory: A proposed system and its Control processes. Psychology of Learning and Motivation-Advances in Research and Theory. 1968;2(C):89–195. doi: 10.1016/S0079-7421(08)60422-3. [DOI] [Google Scholar]
  4. Chudasama Y, Muir JL. A behaviouval analysis of the delayed non matching to position task: The effects of scopolamine, lesions of the fornix and of the prelimbic region on mediating behaviours by rats. Psychopharmacology (Berl) 1997;134(1):73–82. doi: 10.1007/s002130050427. [DOI] [PubMed] [Google Scholar]
  5. Collewijn H, Kowler E. The significance of microsaccades for vision and oculomotor control. Journal of Vision. 2008;8(14):20. doi: 10.1167/8.14.20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Constantinidis C, Franowicz MN, Goldman-Rakic PS. Coding specificity in cortical microcircuits: a multiple-electrode analysis of primate prefrontal cortex. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience. 2001;21(10):3646–3655. doi: 10.1523/JNEUROSCI.21-10-03646.2001. [pii] [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Constantinidis Christos, Funahashi S, Lee D, Murray JD, Qi X-L, Wang M, Arnsten AFT. Persistent spiking activity underlies working memory. The Journal of Neuroscience. 2018;38(32):7020–7028. doi: 10.1523/JNEUROSCI.2486-17.2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Darusman HS, Call J, Sajuthi D, Schapiro SJ, Gjedde A, Kalliokoski O, Hau J. Delayed response task performance as a function of age in cynomolgus monkeys (Macaca fascicularis) Primates. 2014;55(2):259–267. doi: 10.1007/s10329-013-0397-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Deadwyler SA, Bunn T, Hampson RE. Hippocampal ensemble activity during spatial delayed-nonmatch-to-sample performance in rats. The Journal of Neuroscience. 1996;16(1):354–372. doi: 10.1523/JNEUROSCI.16-01-00354.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Doncic A, Falleur-Fettig M, Skotheim JM. Distinct interactions select and maintain a specific cell fate. Molecular Cell. 2011;43(4):528–539. doi: 10.1016/j.molcel.2011.06.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Dudchenko PA. An overview of the tasks used to test working memory in rodents. Neuroscience and Biobehavioral Reviews. 2004;28(7):699–709. doi: 10.1016/j.neubiorev.2004.09.002. [DOI] [PubMed] [Google Scholar]
  12. Dudchenko P, Sarter M. Behavioral microanalysis of spatial delayed alternation performance: Rehearsal through overt behavior, and effects of scopolamine and chlordiazepoxide. Psychopharmacology (Berl) 1992;107(2-3):263–270. doi: 10.1007/BF02245146. http://www.springerlink.com/content/f43830ht1r173287/fulltext.pdf . [DOI] [PubMed] [Google Scholar]
  13. Dudchenko PA, Talpos J, Young J, Baxter MG. Animal models of working memory: A review of tasks that might be used in screening drug treatments for the memory impairments found in schizophrenia. Neuroscience and Biobehavioral Reviews. 2013;37(9):2111–2124. doi: 10.1016/j.neubiorev.2012.03.003. [DOI] [PubMed] [Google Scholar]
  14. Dudukovic NM. Serial recall. Encyclopedia of clinical neuropsychology. 2017:1–2. doi: 10.1007/978-3-319-56782-2_1157-3. [DOI] [Google Scholar]
  15. Dunnett SB. Comparative effects of cholinergic drugs and lesions of nucleus basalis or fimbria-fornix on delayed matching in rats. Psychopharmacology (Berl) 1985;87(3):357–363. doi: 10.1007/BF00432721. [DOI] [PubMed] [Google Scholar]
  16. Dunnett SB, Wareham AT, Torres EM. Cholinergic blockade in prefrontal cortex and hippocampus disrupts short-term memory in rats. NeuroReport. 1990;1(1):61–64. doi: 10.1097/00001756-199009000-00017. [DOI] [PubMed] [Google Scholar]
  17. Fassihi A, Akrami A, Esmaeili V, Diamond ME. Tactile perception and working memory in rats and humans. Proceedings of the National Academy of Sciences. 2014;111(6):2331–2336. doi: 10.1073/pnas.1315171111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Funahashi S, Bruce CJ, Goldman-Rakic PS. Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex. Journal of Neurophysiology. 1989;61(2):331–349. doi: 10.1152/jn.1989.61.2.331. http://jn.physiology.org/content/61/2/331.abstract . [DOI] [PubMed] [Google Scholar]
  19. Fuster JM, Alexander GE. Neuron activity related to short-term memory. Science. 1971;173(3997):652–654. doi: 10.1126/science.173.3997.652. [DOI] [PubMed] [Google Scholar]
  20. Fuster Joaquin M, Bauer RH, Jervey JP. Functional interactions between inferotemporal and prefrontal cortex in a cognitive task. Brain Research. 1985;330(2):299–307. doi: 10.1016/0006-8993(85)90689-4. [DOI] [PubMed] [Google Scholar]
  21. Goldman-Rakic P. Cellular basis of working memory. Neuron. 1995;14(3):477–485. doi: 10.1016/0896-6273(95)90304-6. [DOI] [PubMed] [Google Scholar]
  22. Hernández A, Salinas E, García R, Romo R. Discrimination in the sense of flutter: New psychophysical measurements in monkeys. The Journal of Neuroscience. 1997;17(16):6391–6400. doi: 10.1523/JNEUROSCI.17-16-06391.1997. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Herremans AHJ, Hijzen TH, Slangen JL. Validity of a delayed conditional discrimination task as a model for working memory in the rat. Physiology and Behavior. 1994;56(5):869–875. doi: 10.1016/0031-9384(94)90317-4. [DOI] [PubMed] [Google Scholar]
  24. Hunter WS. The delayed reaction in animals and children. Studies from the psychological laboratory of the University of Chicago. 1913:30–62. doi: 10.1037/h0066711. [DOI] [Google Scholar]
  25. Kim CH, Romberg C, Hvoslef-Eide M, Oomen CA, Mar AC, Heath CJ, et al. Saksida LM. Trial-unique, delayed nonmatching-to-location (TUNL) touchscreen testing for mice: Sensitivity to dorsal hippocampal dysfunction. Psychopharmacology (Berl) 2015;232(21-22):3935–3945. doi: 10.1007/s00213-015-4017-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kubota K, Niki H. Prefrontal cortical unit activity and delayed alternation performance in monkeys. Journal of Neurophysiology. 1971;34(3):337–347. doi: 10.1152/jn.1971.34.3.337. [DOI] [PubMed] [Google Scholar]
  27. Levy R, Goldman-Rakic PS. Segregation of working memory functions within the dorsolateral prefrontal cortex. Experimental Brain Research Experimentelle Himforschung Expérimentation Cérébrale. 2000;133(1):23–32. doi: 10.1086/425589. [DOI] [PubMed] [Google Scholar]
  28. Li Z, Chen X, Wang T, Gao Y, Li F, Chen L, et al. Chen JF. The corticostriatal adenosine A2A receptor controls maintenance and retrieval of spatial working memory. Biological Psychiatry. 2018;83(6):530–541. doi: 10.1016/j.biopsych.2017.07.017. [DOI] [PubMed] [Google Scholar]
  29. Lisberger SG. Visual guidance of smooth pursuit eye movements. Annual Review of Vision Science. 2015;1(1):447–468. doi: 10.1146/annurev-vision-082114-035349. [DOI] [PubMed] [Google Scholar]
  30. Liu D, Gu X, Zhu J, Zhang X, Han Z, Yan W, et al. Li CT. Medial prefrontal activity during delay period contributes to learning of a working memory task. Science. 2014;346(6208):458–463. doi: 10.1126/science.1256573. [DOI] [PubMed] [Google Scholar]
  31. Lundqvist M, Herman P, Miller EK. Working memory: Delay activity, Yes! Persistent activity? Maybe not. The Journal of Neuroscience. 2018;38(32):7013–7019. doi: 10.1523/JNEUROSCI.2485-17.2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. McQuail JA, Beas BS, Kelly KB, Simpson KL, Frazier CJ, Setlow B, Bizon JL. NR2A-containing NMDARs in the prefrontal cortex are required for working memory and associated with age-related cognitive decline. The Journal of Neuroscience. 2016;36(50):12537–12548. doi: 10.1523/JNEUROSCI.2332-16. 2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Miller EK, Li L, Desimone R. Activity of neurons in anterior inferior temporal cortex during a short-term memory task. The Journal of Neuroscience. 1993;13(4):1460–1478. doi: 10.1016/j.conb.2004.03.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Moore TL, Schettler SP, Killiany RJ, Rosene DL, Moss MB. Impairment in delayed nonmatching to sample following lesions of dorsal prefrontal cortex. Behavioral Neuroscience. 2012;126(6):772–780. doi: 10.1037/a0030493. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Otto T, Eichenbaum H. Complementary roles of the orbital prefrontal cortex and the perirhinal-entorhinal cortices in an odor-guided delayed-nonmatching-to-sample task. Behavioral Neuroscience. 1992;106(5):762–775. doi: 10.1037/0735-7044.106.5.762. [DOI] [PubMed] [Google Scholar]
  36. Pontecorvo MJ, Sahgal A, Steckler T. Further developments in the measurement of working memory in rodents. Cognitive Brain Research. 1996;3(3-4):205–213. doi: 10.1016/0926-6410(96)00007-9. [DOI] [PubMed] [Google Scholar]
  37. Prusky GT, Douglas RM, Nelson L, Shabanpoor a, Sutherland RJ. Visual memory task for rats reveals an essential role for hippocampus and perirhinal cortex. Proceedings of the National Academy of Sciences of the United States of America. 2004;101(14):5064–5068. doi: 10.1073/pnas.0308528101. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Richardson JTE. Measures of short-term memory: A historical review. Cortex. 2007;43(5):635–650. doi: 10.1016/S0010-9452(08)70493-3. [DOI] [PubMed] [Google Scholar]
  39. Romo R, Brody CD, Hernández A, Lemus L. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature. 1999;399(6735):470–473. doi: 10.1038/20939. [DOI] [PubMed] [Google Scholar]
  40. Roozendaal B. The basolateral amygdala interacts with the medial prefrontal cortex in regulating glucocorticoideffects on working memory impairment. The Journal of Neuroscience. 2004;24(6):1385–1392. doi: 10.1523/JNEUROSCI.4664-03.2004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Sakurai Y, Takahashi S. Dynamic synchrony of firing in the monkey prefrontal cortex during working-memory tasks. The Journal of Neuroscience. 2006;26(40):10141–10153. doi: 10.1523/JNEUROSCI.2423-06.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Sloan HL, Good M, Dunnett SB. Double dissociation between hippocampal and prefrontal lesions on an operant delayed matching task and a water maze reference memory task. Behavioural Brain Research. 2006;171(1):116–126. doi: 10.1016/j.bbr.2006.03.030. [DOI] [PubMed] [Google Scholar]
  43. Sofroniew NJ, Cohen JD, Lee AK, Svoboda K. Natural whisker-guided behavior by head-fixed mice in tactile virtual reality. The Journal of Neuroscience. 2014;34(29):9537–9550. doi: 10.1523/JNEUROSCI.0712-14.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Spaak E, Watanabe K, Funahashi S, Stokes MG. Stable and dynamic coding for working memory in primate prefrontal cortex. The Journal of Neuroscience. 2017 doi: 10.1523/JNEUROSCI.3364-16.2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Spellman T, Rigotti M, Ahmari SE, Fusi S, Gogos JA, Gordon JA. Hippocampal-prefrontal input supports spatial encoding in working memory. Nature. 2015;522(7556):309–314. doi: 10.1038/nature14445. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Stark E, Abeles M. Predicting movement from multiunit activity. The Journal of Neuroscience. 2007;27(31):8387–8394. doi: 10.1523/JNEUROSCI.1321-07. 2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Stark E, Asher I, Abeles M. Encoding of reach and grasp by single neurons in premotor cortex is independent of recording site. Journal of Neurophysiology. 2007;97(March):3351–3364. doi: 10.1152/jn.01328.2006. [DOI] [PubMed] [Google Scholar]
  48. Stark E, Globerson A, Asher I, Abeles M. Correlations between Groups of premotor neurons carry information about prehension. The Journal of Neuroscience. 2008;28(42):10618–10630. doi: 10.1523/JNEUROSCI.3418-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Takeda K, Funahashi S. Population vector analysis of primate prefrontal activity during spatial working memory. Cerebral Cortex. 2004;14(12):1328–1339. doi: 10.1093/cercor/bhh093. [DOI] [PubMed] [Google Scholar]
  50. van Hest A, Steckler T. Effects of procedural parameters on response accuracy: Lessons from delayed (non-)matching procedures in animals. Cognitive Brain Research. 1996;3(3-4):193–203. doi: 10.1016/0926-6410(96)00006-7. [DOI] [PubMed] [Google Scholar]
  51. Wang X-J. Synaptic reverberation underlying mnemonic persistent activity. Trends in Neurosciences. 2001;24(8):455–463. doi: 10.1016/S0166-2236(00)01868-3. [DOI] [PubMed] [Google Scholar]
  52. Yuen EY, Liu W, Karatsoreos IN, Ren Y, Feng J, McEwen BS, Yan Z. Mechanisms for acute stress-induced enhancement of glutamatergic transmission and working memory. Molecular Psychiatry. 2011;16(2):156–170. doi: 10.1038/mp.2010.50. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES