Abstract goal representation in visual search by neurons in the human pre-supplementary motor area

The role of the pre-supplementary motor area (pre-SMA) in linking goals to actions remains unclear. Using single-neuron recordings in human pre-SMA, Wang et al. identify cells that signal when a fixation lands on a target across several different visual search tasks. One function of pre-SMA may thus be goal detection.

Abstract

The medial frontal cortex is important for goal-directed behaviours such as visual search. The pre-supplementary motor area (pre-SMA) plays a critical role in linking higher-level goals to actions, but little is known about the responses of individual cells in this area in humans. Pre-SMA dysfunction is thought to be a critical factor in the cognitive deficits that are observed in diseases such as Parkinson’s disease and schizophrenia, making it important to develop a better mechanistic understanding of the pre-SMA’s role in cognition. We simultaneously recorded single neurons in the human pre-SMA and eye movements while subjects performed goal-directed visual search tasks. We characterized two groups of neurons in the pre-SMA. First, 40% of neurons changed their firing rate whenever a fixation landed on the search target. These neurons responded to targets in an abstract manner across several conditions and tasks. Responses were invariant to motor output (i.e. button press or not), and to different ways of defining the search target (by instruction or pop-out). Second, ∼50% of neurons changed their response as a function of fixation order. Together, our results show that human pre-SMA neurons carry abstract signals during visual search that indicate whether a goal was reached in an action- and cue-independent manner. This suggests that the pre-SMA contributes to goal-directed behaviour by flexibly signalling goal detection and time elapsed since start of the search, and this process occurs regardless of task. These observations provide insights into how pre-SMA dysfunction might impact cognitive function.

Introduction

The prefrontal cortex plays a critical role in exerting cognitive control, a process that forms the core of goal-directed behaviour (Miller, 2000). The prefrontal cortex is composed of different brain areas, each of which plays distinct roles in this process. Our focus in this study is on the pre-supplementary motor area (pre-SMA), which plays a key role in representing cognitive attributes and linking them to actions (Nachev et al., 2008). There are three aspects of pre-SMA function that are relevant to our study. First, the pre-SMA is thought to be particularly important for self-initiated voluntary actions (Jenkins et al., 2000; Cunnington et al., 2002; Nachev et al., 2005), a point of view also supported by the finding that pre-SMA neurons seem to be one of the sources of the readiness potential, which precedes voluntary actions (Passingham, 1995; Fried et al., 2011). Second, the pre-SMA is involved in cognitive control and is thought to contribute to resolving response conflict by inhibiting inappropriate responses until the appropriate action is selected (Rushworth et al., 2002; Ridderinkhof et al., 2004; Wager et al., 2004; Curtis et al., 2005; Aron and Poldrack, 2006; Crone et al., 2006; Fu et al., 2019). For example, during a saccade-switching task, a monkey’s ability to successfully switch saccades depends on rapid activation of the pre-SMA, which provides inhibitory output to prevent the anticipated action (Isoda and Hikosaka, 2007). Third, the pre-SMA is involved in the generation of complex sequences of movements (Shima et al., 1996; Nachev et al., 2008), with many neurons signalling the position in a current action sequence or the transition from one state to the next.

Non-invasive human neuroimaging and post-mortem anatomical studies have revealed that pre-SMA dysfunction is prominent in multiple diseases that manifest in complex cognitive deficits (Nachev et al., 2008). One such disease is Parkinson’s disease. Patients with Parkinson’s disease frequently suffer from prominent but poorly understood cognitive control deficits. Pre-SMA hypoactivity as well as prominent loss of neurons are well documented in Parkinson’s disease (MacDonald and Halliday, 2002; Nachev et al., 2008), but the relationship between cognitive deficits and pre-SMA dysfunction is unclear. Conversely, in autism, pre-SMA hyperactivity has been linked to abnormal use of higher cognitive circuits for processes that would usually be achieved using lower-level sensorimotor circuitry alone (Iacoboni, 2005; Minshew and Keller, 2010). Given the hypothesized role of pre-SMA dysfunction in disease, it is thus important to gain a better understanding of the mechanisms by which pre-SMA neurons support executive function.

Goal-directed behaviours require highly flexible planning of actions in order to achieve the goal, which is often defined at an abstract level. A well-studied goal-directed behaviour is visual search, where strategies for sequences of eye movements are sensitive to high-level goals. We recently reported that the human medial temporal lobe contains neurons encoding whether or not a visual target is fixated (Wang et al., 2018). Moreover, that prior study found that these target-selective neurons are distinct from the well-known category-selective neurons (Kreiman et al., 2000). The former encode whether a fixated object is the target of a search regardless of its category membership, whereas the latter encode whether the object is a member of a category irrespective of whether it is the target in a search. We also presented preliminary evidence that target-selective neurons exist in the prefrontal cortex, and discovered that those neurons respond to targets with latencies ∼200 ms earlier than medial temporal lobe target-selective neurons (Wang et al., 2018). This prior work suggests that the prefrontal cortex may be the source of abstract target representations, providing top-down signals to other brain regions such as the medial temporal lobe during visual search. However, in that previous work we were not able to directly test this hypothesis because we did not compare the response of the same cells between different types of search tasks. Here we investigate this hypothesis directly by comparing the response of the target cells recorded in the pre-SMA of human patients with epilepsy between different tasks.

The data summarized above indicate that the pre-SMA plays a critical role in visual search, in which complex voluntary sequences of movements are executed until the target is found, at which point movements are inhibited to indicate that the target has been found. However, to date little is known about what the specific features are that human pre-SMA neurons represent during visual search. In macaques, many pre-SMA neurons represent combinations of sensory cues and movements, i.e. a neuron would only fire if a particular visual cue indicates a particular movement that should be executed (Nachev et al., 2008). Other macaque pre-SMA neurons signal only when a particular planned movement is inhibited (Isoda and Hikosaka, 2007). At present, it remains unknown whether similar action- and cue-specific representations are engaged in the human pre-SMA during visual search. Alternatively, non-invasive human neuroimaging work suggests that the human pre-SMA carries abstract cognitive signals related to attention, the detection of conflict, and the detection of goal-relevant information even if no overt motor response is involved (Picard and Strick 1996; Coull et al., 2004; Nachev et al., 2008).

In our prior work (Wang et al., 2018), subjects were instructed to locate a visual target shown among other objects and to indicate, with a button press, whether the current trial contained the target or not. The target changed trial-by-trial, requiring rapid implementation of changing goals (Chelazzi et al., 1993, 1998; Wolfe, 1994; Itti and Koch, 2000; Shulman et al., 2003; Bichot et al., 2005; Rutishauser and Koch, 2007). We performed simultaneous single-neuron recordings and eye tracking while neurosurgical patients performed this task. In our previous study, we had found that MTL and pre-SMA neurons signalled target detection, i.e. they indicated whether the currently attended item was the goal of the search or not (Wang et al., 2018). That study left open three unanswered questions on which we focus here:

1. Do human pre-SMA neurons signal target detection in an abstract way or, alternatively, do they signal the need to indicate such by a button press? In the original study, we could not differentiate between these two possibilities because subjects were required to report that they had seen the target right away. Here, we used a task variant where subjects did not need to press a button.

2. What is the granularity of target representations in the pre-SMA, i.e. do pre-SMA target neurons also contain information about how the target was specified? This is a critical question, because previous macaque work indicates a strong dependence on the sensory cue that indicates the need for a particular action (Nachev et al., 2008). Our prior work found target representations for both visually encoded targets that were held in working memory, and for targets that pop out by being distinctive, but did not yet probe this question in detail.

3. Does the pre-SMA contain representations only for the goal (target) or does it also represent information about how close one is to the goal (i.e. the current state relative to the goal state)?

Here, we present new data and a detailed reanalysis of previously published data to answer these questions by comparing the response of the same pre-SMA target neurons between three versions of a visual search task. In the first new experiment (no-response task), patients did not use actions to respond to search targets, allowing us to dissociate target detection from motor action. In the second experiment (pop-out task), no explicit search cue was specified, allowing us to dissociate target detection from sensory cues. We found that the same population of pre-SMA neurons signalled target detection in all three versions. Together, our results suggest an important role for the pre-SMA in goal-directed behaviour: it signals task- and action-invariant goal detection during visual search.

Materials and methods

Unless noted, stimuli, task, electrophysiology and analyses essentially followed those of Wang et al. (2018). Single-neuron data from the pre-SMA for the standard task were briefly analysed in Wang et al. (2018), and we here analysed these data in more detail. We also included new data for the no-response task and pop-out task recorded from the same neurons that has not been published previously

Subjects

There were 10 sessions from nine patients in total (Supplementary Table 1). Patients had pharmacologically intractable localization-related epilepsy and were undergoing depth electrode implantation for the purpose of better localizing their seizures for potential surgical resection. Electrode target locations were determined by clinical criteria alone. All subjects provided written informed consent. Protocols were approved by the institutional review boards of the Cedars-Sinai Medical Center and the California Institute of Technology. We also compared behaviour of our patients with eight healthy control subjects to confirm that our patients performed the task normally. These control subjects were characterized previously (Wang et al., 2014b).

Tasks and stimuli

Three tasks were presented in a fixed order: (i) standard task; (ii) no-response task; and (iii) pop-out task. Patients finished one or two blocks of each task, depending on available time. Note that the no-response task and pop-out task were always performed in the same session as the standard task. All experiments used a randomized within-subject design, with the condition tested in a given trial randomized (target location, target identity).

The standard task (Fig. 1A and B) was used in a previous study (Wang et al., 2014b, 2018). A target was presented for 1 s, followed by the search array. Patients were instructed to find the item in the array that matched the target and were explicitly told that the array might or might not contain the target. The search array stayed up for a maximum of 14 s, or until the subject responded with a Cedrus^TM button box, either by pushing the left button to indicate that the target was found in the array, or by pushing the right button to indicate that the target was absent from the array. A feedback message (‘Correct’ or ‘Incorrect’) was then displayed for 1s. Subjects were instructed to respond as quickly and as accurately as possible. If subjects did not respond within 14 s after array onset, a message ‘Time Out’ was displayed. An intertrial interval was jittered between 1 and 2 s. The array and target orders were completely randomized for each subject. Subjects practiced five trials before the experiment to familiarize themselves with the task. In the end, the overall percentage of correct answers was displayed to patients as a motivation.

Figure 1.

Task and behaviour. (A and B) Standard task. (C and D) No-response task. (E and F) Pop-out task. (A) The search cue is shown for 1 s, immediately followed by the search array. Subjects are instructed to indicate by button press whether the target is present or absent (timeout 14 s). Trial-by-trial feedback is given immediately after button press (‘Correct’, ‘Incorrect’, or ‘Time Out’), followed by a blank screen for 1–2 s. (C) The task is the same as the standard task except that the search array was displayed for a fixed duration (3 s) and no button press was required. In catch trials (every 10 trials), we prompted a cue to subjects and asked them to report target presence (the same button press as the standard task). (E) The target was defined by ‘oddball’: there was either one face among vehicles or one vehicle among faces. Subjects were instructed to indicate by button press as soon as they determined whether there was an oddball (the same button press as the standard task). (B, D and F) Example visual search arrays with fixations indicated. Each circle represents a fixation. Green circle: first fixation; magenta circle: last fixation; yellow line: saccades; blue dot: raw gaze position; red box: target. (G) Location of recording sites in MNI152 space. Each dot represents the location of a microwire bundle in a patient. Three views (left: sagittal; middle: axial; right: coronal) are shown. Note that all recording locations are anterior to y = 0 (vertical dashed line), which is typically taken as the boundary between the SMA and pre-SMA in most studies. Only recording locations that had neurons are plotted. ITI = inter-trial interval; RT = reaction time.

We used 20 distinct visual search arrays. In each array there were 24 items whose spatial locations were randomized between the 20 arrays. Twelve items were social (faces and people with different postures, emotions, ages, and genders, etc.) and 12 items were non-social (furniture, toys, food, etc.). These social and non-social items composing the array stimuli have been characterized and described previously (Sasson et al., 2012; Wang et al., 2014b). From each array stimulus, we randomly assigned four social items and four non-social items as targets (in eight distinct trials). For each array, we also had two target-absent trials, i.e. the target was not among the objects in the search array (one catch trial with a social target, and one with a non-social target). Therefore, in total we had 100 trials with social targets and 100 trials with non-social targets (in randomized order), and 20% of trials were catch trials. The entire task was separated into two blocks. Each block had 100 trials. Patients finished at least one block. Importantly, low-level properties of social and non-social items were equalized within each search array. The social and non-social items did not differ in standard low-level saliency as quantified by the Itti-Koch model (Itti et al., 1998; Itti and Koch, 2001), distance to centre or size (all P-values > 0.79). In the analysis of visual tuning to object category, we further categorized the items into 14 finer categories: face, clock, vehicle, furniture, electronics, stationery, sign, plant, toy, sport, bag, comb, clothes, and food.

The no-response task (Fig. 1C and D) was identical to the standard task except that there was no trial-by-trial button press required from the patients. Instead, each search array was presented for 3 s and patients were instructed to find the target and look at the target during this period. Given the reaction time from the standard task, patients were generally able to detect the target within 3 s. To confirm this, we prompted a cue to patients and asked them to report target presence (the same button press as the standard task) every 10 trials. These catch trials had both target-present trials and target-absent trials. Percentage of correct response from catch trials was displayed at the end of the task. Each block had 100 trials. The no-response task used the identical stimuli as the standard task (but in a different random order).

The pop-out task (Fig. 1E and F) used different search arrays but the same task as the standard task, except that the targets were defined by their distinctiveness; there was no search template shown (see below). Patients were still instructed to report target presence/absence as quickly and accurately as possible. Each block had 100 trials. To create the stimuli, we only used face items and vehicle items from those in the standard task. There were still 24 items in total in each search array, but each array was generated from a randomly selected subset of face and vehicle items. The spatial location of each item was randomized. Target-present arrays contained either one face among otherwise all vehicles or one vehicle among otherwise all faces. Thus, the target was defined as a ‘pop-out’ in terms of category membership because it only existed once. Target-absent arrays contained homogeneously only faces or vehicles. Patients were explicitly instructed as to what the pop-out target was with respect to category membership.

In all three tasks, patients were instructed that there were both target-present trials and target-absent trials.

Electrophysiology

We recorded bilaterally from standard FDA-approved hybrid Behnke-Fried implanted depth electrodes in the medial frontal cortex. At each site, we recorded from eight 40-μm microwires as described previously (Rutishauser et al., 2006a, 2010). Effort was always made to avoid passing the electrode through a sulcus, and its attendant sulcal blood vessels, and thus the location varied but was always well within the body of the targeted area. Bipolar wide-band recordings (0.1–9 kHz), using one of the eight microwires as reference, were sampled at 32 kHz and stored continuously for off-line analysis with a Neuralynx system. The raw signal was filtered with zero-phase lag 300 Hz – 3 kHz bandpass filter and spikes were sorted using a semi-automated template-matching algorithm as described previously (Rutishauser et al., 2006b). Units were carefully isolated and recording and spike sorting quality were assessed quantitatively (Supplementary Fig. 1). Note that the no-response and pop-out tasks were always performed in the same session as the standard task, and to compare the neuronal response across tasks, neurons were always sorted across the entire session (not for each individual task).

Electrode localization

We recorded from electrodes located in the pre-SMA, which is located in the part of the superior frontal gyrus that lies rostral to the vertical commissure anterior line and caudal to a parallel line through the genu of the corpus callosum. The vertical commissure anterior line forms the boundary between the pre-SMA and SMA (Picard and Strick, 1996; Zilles et al., 1996; Baleydier et al., 1997; Kim et al., 2010), and we targeted electrodes to lie ∼10 mm anterior to the vertical commissure anterior line. We verified recording locations with post-operative T₁ and T₂ MRI scans and only included recording sites judged to be within the pre-SMA based on the above criteria (Fig. 1G). For quantitative analysis of electrode position, we first aligned the pre- and postoperative MRI scans of each patient using Freesurfer’s mri_robust_register (Reuter et al., 2010). We then mapped the preoperative brain onto an atlas brain (the CIT168 brain) (Tyszka and Pauli, 2016) in MNI152 coordinates using a concatenation of an affine transformation followed by a symmetric image normalization (SyN) diffeomorphic transform computed by the ANTs suite of programs (http://stnava.github.io/ANTs/). We have described this procedure in more detail previously (Kaminski et al., 2017).

Eye tracking

Monocular eye tracking at 500 Hz was done with a remote non-invasive infrared Eyelink^® 1000 system (SR Research). The eye tracker was calibrated with the built-in 9-point grid method at the beginning of each block. Fixation extraction was carried out using software supplied with the Eyelink^® eye tracking system. Saccade detection required a deflection of greater than 0.1°, with a minimum velocity of 30°/s and a minimum acceleration of 8000°/s², maintained for at least 4 ms. Fixations were defined as the complement of a saccade, i.e. periods without saccades. Analysis of the eye movement record was carried out off-line after completion of the experiments.

Patients sat ∼60 cm in front of a 17-inch LCD display (screen resolution: 1024 × 768). The refresh rate of the display was 60 Hz and the stimuli occupied the centre of the display (31.5° × 25.4° visual angle). Stimuli were presented using MATLAB with the Psychtoolbox 3 (Brainard, 1997) (http://psychtoolbox.org).

Data analysis

Spikes

Only units with an average firing rate of at least 0.2 Hz (entire task), and characteristics of well-isolated single units were considered (Supplementary Fig. 1). Trials were aligned to stimulus onset or button press. Fixations were aligned to fixation onset. Average firing rates (peri-stimulus time histogram, PSTH) were computed by counting spikes across all trials in consecutive 250-ms bins and across all fixations in consecutive 50-ms bins. Pairwise comparisons were made using a two-tailed t-test at P < 0.05 and Bonferroni-corrected for multiple comparisons in time bins in the group PSTH. Asterisks in figures indicate a significant difference after Bonferroni correction.

Target selectivity index

We quantified for each neuron whether its response differed between fixations on targets and fixations on distractors using a single-fixation target selectivity index (TSI) (Equation 1). The TSI facilitates group analysis and comparisons between different types of cells (i.e. target- and distractor-preferring cells in this study), as motivated by previous studies (Rutishauser et al., 2008, 2011; Wang et al., 2014a). The TSI quantifies the response during fixation i relative to the mean response to fixations on distractors and baseline (a 1-s interval of blank screen right before target cue onset). The mean response and baseline were calculated individually for each neuron.

$$\mathit{TS}{I}_i=\frac{F{R}_i-\mathit{mean}\left(F{R}_{\mathit{Distractor}}\right)}{\mathit{mean}\left(F{R}_{\mathit{Baseline}}\right)}·100\mathrm{\%}$$ (1)

For each fixation i, which can be either on a target or a distractor, TSI_i is the baseline normalized mean firing rate (FR) during an interval from 200 ms before fixation onset to 200 ms after fixation offset (the same time interval as cell selection). Different time intervals were also tested, to ensure that results were qualitatively the same and not biased by particular spike bins. The same definition of fixations was used across the three task variants.

If a neuron distinguishes fixations on target from fixations on distractors, the average value of TSI_i of all fixations will be significantly different from 0. As target-preferring neurons have more spikes in fixations on targets and distractor-preferring neurons have more spikes in fixation on distractors (the selection process is described above), on average TSI_i is positive for target-preferring neurons and negative for distractor-preferring neurons. To get an aggregate measure of activity that pools across neurons, TSI_i was multiplied by −1 if the neuron is classified as a distractor-preferring neuron (Equation 2). This makes TSI_i on average positive for both types of target neurons. Note that the factor −1 depends only on the neuron type, which is determined by t-tests on fixations as described above, but not fixation type. Thus, negative TSI_i values are still possible.

$$\mathit{TS}{I}_i=-\frac{F{R}_i-\mathit{mean}\left(F{R}_{\mathit{Distractor}}\right)}{\mathit{mean}\left(F{R}_{\mathit{Baseline}}\right)}·100\mathrm{\%}$$ (2)

After calculating TSI_i for every fixation, we subsequently averaged all TSI_i of fixations that belong to the same category. By definition, the average value of TSI_i for fixation on distractors will be equal to zero because the definition of TSI_iis relative to the response to fixation on distractors (Equation 2). The mean baseline firing rate was calculated across all trials. The same FR_Distractor was subtracted for both types of fixations.

The cumulative distribution function (CDF) was constructed by calculating for each possible value x of the TSI how many examples are smaller than x. That is, F(x) = P(X ≤ x), where X is a vector of all TSI values. The CDF of fixations on targets and distractors were compared using two-tailed two-sample Kolmogorov–Smirnov tests. All error bars are ± standard error (SE) unless indicated otherwise.

Single-neuron receiver operating characteristic curve analysis

Neuronal receiver operating characteristic curves (ROCs) were constructed based on the spike counts in a time window of 200 ms before fixation onset to 200 ms after fixation offset for fixation-wise analysis. We varied the detection threshold between the minimal and maximal spike count observed, linearly spaced in 20 steps. The area under the curve (AUC) of the ROC was calculated by integrating the area under the ROC curve (trapezoid rule). The AUC value is an unbiased estimate for the sensitivity of an ideal observer that counts spikes and makes a binary decision based on whether the number of spikes is above or below a threshold. We defined the category with higher overall firing rate as ‘true positive’ and the category with lower overall firing rate as ‘false positive’. Therefore, the AUC value was always >0.5 by definition.

Comparison of cell types

We quantified basic electrophysiological parameters following previous studies (Viskontas et al., 2007; Rutishauser et al., 2013). To compare the variability of spike times, we computed the inter-spike interval (ISI) distribution of each cell by considering all spikes fired during the experiment and quantified it using two metrics: the burst index (BI) and the modified coefficient of variation (CV₂). The BI was defined as the proportion of ISIs less than 10 ms (Wyler et al., 1975). The CV₂ (Equation 3) is a function of the difference between two adjacent ISIs and is a standard measure to quantify spike-train variability that is robust to underlying rate changes (Holt et al., 1996). In contrast, the coefficient of variation measure (CV) is only valid for stationary processes (i.e. fixed mean firing rate) and is thus not applicable for this analysis.

$$C{V}_2=\frac{1}{N}\stackrel{N}{\sum _{i=1}}\frac{2|\mathit{IS}{I}_{i+1}-\mathit{IS}{I}_i|}{\mathit{IS}{I}_{i+1}+\mathit{IS}{I}_i}$$ (3)

We compared the waveform of different neurons based on the trough-to-peak time of the mean waveform (Mitchell et al., 2007). The mean waveform is the average of all spikes assigned to the cluster. The polarity of the mean waveforms was inverted if necessary such that the trough always occurs before the peak. We also verified whether there is a correlation between the trough-to-peak time and the mean firing rate of a unit. For this, the mean firing rate was defined as the mean rate over the entire duration of all valid trials.

Regression analysis

We used the regression model $S\left(t\right)=a_0\left(t\right)+a_1\left(t\right)·L$ to estimate whether the firing rate S was significantly related to one of the following factors (L): target or distractor (0 or 1), and fixation order (1–10). Separate models were fit for each factor. The model was fit to the total spike count in a time window of −200 ms before fixation onset to 200 ms after fixation offset. We estimated the significance of each factor using ω² as described previously (Rutishauser et al., 2015), which is less biased than percentage variance explained (Olejnik and Algina, 2003). Here,

$${\omega }_i^2=\frac{S{S}_i-d{f}_i·\mathit{MSE}}{S{S}_{\mathit{tot}}+\mathit{MSE}}$$ (4)

where SS_i is the sum of squares of factor i, SS_tot is the total sum of squares of the model, and MSE is the mean square error of the model. Effect sizes were calculated using the effect size toolbox (Hentschke and Stuttgen, 2011). The null distribution was estimated by randomly scrambling the labels and fitting the same model. This was repeated 500 times to estimate the statistical significance.

Population decoding

We pooled all recorded neurons into a large pseudo-population (Rutishauser et al., 2015). Firing rates were z-scored individually for each neuron to give equal weight to each unit regardless of firing rate. We used a maximal correlation coefficient classifier (MCC) as implemented in the NDT toolbox (Meyers, 2013). The MCC estimates a mean template for each class i and assigns the class for test fixation to the class j for which the test fixation has the largest correlation with the mean template. We used 20-fold cross-validation, i.e. for each iteration 20 fixations from each class were chosen from each neuron, among which 19 fixations were used for training and the remaining one fixation was used for testing (30-fold for cross-task generalization analysis). All possible train/test splits were tested and this process was repeated 30 times with different subsets of fixations (50 times for cross-task analysis), resulting in a total of 600 tests to estimate the test performance (1500 tests for cross-task analysis). Spikes were counted in bins of 500-ms size and advanced by a step size of 50 ms. The first bin started −200 ms relative to fixation onset (bin centre was thus 50 ms after fixation onset), and we tested seven consecutive bins (the last bin was thus from 100 ms to 600 ms after fixation onset). For each bin, a different classifier was trained/tested.

Visual selectivity

When we selected visually selective neurons, we used a tighter fixation time window of 0–300 ms after fixation onset to ensure that visual inputs were restricted to fixations. However, using the identical time window for target neurons that included saccades before and after fixations, we derived qualitatively the same results.

Data availability

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Results

Tasks and behaviour

Recently, we identified pre-SMA neurons that signal whether a currently fixated item is the search target in a present/absent guided search task (‘standard task’) (Wang et al., 2018). In this study, we first summarize the results from the standard task, which has the same neurons as in our previous report; and then we show new data from additional comparison experiments, which we will refer to as ‘no-response’ and ‘pop-out’ (Fig. 1), respectively, to systemically characterize the response of the same pre-SMA target neurons in different tasks.

Nine patients (10 sessions; Supplementary Table 1) performed the standard search task (Fig. 1A and B). Patients performed well with an average reporting accuracy of 93.6 ± 7.20% [mean ± standard deviation (SD)]. Considering only correct trials, the average reaction time (RT, relative to search array onset) for target-present trials was significantly faster compared to target-absent trials [1.64 ± 0.25 s versus 3.67 ± 1.02 s, two-tailed paired t-test, t(9) = 7.99, P = 2.23 × 10⁻⁵]. The accuracy and reaction time of the patients was comparable to that of eight healthy control subjects performing the same task in a previous study (Wang et al., 2014b), confirming that our patients’ behavioural performance was not significantly different from that of controls [two-tailed two-sample t-test, t(16) = 0.89, P = 0.39 and t(16) = 1.54, P = 0.14 for accuracy and reaction time, respectively].

Eight patients (nine sessions) who had performed the standard task also performed the no-response task (Fig. 1C and D). In this task, the search array was displayed for a fixed duration (3 s) and no button press was required (Fig. 1C and D). This allowed us to examine the relationship between target responses and motor planning. Patients were instructed to fixate on the target when they found it and to maintain fixation until the search array disappeared. The fixed duration of 3 s was sufficient for patients to perform the task because the average reaction time observed in the standard task was 1.83 ± 0.71 s. Every 10th trial was a catch trial, in which patients were required to report target presence/absence in that trial with a button press, as in the standard task. Patients responded in 63.3 ± 21.2% of catch trials (patients did not respond within the 3-s timeout in the remainder) and all such responses were correct. Also, we found that in 85.8 ± 16.0% of the catch trials in which patients reported target presence, they also fixated on the target [compared to 90.7 ± 7.39% for the standard task; unpaired two-tailed t-test, t(17) = 0.87, P = 0.40; in 75.9 ± 24.8% of all target-present trials patients fixated on the target, compared to 85.6 ± 10.5% in the standard task, t(17) = 1.14, P = 0.27; similar results were derived when excluding trials with reaction time >3 s in the standard task: both P-values > 0.37].

All eight patients (nine sessions) who had performed the no-response task also performed the pop-out task (Fig. 1E and F). In this task, no explicit search cue was given. Instead, the target was defined as an ‘oddball’: there was either one face among vehicles or one vehicle among faces. Patients were instructed to indicate by button press as soon as they determined whether there was an oddball (target). This allowed us to examine target or distractor responses in the absence of an explicitly provided reference. Patients performed well with an average reporting accuracy of 95.0 ± 4.72% (mean ± SD), and the average reaction time (relative to search array onset) for target-present and target-absent trials were 2.11 ± 0.44 s and 3.78 ± 0.74 s, respectively [two-tailed paired t-test, t(8) = 7.90, P = 4.79 × 10⁻⁵].

Single neurons in the pre-SMA encode search targets in the standard task

One hundred and eighty-two single neurons were recorded from the pre-SMA. One hundred and twenty-nine neurons had a spontaneous firing rate >0.2 Hz and we restricted our analysis to this subset. All recording locations were carefully verified to be within the pre-SMA (Fig. 1G) (mean y-coordinate of electrodes was 19.4 ± 4.69 mm in MNI152 coordinates) based on postoperative MRI scans (see ‘Materials and methods’ section). The anatomical location of the pre-SMA in humans is well understood and can be derived based on anatomical landmarks alone (Picard and Strick, 1996; Nachev et al., 2008; cfNachev et al., 2005) without the need to perform functional mapping as in the original pioneer monkeys studies (Matsuzaka et al., 1992).

This paragraph briefly summarizes the previously published target response in the standard task (Wang et al., 2018). Responses of neurons distinguished between target-present and target-absent trials (Fig. 2A and B for responses aligned at button-press). To investigate the neural substrates of target detection, we aligned neuronal responses at fixation onset and compared the response of each neuron between fixations on distractors and targets. As the response we used the mean firing rate in a time window starting 200 ms before fixation onset and ending 200 ms after fixation offset (next saccade onset). Fifty-one neurons (39.5%; binomial P < 10⁻⁵⁰) had a response that differed significantly between fixations on targets versus distractors in target-present trials (two-tailed t-test with P < 0.05). We identified two types of such ‘target-selective’ neurons in the pre-SMA: one type had a greater response to targets relative to distractors (target-preferring; 31/51 cells) (Fig. 2C and Supplementary Fig. 2A–F) and the second had a greater response to distractors relative to targets (distractor-preferring; 20/51 cells) (Fig. 2D and Supplementary Fig. 2G–L; see Fig. 2E and F for group PSTH). The proportion of target-selective cells identified was highly significant (permutation test: P < 0.001): a control revealed 6.38 ± 2.52 neurons expected by chance (mean ± SD; 3.61 ± 1.92 target-preferring and 2.78 ± 1.64 distractor-preferring). This demonstrates that a subset of pre-SMA neurons encode whether the present fixation landed on a target or not. Target-selective cells were found in all patients (Supplementary Table 1).

Figure 2.

Standard task. (A and B) Button press aligned examples. (A) Neuron that increased its firing rate for target-present trials, but not for target-absent trials [two-tailed t-test in a time window of −500 ms to 500 ms around button press (RT): P < 10⁻⁹]. (B) Neuron that decreased its firing rate for target-present trials, but not for target-absent trials (P < 0.05). Trials are aligned to the button press (grey line), and are sorted by reaction time (RT). Black lines represent the onset and offset of the search cue (1-s duration). The inset shows waveforms for each unit. Asterisk indicates a significant difference between target-present and absent trials in that bin (P < 0.05, two-tailed t-test, after Bonferroni correction; bin size = 250 ms). Shaded area denotes ±SEM across trials. (C and D) Fixation-aligned examples. t = 0 is fixation onset. (C) Neuron that increased its firing rate when fixating on targets, but not distractors (selection by two-tailed t-test in a time window of −200 ms before fixation onset to 200 ms after fixation offset: P < 10⁻²⁶). (D) Neuron that decreased its firing rate when fixating on targets but not distractors (P < 10⁻¹⁴). Fixations are sorted by fixation duration (black line shows start of the next saccade). Asterisk indicates a significant difference between fixations on targets and distractors in that bin (P < 0.05, two-tailed t-test, after Bonferroni correction; bin size = 50 ms). (E–H) Population summary of all target-selective neurons. (E) Average normalized firing rate of target-preferring neurons (n = 31). (F) Average normalized firing rate of distractor-preferring neurons (n = 20). Shaded area denotes ±SEM across neurons. Asterisk indicates a significant difference between the conditions in that bin (P < 0.05, two-tailed t-test, after Bonferroni correction). (G) Single fixation analysis using the TSI. Shown is the cumulative distribution of the single fixation response of fixation-aligned target- and distractor-preferring neurons for fixations on targets and distractors (n = 51 neurons). (H) Population summary using ROC analysis. Shown are histograms of AUC values of target-preferring neurons (red), distractor-preferring neurons (blue), and neurons that are neither target-preferring or distractor-preferring (grey). All fixations on targets and all fixations on distractors in target-present trials were included in these plots.

Furthermore, note that while distractor-preferring neurons had higher firing rates for distractors compared to targets, the pattern of modulation was different: rather than increasing their activity for distractors, they decreased their activity for targets. It was this decrease that carried the target signal for this subtype of target-selective cells. Lastly, we tested whether the properties of target-preferring and distractor-preferring neurons differed. However, we found no statistically significant differences in mean firing rate [Fig. 3A; target-preferring: 2.52 ± 2.95 Hz (mean ± SD across cells), distractor-preferring: 3.94 ± 3.65 Hz; two-tailed two-sample t-test: t(49) = 1.53, P = 0.13] nor the variability of spike times (see ‘Materials and methods’ section), as quantified by the burst index [Fig. 3B; target-preferring: 0.025 ± 0.026, distractor-preferring: 0.062 ± 0.10; t(49) = 1.91, P = 0.062] and the modified coefficient of variation (CV₂) [Fig. 3C and D; target-preferring: 0.89 ± 0.11, distractor-preferring: 0.90 ± 0.13; t(49) = 0.29, P = 0.77]. Moreover, waveforms of target-preferring and distractor-preferring neurons looked similar (Fig. 3E and F) and the trough-to-peak times were statistically indistinguishable [Fig. 3G and H; target-preferring: 0.78 ± 0.28 ms, distractor-preferring: 0.79 ± 0.28 ms; t(49) = 0.20, P = 0.84; Kolmogorov-Smirnov (KS) test: P = 0.75; proportion of neurons with trough-to-peak times > 0.5 ms: target-preferring: 24/31, distractor-preferring: 16/20; χ²-test: P = 0.83]. Lastly, neither target-preferring [r(31) = −0.20, P = 0.29] nor distractor-preferring neurons [r(20) = 0.072, P = 0.76] showed a significant correlation between mean firing rate and waveform as quantified by trough-to-peak time. Together, the basic electrophysiological signatures suggest that target-preferring and distractor-preferring neurons were not different cell types.

Figure 3.

Comparison of cell type between target-preferring neurons and distractor-preferring neurons. (A) Mean firing rate. Error bar denotes ±SEM across neurons and circles show individual values. Red: target-preferring neurons; blue: distractor-preferring neurons. (B) Burst index. (C and D) Distribution of the modified coefficient of variation (CV₂). (E and F) Mean action potential waveforms. (G and H) Distribution of trough-to-peak times. (I and J) Relationship between mean firing rate and trough-to-peak time. (C, E, G and I) Target-preferring neurons (n = 31). (D, F, H and J) Distractor-preferring neurons (n = 20).

Properties of target-selective cells in the pre-SMA

To assess whether the response of target-selective cells could be attributed to sensory input alone, we compared their response between fixations on identical objects (keeping visual input the same) as a function of whether the fixated object was presently a target or a distractor (a top-down factor). Target-preferring neurons increased firing rate when the identical object was a target [n = 31, two-tailed paired t-test against 0: t(30) = 4.30, P = 1.66 × 10⁻⁴] whereas distractor-preferring neurons increased firing rate when the identical object was a distractor [n = 20, t(19) = 3.77, P = 0.0013]. Because in this comparison, visual inputs are identical, this result indicates that the response of target-selective neurons in the pre-SMA was determined by top-down factors.

To investigate the relationship between the response of target-selective cells and behaviour, we quantified the response of target-selective cells during individual fixations using a TSI. The TSI is equal to the response during an on-target fixation, normalized by the average response during fixations on distractors (Equations 1 and 2). As expected, the TSI for target-selective cells was significantly larger during fixations on targets compared to fixations on distractors (n = 51, two-tailed two-sample KS test, KS = 0.17, P = 8.30 × 10⁻¹⁹⁰) (Fig. 2G). This confirms that the single-fixation response of target-selective cells is strong enough to allow single-fixation analysis (Fig. 2H). Permutation tests by shuffling the label of target and distractor further confirmed our results: target-selective cells (81.2 ± 106.5%, mean ± SD across cells) had a significantly higher TSI compared to chance (15.4 ± 2.33%, permutation P < 0.001), whereas the TSI of all non-target-selective cells (21.8 ± 35.8%) was only weakly above chance (17.3 ± 2.33%, permutation P = 0.040).

Single neuron response from the no-response and pop-out tasks

To further characterize the nature of target response, we recorded from the same neurons to compare between the three tasks. Note that the no-response task and pop-out task were always performed in the same session as the standard task, and to compare the neuronal response across tasks, neurons were always sorted across the entire session. In both the no-response and pop-out tasks, responses of neurons distinguished between target-present and target-absent trials (Figs 4A, B, 5A and B), similar to the standard task (Fig. 2A and B). Comparing fixations on targets versus distractors (the same selection as in the standard task), we found target neurons comparable to those in the standard task: in the no-response task, 122 neurons had an overall firing rate >0.2 Hz, out of which 35 were target-selective neurons [28.7%; binomial P < 10⁻⁵⁰; 16 target-preferring and 19 distractor-preferring; TSI: 46.3 ± 33.5% (mean ± SD across cells)] (Fig. 4C and D and Supplementary Fig. 3A and B for single-unit examples, and Fig. 4E–H for group results). Although dwell times of fixations on targets (581 ± 150 ms) were significantly longer than those on distractors (302 ± 35 ms; two-tailed t-test, P = 0.00011), qualitatively the same results were derived if we restricted our analysis to fixations with comparable dwell times.

Figure 4.

No-response task. (A–D) Single-unit examples of target-selective neurons. (A and B) Array-on-aligned examples. (C and D) Fixation-aligned examples. (E) Average normalized firing rate of target-preferring neurons (n = 16). (F) Average normalized firing rate of distractor-preferring neurons (n = 19). (G) Cumulative distribution of TSI. (H) Histograms of AUC values. Conventions as in Fig. 2.

Figure 5.

Pop-out task. (A–D) Single-unit examples of target-selective neurons. (A and B) Button press aligned examples. (C and D) Fixation-aligned examples. (E) Average normalized firing rate of target-preferring neurons (n = 20). (F) Average normalized firing rate of distractor-preferring neurons (n = 8). (G) Cumulative distribution of TSI. (H) Histograms of AUC values. Conventions as in Fig. 2. RT = reaction time.

In the pop-out task, 128 neurons had an overall firing rate >0.2 Hz, of which 28 were target-selective neurons (21.9%; binomial P = 6.55 × 10⁻¹²; 20 target-preferring and eight distractor-preferring; TSI: 82.9 ± 113.2%) (Fig. 5C and D and Supplementary Fig. 3C–F for single-unit examples, and Fig. 5E–H for group results). Notably, a significant proportion of target-selective neurons in the no-response task (21; 60.0%; χ²-test: P = 0.0013) and pop-out task (21; 75.0%; χ²-test: P = 1.09 × 10⁻⁶) were also target-selective neurons in the standard task (45.7% in the sessions with comparison experiments). This suggested a consistency in encoding search target across different task formats, a point that we will turn to after we analyse the population response.

Population analysis

How representative were the subsets of neurons described so far of the entire population of recorded pre-SMA neurons? In particular, although non-target-selective neurons could not distinguish between fixations on targets versus distractors individually, could they still do so as a population? To answer these questions, we conducted two population analyses. First, a population regression analysis showed that for all three tasks, only the activity of target-selective neurons explained a significant proportion of variance, but not the activity of non-target-selective neurons [permutation test with 500 runs; standard task: P < 0.001 for target-selective neurons and P = 0.082 for non-target-selective neurons (Fig. 6A); no-response task: P < 0.001 for target-selective neurons and P = 0.85 for non-target-selective neurons (Fig. 6B); pop-out task: P < 0.001 for target-selective neurons and P = 0.44 for non-target-selective neurons (Fig. 6C)]. Second, we used population decoding. Decoding from all recorded neurons together revealed strong ability to differentiate between fixations on targets versus distractors in all three tasks (permutation P < 0.05) (Fig. 6D–F). However, this ability was primarily driven by target-selective neurons (Fig. 6G–I) because decoding from non-target-selective neurons was not possible at levels significantly larger than chance (Fig. 6J–L). This was the case for all three tasks and most of the bins. Together, these population analyses suggest that only the activity of target-selective neurons but not non-target-selective neurons could distinguish between targets and distractors.

Figure 6.

Population analysis of target response using regression and decoding. (A, D, G and J) Standard task. (B, E, H and K) No-response task. (C, F, I and L) Pop-out task. (A–C) Population analysis using a regression model and effect size metric ω². Effect size was computed in a time window of −200 ms before fixation onset to 200 ms after fixation offset and was averaged across neurons for each run. Magenta and grey vertical lines indicate the observed effect size for target-selective (TS) neurons and non-target-selective neurons, respectively. The null distributions for target-selective neurons (magenta) and non-target-selective neurons (grey) were estimated by randomly scrambling the labels and fitting the same models (500 repetitions). For three tasks, the observed effect size was well above 0 for target-selective neurons but not non-target selective neurons. (D–L) Population decoding. Bin size is 500 ms and step size is 50 ms. The first bin is from −200 ms to 300 ms relative to fixation onset, and the last bin is from 100 ms to 600 ms after fixation onset. Error bars denote ±SEM across bootstrap runs and individual circles show values for each bootstrap run. The horizontal black lines indicate the chance level (50%). Asterisks indicate statistical significance estimated from bootstrap: ⁺P < 0.1, *P < 0.05 and ^***P < 0.001. (D–F) Decoding with all neurons. (G–I) Decoding with target-selective neurons only. (J–L) Decoding with non-target-selective neurons only.

Abstract representation of visual targets

We then used single-fixation analysis (TSI and AUC) to compare target selectivity across tasks in more detail. We selected units in one task (the standard task) and then evaluated the same neuron’s response in the no-response and pop-out tasks (note that the same definition of fixations was used). In the sessions with comparison experiments, there were 46 target-selective neurons (30 target-preferring and 16 distractor-preferring) selected from the standard task.

We examined the consistency of target responses between tasks by correlating the normalized firing rate between pairs of tasks. Responses were significantly correlated between the standard and no-response tasks [Fig. 7A; Pearson correlation: r(46) = 0.75, P = 2.53 × 10⁻⁹ for combined target- and distractor-preferring neurons and r(109) = 0.60, P = 4.74 × 10⁻¹² for all neurons]. Similarly, responses were significantly correlated between the standard and pop-out tasks [Fig. 7B; r(46) = 0.90, P = 9.30 × 10⁻¹⁸ for combined target- and distractor-preferring neurons and r(109) = 0.81, P = 6.31 × 10⁻²⁷ for all neurons]. These results suggest that pre-SMA neurons are neither related to motor planning (no button press in the no-response task) nor target or object matching from working memory (no targets shown in the pop-out task). The correlation results were further confirmed by comparison with chance performance (Fig. 7C; note that this test was independent from selection): target-selective neurons still had above-chance discrimination of targets versus distractors in both no-response task [two-tailed one-sample t-test: t(45) = 4.63, P = 3.13 × 10⁻⁵] and pop-out task [t(45) = 3.19, P = 0.0026], showing that target-selective neurons from the standard task could still encode targets in the no-response and pop-out tasks. As the neurons were selected in the standard task only, the smaller out-of-sample predicted mean TSI in the no-response and pop-out tasks (two-tailed paired t-test) was as expected. Note that we only flipped the sign of the TSI for distractor-preferring neurons (Equation 2) to combine target- and distractor-preferring neurons (Fig. 7C) but not in the scatter plot (Fig. 7A and B), where the neurons were only baseline normalized to fixations on distractors and a negative mean normalized firing rate indicated a higher mean firing rate for distractors. Also note that because we correlated the normalized firing rate (i.e. the difference between targets and distractors normalized by the baseline), the above observed correlations could not be simply attributed to differences in absolute firing rate—if there was a significant correlation, it was due to a difference between targets and distractors across tasks but not due to correlated firing rates.

Figure 7.

Comparison of response between task variants. (A and B) Scatter plot of the mean normalized firing rate between the standard and no-response tasks (A) and between the standard and pop-out tasks (B). Each circle represents the mean normalized firing rate for a neuron. Magenta circles denote the target-selective (TS) neurons (derived from the standard task; n = 47) and grey circles denote the unselected neurons. The magenta line is the linear fit for target-selective neurons and the grey line is the linear fit for all neurons that had an overall firing rate >0.2 Hz for both tasks in comparison (n = 109). (C) Mean TSI for each task. Note that to combine target- and distractor-preferring neurons, we flipped the sign of TSI for distractor-preferring neurons (Equation  2). But for the scatter plot, we did not flip the sign of the normalized firing rate for distractor-preferring neurons (thus all neurons were only baseline normalized to fixations on distractors), and therefore, neurons with a negative mean normalized firing rate had a higher mean firing rate for distractors. As the neurons were selected in the standard task only, the mean TSI was smaller in the no-response and pop-out tasks. Error bars denote ±SEM across neurons and circles show individual values. Asterisks indicate significant difference between tasks using paired t-test or against chance performance (0) for each task using paired t-test: ^**P < 0.01 and ^***P < 0.001. SD = standard task; NR = no-response task; PO = pop-out task. (D–E) Scatter plot of the single neuron AUC values between the standard and no-response tasks (D) and between the standard and pop-out tasks (E). Each circle represents the AUC for a neuron calculated based on fixations. (F) Mean AUC values for each task. Asterisks indicate significant difference between tasks using paired t-test or against chance performance (unselected neurons) for each task using paired t-test: ^***P < 0.001. Legend follows A–C. (G–L) Cross-task generalization of population decoder. (G–I) No-response task. (J–L) Pop-out task. (G and J) Decoding with all neurons. (H and K) Decoding with target-selective neurons only. (I and L) Decoding with non-target-selective neurons only. Error bars denote ±SEM across bootstrap runs and individual circles show values for each bootstrap run. The horizontal black lines indicate the chance level (50%). Asterisks indicate statistical significance estimated from bootstrap: ⁺P < 0.1, *P < 0.05 and ^***P < 0.001.

We also repeated the above analysis using distractors before the first fixation on targets and/or only the first fixation on target and found qualitatively the same results. Furthermore, when we selected target neurons from the no-response task in order to predict response type in the standard task, we also observed a significant correlation [r(35) = 0.78, P = 4.77 × 10⁻⁸] for combined target- and distractor-preferring neurons; same correlation for all neurons as shown above. When we selected neurons from the pop-out task in order to predict response type in the standard task, we again observed a significant correlation [r(28) = 0.94, P = 4.19 × 10⁻¹³].

We further used AUC to compare across tasks, which mirrored the above results. We observed a significant correlation between the standard and no-response tasks [Fig. 7D; r(46) = 0.48, P = 0.00071 for combined target- and distractor-preferring neurons and r(109) = 0.50, P = 3.00 × 10⁻⁸ for all neurons], as well as a significant correlation between the standard and pop-out tasks [Fig. 7E; r(46) = 0.58, P = 2.10 × 10⁻⁵ for combined target- and distractor-preferring neurons and r(109) = 0.65, P = 1.53 × 10⁻¹⁴ for all neurons]. We also found above-chance AUC values for each task (Fig. 7F) by permutation tests (all P-values < 0.001) and by comparing against baseline AUC values (two-tailed unpaired t-test: all P-values < 10⁻⁴), which was computed using fixations in each task from neurons that were neither selected as target- nor distractor-preferring in the standard task because we always had AUC values > 0.5. As the neurons were selected in the standard task only, the smaller out-of-sample predicted AUC in the no-response and pop-out tasks was as expected.

Lastly, we used population decoding to test the generalizability of the target response across tasks. To achieve this, we first trained a decoder to differentiate between fixations on targets and distractors based on the activity of the neurons recorded in the standard task. We subsequently tested the decoder in the no-response (Fig. 7G–I) and pop-out (Fig. 7J–L) tasks. We found that a decoder trained this way using all recorded neurons was able to classify fixations as targets versus distractors with above chance performance in both no-response (Fig. 7G) and pop-out (Fig. 7J) tasks. Similarly, consistent with the within-task population decoding (Fig. 6D–L), cross-task prediction was also primarily driven by target-selective neurons (Fig. 7H for no-response task and Fig. 7K for pop-out task) and non-target-selective neurons did not show above-chance cross-task generalization (Fig. 7I for no-response task and Fig. 7L for pop-out task). Furthermore, consistent with the cross-task comparison results using TSI (Fig. 7A–C) and AUC (Fig. 7D–F), target-selective neurons showed better generalization to the pop-out task compared to the no-response task.

Taken together, our results show that pre-SMA target-selective neurons encode search goals irrespective of motor output, goal format, or a template in working memory. This suggests that the activity of pre-SMA target-selective neurons represents targets in an abstract format irrespective of these variables during goal-directed visual search.

Temporal response characteristics of target-selective neurons

Above, we considered all fixations on distractors regardless of whether these occurred before or after the target had been fixated intermittently. However, in a subset of 34.4 ± 12.6% of trials, subjects continued to fixate on at least one of the distractors after they first fixated on the target. We used these trials to explore how target-selective neurons responded as a consequence of fixating on targets.

In the standard task (Supplementary Fig. 4A), target-selective neurons had a significantly higher TSI for fixations on targets (81.2 ± 14.9%, mean ± SEM across cells) compared to those on distractors fixated before first fixating the target (−2.35 ± 0.61%; KS-test: P = 7.78 × 10⁻²⁴). This was also true when considering only the first fixation on the target in each trial compared to the distractor fixations that preceded this first target fixation (71.8 ± 12.8% versus −2.35 ± 0.61%; KS-test: P = 3.66 × 10⁻²¹), and it remained true when only considering distractor fixations that followed a fixated target (81.2 ± 14.9% versus 52.8 ± 12.2%; KS-test: P = 0.0024). Thus, target responses remained sensitive to the type of stimulus fixated (target or distractor) even after the target had been fixated. However, the TSI for distractors was significantly greater for fixations on distractors that were fixated after the target was already fixated, than for distractors fixated before the target was fixated (52.8 ± 12.2% versus −2.35 ± 0.61%; KS-test: P = 9.54 × 10⁻¹³). This indicates that despite subjects not stopping the search immediately by button press, the presence of the target did influence the response of target-selective cells immediately following the first fixation on the target.

In the no-response task (Supplementary Fig. 4B), target-selective neurons had a significantly higher TSI for fixations on targets (46.3 ± 5.66%) compared to those on distractors fixated before first fixating the target (−1.89 ± 2.27%; KS-test: P = 8.63 × 10⁻¹⁶) and this was also true for the first target fixation (31.5 ± 6.54%; KS-test: P = 4.42 × 10⁻⁸). In the pop-out task (Supplementary Fig. 4C), target-selective neurons had a significantly higher TSI for fixations on targets (82.9 ± 21.4%) compared to those on distractors fixated before first fixating the target (−13.5 ± 8.16%; KS-test: P = 1.18 × 10⁻¹²) and this was also true for the very first target fixation alone (90.2 ± 27.0%; KS-test: P = 1.18 × 10⁻¹²). Similar to the standard task, target-selective neurons remained sensitive after the target had been fixated (82.9 ± 21.4% versus 49.5 ± 24.0%; KS-test: P = 0.019), and fixating on the target increased TSI of distractors (KS-test: P = 1.76 × 10⁻⁶)

In summary, we found that target-selective neurons had similar temporal response characteristics across tasks, distinguishing targets from distractors across all epochs. This result further supports task-invariant response of pre-SMA neurons.

Visual selectivity is weak in the pre-SMA

We next examined visual selectivity of pre-SMA cells by quantifying their category selectivity. Visual selectivity is a prominent feature of medial temporal lobe cells during both passive and active viewing of stimuli (Kreiman et al., 2000; Minxha et al., 2017); both are generally weak or absent in medial frontal cortical areas in humans (Kaminski et al., 2017). We thus investigated whether this result also held during visual search, which has not been investigated. This analysis was based on the standard task. To identify face-selective neurons, we compared the response between face and non-face stimuli during cue presentation. Only 15 neurons (11.6%; binomial P = 7.48 × 10⁻⁴) had a significant response difference (9/15 increased their firing rate for faces compared to non-faces, whereas 6/15 increased their firing rate for non-faces compared to faces but did not further distinguish non-face categories). To select category-selective neurons, we grouped the objects into 14 visual categories. The response of only 11 neurons (8.53%; binomial P = 0.029) co-varied significantly as a function of visual category during cue presentation (one-way ANOVA of 14 categories with P < 0.05), consistent with the results from a working memory task (Kaminski et al., 2017). Together, our results show that pre-SMA neurons have only weak visual selectivity during visual search. This suggests that pre-SMA neurons were primarily top-down driven in this task.

Pre-SMA neurons signal fixation order

We have shown above that pre-SMA neurons signal the presence of targets. How do pre-SMA neurons respond in target-absent trials? Because the pre-SMA is involved in generating complex sequences of movements (Shima et al., 1996; Nachev et al., 2008), we hypothesize that there is a subset of pre-SMA neurons that signal the sequential order of fixations by gradually increasing or increasing their firing rate during a trial (or, alternatively, time elapsed). Indeed, we found such ‘fixation order’ neurons in the pre-SMA (Fig. 8A and B). Apart from its own importance, this offered us a second opportunity to assess the consistency of representation in the pre-SMA between tasks.

Figure 8.

Fixation order neurons. (A) Example neuron that increased firing rate over the time course of search. (B) Example neuron that decreased firing rate over the time course of search. Each raster (top) and peristimulus time histogram (PSTH) (bottom) is shown with colour coding as indicated. Trials are aligned to cue presentation (grey lines). Trials within each category are sorted according to reaction time (black line). Waveforms for each unit are shown in the raster plot. Shaded area denotes ± SEM across trials. (C–N) Fixation order neurons were selected using the first 10 fixations from each target-absent trial. (C–F) Standard task. (G–J) No-response task. (K–N) Pop-out task. (C, G and K) Single-neuron examples. Each circle represents a single fixation. (D, H and L) Mean normalized firing rate for neurons that increased firing rate as a function of fixation order. (E, I and M) Mean normalized firing rate for neurons that decreased firing rate as a function of fixation order. Error bars denote ±SEM across neurons and circles show individual values. (F, J and N) Overlap between target-selective neurons and fixation order neurons.

To formally capture this result, we used a linear regression to identify neurons whose firing rate correlated with fixation order. Here, we focused on target-absent trials in order to disentangle the target response. We found that in all three task variants, there was a population of such ‘fixation order’ neurons (standard task: 73 neurons, 56.6%, binomial P < 10⁻⁵⁰; 30 and 43 neurons increased and decreased firing rate with fixation order, respectively; Supplementary Fig. 5B and C; no-response task: 46 neurons, 37.7%, binomial P < 10⁻⁵⁰; 26 and 20 neurons increased and decreased firing rate with fixation order, respectively; Supplementary Fig. 5F and G; pop-out task: 59 neurons, 46.1%, binomial P < 10⁻⁵⁰; 31 and 28 neurons increased and decreased firing rate with fixation order, respectively) (Supplementary Fig. 5J and K; see Supplementary Fig. 5A, E and I for single cell examples). Similar to target-selective neurons, this again showed a consistency of representation in the pre-SMA, but this time for neurons encoding fixation order across different task formats.

To ensure that the correlation was not driven by outliers (Supplementary Fig. 5A, E and I), we repeated our analysis by using only the first 10 fixations in the serial order. We chose 10 fixations for analyses because this could capture most of the visual search process on the one hand and on the other hand, would not leave the average at later data points too noisy (Wang et al., 2014b, 2018). We found similar results (standard task: 47 neurons, 36.4%, binomial P < 10⁻⁵⁰; 21 and 26 neurons increased and decreased firing rate with fixation order, respectively) (Fig. 8D and E; no-response task: 33 neurons, 27.1%, binomial P = 1.11 × 10⁻¹⁶; 12 and 21 neurons increased and decreased firing rate with fixation order, respectively; Fig. 8H and I; pop-out task: 42 neurons, 32.8%, binomial P < 10⁻⁵⁰; 30 and 12 neurons increased and decreased firing rate with fixation order, respectively; Fig. 8L and M; see Fig. 8C, G and K for single cell examples).

Were fixation order cells dependent on the response of target-selective cells? We found that 22 neurons from the standard task (Fig. 8F), nine neurons from the no-response task (Fig. 8J), and 10 neurons from the pop-out task (Fig. 8N) were selected as both target-selective and fixation order cells. However, the proportion of cells that qualified as both was not greater than expected from independence of these two attributes, i.e. target-selective neurons had a similar percentage of fixation order neurons as the entire population (χ²-test; standard task: P = 0.40, Fig. 8F; no-response task: P = 0.88, Fig. 8J; pop-out task: P = 0.77, Fig. 8N) and fixation order neurons had a similar percentage of target-selective neurons as the entire population (χ²-test; standard task: P = 0.39, Fig. 8F; no-response task: P = 0.87, Fig. 8J; pop-out task: P = 0.79, Fig. 8N). Similar results were found for selections using all fixations (χ²-test; all Ps > 0.29; Supplementary Fig. 5D, H and L). Together, this result suggests that target-selective and fixation order cells are largely distinct.

Lastly, similar results were derived when we repeated our analysis by correlating the elapsed time (250 ms consecutive bins, starting from array onset) with firing rate (standard task: 54 neurons, 41.9%; no-response task: 33 neurons, 27.1%; pop-out task: 34 neurons, 26.7%). This result suggests that fixation order cells were not sensitive to fixation properties but time elapsed.

Discussion

In this study, we characterized pre-SMA target cells by systematically comparing their response between three different tasks and by correlating their response with behaviour in each task. We found that the target-selective neurons that we identified previously (Wang et al., 2018) were not driven by motor output and did not depend on the format of search goals. Together, our results show that pre-SMA neurons play a critical role in task-invariant goal detection of ongoing search.

What does this target signal mean? First, the target response is not merely a task-set signal (Dosenbach et al., 2006) because both target-present and target-absent trials should otherwise have a similar response. We did not observe this, and to the contrary, target-selective cells were highly selective to finding the target. Relatedly, although monkey studies have shown that the pre-SMA is involved in maintaining or updating action plans (Matsuzaka and Tanji, 1996; Shima et al., 1996; Isoda and Hikosaka, 2007), our target responses do not represent only an ‘end-of-search’ signal because both target-present and target-absent trials contain a termination decision. Second, visual search typically requires holding the target in working memory, and neurons in the pre-SMA have been shown to encode aspects of working memory during working memory maintenance (Kaminski et al., 2017). However, the pop-out task results indicate that pre-SMA target neurons do not depend on explicitly defined targets held in working memory, but can encode conceptually defined targets whose identity needs to be discovered from the stimulus array. Consistent with the abstract presentation of search goals observed in this study, it has been shown in monkeys that the same pre-SMA neurons encode both tactile and acoustic information in working memory, with the same representation for both modalities (Vergara et al., 2016). Third, when subjects attend to their intention rather than their movement, there is an enhancement of activity in the pre-SMA (Lau et al., 2004), consistent with the present finding that target neurons encode search goals independent of motor output. Therefore, it is possible that the target response might, at least in part, reflect attentional arousal that follows target detection.

A possible interpretation of the significance of the target-selective cell response is that these neurons predict behavioural events (Hoshi and Tanji, 2004) and/or encode anticipated reward (Berdyyeva and Olson, 2011), following successful detection of a target. Detecting search targets are behaviourally important events in all three tasks in the present study. Thus, pre-SMA target-selective neurons may encode behaviourally relevant events, which in turn explains why they had consistent activity in response to various targets and actions. Alternatively, the activity of target-selective cells may be related to the transition from exploration to exploitation, which has been reported in the supplementary eye field in monkeys (Kawaguchi et al., 2014). It has been reported that pre-SMA activity is low when monkeys perform repeated actions, whereas pre-SMA neurons are activated when new actions are performed (Shima et al., 1996). In addition, it has been shown that the pre-SMA has a function in switching from automatic to volitionally controlled action in macaques (Isoda and Hikosaka, 2007). This is consistent with our present result that the pre-SMA dynamically encodes trial-by-trial search goals independent of task formats.

Prior monkey literature has shown that the pre-SMA plays a key role in motor learning (Matsuzaka et al., 1992), planning (Tanji and Shima, 1994), updating (Shima et al., 1996), and sequencing multiple saccades (Isoda and Tanji, 2004). However, in the present study, we found that the human pre-SMA encodes search targets independent of motor output. Our fixation-based analysis revealed target responses in the no-response task, regardless of where the target was located on the screen, arguing against a simple role of the pre-SMA in coding hand movements. It has been shown that neurons in the pre-SMA are related not only to switching eye-movement responses, but also to response inhibition or facilitation (Isoda and Hikosaka, 2007). The pre-SMA lies at the intersection between the SMA and the supplementary eye field and our findings bear some resemblance to the function of the supplementary eye field. The supplementary eye field may be considered an ocular motor extension of the SMA (Schall et al., 2002) and can exert contextual executive control over saccade generation (Stuphorn and Schall, 2006). Although it has also been argued that the supplementary eye field plays a limited role in controlling ongoing visual search behaviour, it may play a larger role in monitoring search performance (Purcell et al., 2012). Future experiments will be needed to further distinguish pre-SMA’s role in coding hand versus eye movements. Lastly, similar to the role of pre-SMA neurons, the frontal eye fields plays a critical role in visual search by providing top-down attentional influence towards target relevant features (Zhou and Desimone, 2011). This indicates a relationship between the frontal eye fields and pre-SMA in goal-directed behaviour, which needs to be investigated in future studies.

We observed that a subset of pre-SMA neurons gradually increased or decreased their firing rate as a function of fixation order. It is worth noting that these neurons were not specific to a certain sequence of saccade like previously reported for some pre-SMA neurons in macaques (Isoda and Tanji, 2004) and human neuroimaging studies (Hikosaka et al., 1996). One interpretation of this result is that the gradual change in neuronal activity reflects the passage of time from the beginning of the search, consistent with interval time coding by pre-SMA neurons in monkeys (Mita et al., 2009). In our task, time elapsed is an important variable because the task involved deciding that a target was not present.

In humans, there are several physiological indices of target detection and the target neurons observed in the present study may explain these responses. Recordings of scalp event-related potentials (ERP) have shown that target detection elicits a consistent response appearing around 300 ms (P300) after stimulus onset (Polich, 2007). The P300 reliably distinguishes between top-down defined targets and distractors, e.g. in cases where observers are asked to pay attention to the letter ‘p’ presented in a sequence of successively flashed letters (Farwell and Donchin, 1988), and this component might be associated with attention (Polich, 2007). Notably, consistent with the present result, prior studies have suggested that the P300 arises from the medial frontal cortex including the pre-SMA (Yoder and Decety, 2014; Sun et al., 2017). Although the vast majority of ERP experiments involve the detection of a target stimulus embedded in a temporal sequence of distractors while subjects fixate at the centre of the screen, due to the control of contamination by potentials arising from the extraocular muscles, in a simple spatial visual search using letters (E versus inverted E), the P300 as well as an early component (∼150 ms) are also observed to encode target detection (Kamienkowski et al., 2012). Furthermore, in a more complex spatial visual search of face in a crowd, fixation-aligned P300 differentiates fixations on target face versus fixations on distractor face, across multiple frontal and parietal electrodes (Kaunitz et al., 2014). Better preparatory attention (reflected by less pre-stimulus alpha-band oscillatory activity) and better post-stimulus early visual responses (reflected by larger sensory N1 waves) predict the latency of target detection (van den Berg et al., 2016). Interestingly, ERPs aligned at saccade onset distinguish not only between targets and distractors (with a larger P300 amplitude for targets than distractors) but also correctly detected targets and missed targets (Brouwer et al., 2017).

Our data support the view that dysfunction in the pre-SMA might be expected to manifest with poorer ability to perform goal-directed behaviours, independent of the task itself or the end goal. In contrast, hyperfunctioning of the pre-SMA might result in over focused attention on goal-directed behaviours to the detriment of competing behavioural goals such as social interaction, attending to cues in the surrounding environment, and switching away from a goal-directed task when environmentally appropriate. If these data can be further verified and the role of pre-SMA characterized, these observations may lead directly to neuromodulation strategies to improve cognitive functions in patients with disorders of the pre-SMA.

Glossary

Abbreviations

pre-SMA =

pre-supplementary motor area

TSI =

target selectivity index