Temporal continuity shapes visual responses of macaque face patch neurons

Summary

Macaque inferior temporal cortex neurons respond selectively to complex visual images, with recent work showing that they are also entrained reliably by the evolving content of natural movies. To what extent does temporal continuity itself shape the responses of high-level visual neurons? We addressed this question by measuring how cells in face-selective regions of the macaque visual cortex were affected by the manipulation of a movie’s temporal structure. Sampling a 5m movie at 1s intervals, we measured neural responses to randomized, brief stimuli of different lengths, ranging from 800ms dynamic movie snippets to 100ms static frames. We found that the disruption of temporal continuity strongly altered neural response profiles, particularly in the early response period after stimulus onset. The results suggest that models of visual system function based on discrete and randomized visual presentations may not translate well to the brain’s natural modes of operation.

eTOC Blurb

Although we experience the world as a continuous stream of information, our knowledge about the visual brain stems primarily from its responses to brief and isolated stimuli. Here we show in the macaque object pathway that selectivity to natural movie content is strongly determined by visual continuity and temporal context.

Introduction

We experience the world as a continually evolving sequence of events, scenes, and interactions. Humans and other animals use their vision to track and integrate a wide range of information, coordinating the inherent dynamics of the natural world with their own active sampling of important scene elements^1,2. This integration provides a predictive context for interpreting the flow of incoming visual signals. What role might high-level visual neurons, such as those observed throughout the macaque inferior temporal cortex, play in such integration? For example, would the visual response profile of face-selective cortical neurons depend on the temporal continuity of a video? What might be the neural consequences of breaking up normal visual experience into discrete, randomized presentations?

It is difficult to make predictions on this point, since nearly all that is known about visual responses in the brain derives from the brief presentation of stimuli, often in highly reduced forms. For example, most studies of the macaque inferior temporal cortex have used isolated images of natural and artificial stimuli flashed briefly on the screen to study object selectivity among visual neurons^3–7. By comparison, only a few single-unit studies have investigated how such neurons respond under more naturalistic conditions, for example when a subject is allowed to freely gaze within an evolving visual scene^8–10. Thus, much more is known about how the brain responds to flashed presentations of isolated stimuli than how it responds under the continuity and complexity of everyday vision.

In some areas, researchers have studied the brain’s encoding of visual temporal sequences, such as in the processing of bodily actions^4,11–14. These studies have linked the comprehension of bodily actions to the integration of form, movement, and location¹⁵, and in some cases to the generation of one’s own bodily actions^11,16–18. Reductionistic investigations of sequence processing in the inferotemporal cortex has in some cases identified a marked temporal dependency, which has often been linked to Hebbian or other forms of statistical learning through repeated sequence presentation^19–24. These studies provide evidence that object-selective neurons in the inferior temporal cortex exhibit significant temporal dependency in their responses.

A somewhat different facet of temporal integration involves the active sampling of a visual scene provided by the frequent redirection of gaze^1,25,26. For humans and other primates, whose high-resolution vision is sharply concentrated in the fovea, the sequential gathering of important information across fixations is a central feature of visual cognition²⁷. In normal vision, the temporal discontinuities introduced by this active sensing of the visual scene through gaze redirection is superimposed upon the temporal dynamics of actors and events in the external world. The stable perception of the world amid the constantly shifting retinal stimulation is one of the enduring puzzles of visual neuroscience^28–30.

Recently, the use of naturalistic stimulus paradigms has gained popularity and has begun to complement more conventional presentation methods (for a recent review, see Leopold and Park³¹). The human fMRI community has, in particular, taken up the use of movies as a new paradigm for tapping into various aspects of brain function^32–48, with some monkey investigators following suit^49–58. One single-unit study investigated responses in macaque anterior fundus face patch to repeated 5-minute presentations of naturalistic movies full of dynamic social interaction⁸. Neurons in this area were strongly entrained by the movie timeline, with a high level of response repeatability across presentations of the same movie despite the free movement of the eyes. However, the role of temporal continuity in shaping neural responses to complex movie content is at present poorly understood. While not yet studied in detail, this question may be important, since our understanding of high-level vision is at present shaped strongly by the brain’s response to the presentation of discrete and isolated stimuli.

The current study addresses the role of temporal continuity on neural responses in the ventral visual pathway using a paradigm in which we systematically manipulated the temporal continuity of visual content stemming from a naturalistic movie. We recorded from cells in the macaque anterior fundus (AF) and the anterior medial (AM) face patches, in which responses to both static images and dynamic movies have previously been studied^{3,8,54,59–68}. The goal of present study was not specifically related to face processing, but instead used the well characterized responses in these areas to understand the influence of visual continuity (the uninterrupted viewing of stimulus) and temporal context (the effect of preceding meaningful visual content) over high-level visual responses. Our approach was to sample visual snippets from throughout a movie, thus breaking its natural continuity, and to observe whether neurons would maintain their selectivity when the movie content was delivered in discrete and randomized form. We found that this disruption of temporal continuity strongly and significantly affected neural responses to the movie content in both areas we recorded, as well as over multiple timescales. The most profound impact was on the response to the initial onset of the discrete presentation, for which neural responses were nearly uncorrelated to those observed in response to the same content of the intact movie.

Results

Six macaque monkeys (Macaca mulatta) viewed intact movies and randomized movie clips (“snippets”) as visual responses were recorded from single neurons in the anterior fundus (AF) and anterior medial (AM) face patches. These regions were defined functionally using fMRI⁶³ and targeted using methods described previously^59,60 (Figure 1a, see Methods). In total, we recorded from 163 neurons in the AF face patch (Subject T: 18; Subject R: 16; Subject S: 91; Subject SR: 38) and 72 neurons in the AM face patch (Subject D: 49; Subject M: 23). For the main analyses, we combined neurons across regions into a single population after determining that the results were similar in both regions. Corresponding analyses performed separately in the two face patches are presented in supplemental material. The animals viewed multiple repetitions of an intact 5-minute movie. They also viewed randomized short snippets and static images extracted from the movie at each of 300 time points spaced by one second (see Methods). Most of the analyses described below involve comparing the responses to 300 equivalent visual segments from the original, intact movie with either 800 ms movie snippets or 100 ms static images. Corresponding responses to two additional conditions (250 and 100 ms movie snippets) are also discussed. Five of the monkeys (T, R, S, D, M) were free to direct their gaze while viewing the continuous movie and randomized snippets. In addition, two of the monkeys (S and SR), viewed the same movie and snippets under controlled fixation.

Figure 1. Experimental Design.

a) Top row shows representative frames from the intact movie that was presented during the free viewing paradigm. Middle row depicts the results of the fMRI face-patch localizer task used to target the AF (left) and AM (right) face patches. Functional localizer is overlaid on a T1w image taken after the electrodes where implanted (see Methods). The bottom row depicts the spiking response from one AF (left) and one AM (right) neuron to the viewing of multiple trials of the intact movie. The black dots represent the time of a spike with 1 ms bins, and each row is a trial. The red line is the mean spike density function for all trials. b) Schematic of the creation of the individual snippets from the original movie, and the reconstructed response to those snippets. c) Schematic of the pseudorandomized task design that was used to collect the data within and across sessions. d) Schematic for the selection of response periods based on the individual snippet lengths. e) Schematic of the regression analysis between the snippet responses and the original movie. The green ellipse represents a hypothetical goodness of fit measure.

The recordings hinged on the longitudinal recording capabilities of the implanted microwire bundles since the data for individual neurons was amassed over 8–18 daily sessions of data collection^8,59(see Fig S1). Each of these sessions began with several minutes of conventional testing with flashed images from different stimulus categories, which was used both to establish each neuron’s basic selectivity and as a tool for “fingerprinting” to identify the same neurons across subsequent recording sessions. This testing was followed by all other conditions presented in randomized blocks (Figure 1c). We first investigated the effects of temporal continuity on movie content during free viewing. In this condition, the intact movie was viewed between 8 and 24 times (median for viewings per neuron: monkey T = 8, monkey R = 10, monkey S = 15, monkey M = 24, monkey D = 18). The randomized movie snippets were viewed between 7 and 37 times each (median viewings per neuron: 800 ms: T = 7, R = 10, S = 13, M = 13, D = 12; 250 ms: T = 7, R = 11, S = 16, M = 14, D = 17; 100 ms: T = 7, R = 9, S = 17, M = 13, D = 20; static image: T = 13, R = 17, S = 37, M = 33, D = 28).

Effects of manipulating temporal continuity during free viewing

Previous work has shown that neurons in the areas we recorded are reliably entrained to the movie under conditions of free gaze, when the animal actively samples the content of the movie. We first asked whether, under these conditions, the neural responses depend on the temporal continuity of the movie. We found that breaking up the movie into snippets or frames and randomizing their presentation strongly influenced the stimulus response selectivity of neurons in these areas. The differences were most obvious in the early responses, from 100–300 ms following stimulus onset, but were also evident in the later response period to video snippets.

Figure 2 shows an example neuron (neuron AF094) responding to the same video content in continuous versus randomized snippet presentation. This neuron was typical in that it was face-selective and entrained reliably by the movie. Figure 2a shows the raw raster responses during several representative 800 ms snippets and corresponding static frame presentations. From these example rasters, the repeatability across presentations and the effects of randomization are both evident. Figure 2b then shows the reconstructed time courses to the 800 ms snippets early onset responses (red, 100–300 ms following snippet onset) and the later responses (blue, 300–900 ms following snippet onset) in comparison to the equivalent time periods from the continuous movie. The responses to the randomized presentation are reordered to match the native timeline of the movie (see Figure 1b, Methods). Both reconstructed time courses revealed notable deviations in spiking during the corresponding analysis windows of the continuous presentation (gray time courses).

Figure 2. Representative response of a neuron from area AF.

Six representative time points are featured (a), which show the diversity of response across time points and contexts. We compared stimulus-driven response during the intact movie (top row), 800 ms snippets (middle row) and 100 ms static center- frame (bottom row). The rows of spiking data show the raster plots and computed average spike density function during the three conditions for the featured time points. The mean firing rate (Hz) to the static categorical images is presented next to the snippet responses, highlighting the current neuron’s face responsiveness. This neuron’s average firing rate during the category task is shown in the right most inset. Responses to the category task were calculated over a 200ms window starting at stimulus onset and averaged across all presentation within a given category. b) Average intact movie response (gray) shown together with the reconstructed spike count responses to the early response (red, top), the late response (blue, bottom) using matching analysis windows relative to the center frame (see Figure 1b,d). c) Analysis of response to static frames (orange), compared to reconstructed response in same format as b.

Figure 2c applies a similar analysis to spiking responses to the static single frames drawn from the movie, which showed little correspondence to the spiking activity in the continuous condition. Scatterplots showing the response correspondence for the 300 sample time points are shown to the right of each reconstruction. For this example neuron, the initial onset responses to both the snippet (red) and static frame (yellow) were uncorrelated with the responses to the equivalent visual content presented in the continuous condition (snippet: r = 0.0665, p = 0.2506; static frame: r = −0.031, p = 0.5933). The responses to content later in each snippet (blue) were affected by randomization but retained their correlation with the continuous responses (r = 0.55, p = 0.01e-23). Additional single unit examples from both areas are shown in Figure S2.

Across the population, neural selectivity was strongly affected by the fragmented and randomized presentation of the movie content. Comparing the neural responses to the same movie frames presented in its original format, the most prominent changes in selectivity were observed in the response directly following the onset of the snippet or static frame. In fact, we found that for many visually responsive neurons, the selectivity measured following the onset of a snippet was statistically uncorrelated with that measured to the same frames during the native movie presentation. We quantified this relationship with a linear regression applied to the 300 samples compared across the corresponding conditions (see scatterplots in Figure 2). This analysis provided a measure of goodness-of-fit (r²), essentially summarizing the similarity in responses to the same movie content in continuous and discontinuous presentation modes.

Figures 3b and 3e show that for most neurons the visual selectivity to the initial presentation of the 800 ms snippets and static frames, respectively, had a poor correspondence to that observed during the continuous condition (Fig 3b, r²_mean = 0.11 +/− 0.12; Fig 3e, r²_mean =0.06 +/− 0.08). In both cases, the r² distribution dropped sharply from a control analysis, in which a split-halves analysis was used to compute the r² distribution from the movies using the same number of trials (paired t-test, Fig 3b, paired samples t(103)= 25.99, p = 0.5e⁻⁴⁸; Fig 3e, paired samples t(103) = 22.18, p = 0.5e⁻⁴²). These findings demonstrate that the abrupt appearance of a static image or dynamic clip on a blank background elicits visual responses that differ markedly from the responses to the same content presented within the context of the movie (see Figure S3 for analyses for separate analysis of AF and AM). In addition to the 800 ms movie snippet featured above, analysis of the onset period for the independent 250 ms and 100 ms snippet presentations yielded similar results (Figure S3).

Figure 3. Population Response changes across context and time.

a) Schematics of the free-viewing paradigm used during the movie-watching condition. Dotted areas represent the size of the stimulus (white) and reward windows (green). The reward window was not shown to the animals. The density plot shows the summed gaze positions during the free-viewing of the within context movie. b) Paired comparison within context and between context goodness-of-fits (r²) of neurons in the AF and AM face-patches during the onset period. Neurons in the pair comparison were filtered by an r² or 0.2 in the within-context condition. The black lines highlight the example neurons presented in Figures 2 and S2b. The vertical histograms represent the density of r²-values for the split halves within-context (black) and between context (red) for the full population. The horizontal histogram, on the right, shows the r² for all trials of the between context regression for the full population. c) depicts the same data and analyses for the late period of the 800 ms snippets. d) show the evolution of the goodness-of-fit for the within-context (black), out-of-context (blue) and between context (red) regressions using a sliding window of 50 ms (yellow box) with 25 ms jumps, for both the 800 ms and 250 ms snippets. Error bars represent the standard-error of the mean of the population within each bin. e) Shows the same set of comparisons as (b) and (c) for the response to the presentation of the static frames.

Later in the response to the snippet, this disruption of visual selectivity was less pronounced (Figure 3c). Comparing visual responses during this period to the native movie allowed us to assess the effects of temporal continuity itself. By analyzing responses beginning 300ms following the onset of the snippet, the transient effects were largely diminished (see Figure 3d), thus we could focus on potential effects of temporal randomization of responses to the movie content. Individual neurons varied in the level of disruption of their late period responses, with some neurons moderately affected by temporal reordering (e.g. Figure 2, Figure S2a) and others showing little if any influence (e.g. Figure S2b, Figure S2c). Across the combined AF/AM population, the influence of temporal continuity was evident as a broad distribution of r² measures (Figure 3c, r²_mean =0.31 +/− 0.17), which differed markedly from the narrow r² distributions of the early responses (see Figure S3 for separate analysis of AF and AM). As above, the r² values were contrasted with those obtained from a within-session split-halves control condition. Importantly, responses to movie content late in the snippet period were also markedly disrupted by randomized presentation (paired samples t(148) = 30.78, p = 0.3e⁻⁶⁶), though less so than during the early period (repeated measures ANOVA (Context × Time Frame): main effect of Context F(1, 179)=510.46; p=0.3e⁻⁵³; Main effect of Time Frame F(1,179)=862.39 p=0.2e⁻⁷⁰; but no interaction F(1, 179)=0.56; p=0.45). Thus the visually selective responses in these face selective areas are also affected by temporal continuity expressed over longer time scales.

Figure 3d plots the temporal evolution of selectivity for the snippets (800ms and 250ms) relative to the same content presented in the native movie (see Methods). The running r² values show that the movie-to-snippet correlation is very low during the initial response period, climbing gradually and approaching its plateau after approximately 300 ms. The initially low r² values reflect the neural response to the abrupt onset of the visual snippet. Following this initial onset period, the movie-to-snippet r² remains lower than the other within-condition comparisons during the late period, reflecting the different temporal context resulting from snippet randomization. It should be noted that the absolute r² magnitude of the time course is lower than in the other panels because the correlation in the running analysis is computed over brief, 50ms time windows. These results demonstrate two clear response phases following the onset of the movie snippets: namely, an early onset period in which response selectivity is severely disrupted, and a later period in which the onset effects have subsided, but the temporal reordering of the movie content continues to affect visual responses.

Effect of manipulating temporal continuity during controlled fixation

The experiments above demonstrate that, when the monkey is allowed to freely direct its gaze the response to visual content depends on whether it is seen in the context of a continuous movie or isolated within an extracted snippet. While this finding indicates that temporal continuity is critical for shaping high-level neural responses, one potentially confounding factor is the prospect that the animal’s gaze position differs in the continuous versus snippet conditions and is possibly driving the context effects through different receptive field stimulation^{5–7,69–72}. To test this possibility, we repeated the experiments above under conditions of controlled fixation. We recorded 55 neurons from two animals’ AF face patches (17 new neurons from Monkey S, and 38 neurons from task naïve Monkey SR), focusing on the repeated presentation of the full length 5-min movie and the randomized 800ms snippets. In this condition, the animals were required to maintain their gaze continuously within 2 or 1.5 degrees of a central fixation point (Figure 4a, Figure S4a), ensuring similar visual position of scene elements across trials and conditions.

Figure 4. Population Response during gaze control.

Data from the AF face patch while subjects maintained their gaze on a central fixation point. a) Schematics of the fixation paradigm used during the gaze control condition. Dotted areas represent the size of the stimulus (white) and fixation windows (green). The fixation window was not shown to the animals. Small inset windows show the summed density of gaze positions throughout the within context movie presentation. b) and c) show the paired comparison of within context and between context goodness-of-fits (r²) of neurons in the during the early and late response periods respectively in the same format as Figure 3. d) show the evolution of the goodness-of-fit for the within-context (black), out-of-context (blue) and between context (red) regressions using a sliding window of 50 ms (yellow box) with 25 ms jumps. Error bars represent the standard-error of the mean of the population within each bin. e) Paired comparison of within and between context goodness-of fit, in the same format as Figure 3b. The static frames were presented for 300 ms without ISI in this experiment.

The results from the controlled fixation condition were clear. Just as during the free viewing condition, the disruption of temporal continuity strongly and significantly altered the selectivity to specific visual content (Figure 4). As above, we measured this change through the decreased goodness-of-fit between a direct comparison of neural responses during the within-context (i.e. movie) and out-of-context (i.e. snippet) conditions, restricted to periods in which the animals were fixating continuously. For both the early period (100–300 ms; t(54)= 11.8, p = 0.15^e−15; Figure 4b) and the later period (300–900 ms; t(54)= 13.501, p = 0.6^e−18, Figure 4c) of the snippet response, responses were markedly affected by the removal of the video content from the natural temporal context. When we calculated the sliding window as above (Figure 4d), we again found that the correspondence between the within and out-of-context responses began to increase shortly after 100ms into the snippets and plateaued around 400ms, never reaching the r² of the within-context responses. In contrast to the free-viewing analysis, here the snippet-to-snippet comparison showed the highest overall values, after an initial rising phase and then plateau at the same level as the movie-to-movie analysis. We interpret this increased correspondence as reflecting a brief, albeit potentially important, consequence of controlled fixation during initial snippet onset period. Finally, as above, the strongest changes in response selectivity were observed for the flashed static frames extracted from the movie. This finding was observed even when blank periods were eliminated between presentations of the static frames (Figure 4e). Thus, the nature and magnitude of the context effects closely resembled those measured during the original free-viewing condition for all the comparisons, suggesting that differences in gaze position did not underly the differences observed during continuous versus discrete and randomized stimulus presentation. This conclusion was further supported by analysis of gaze position during the free viewing experiment described earlier (Figure S4).

Relationship to face selectivity

While neural face selectivity measured with static, isolated images did not figure prominently into the design of the present study, we did confirm that many of the neurons we tested in and around the AF and AM face patches exhibited a preference for faces over other images (Figure 5a; Figure S3 for separate analysis of AF and AM), in accordance with many previous studies^3,62,73–75. Specifically, we found that 88 of the 180 recorded neurons, or 49% of our free-viewing population, had a d’ greater than 0.65 which is a criterion similar to that used by previous studies^73,75.

Figure 5. Interaction between face selectivity and context responses.

a) Selectivity of the combined face patch neurons to faces over scenes and objects. The left histograms depicting the density of d’ values for all neurons. Values of d’ greater than zero represent a preference for larger responses to faces. The right heat maps show normalized firing rate, to the maximum response of a given cell, for all neurons within the populations sorted by their d’ score. The blue bar on the side of the heat map shows neurons that had significant d’ values, and thus would be categorized as face selective. b) and c) show the relationship between face selectivity and our goodness-of-fit (r²) measure for the early and late response periods for the different contexts. b) depicts relationship, and lines of best fit, for the two movie-to-movie (black; early: Pearson’s correlation = 0.1109, p = 0.1383; late: Pearson’s correlation = 0.0698, p = 0.3517) and snippet-to-snippet (blue; early: Pearson’s correlation = 0.3852, p = 0.9e−7; late: Pearson’s correlation = 0.1708, p = 0.0219) goodness-of-fit measure and d’, for the early (left) and late (right) response periods). c) depicts the same comparison for the between context goodness-of-fits and a given neurons d’ score (red; early: Pearson’s correlation = 0.1706, p = 0.022; late: Pearson’s correlation = 0.0738, p = 0.3248). All p-values are Bonferroni corrected for multiple comparisons, 0.05 alpha = 0.0083.

To test whether a cell’s category selectivity had bearing on its observed dependency on temporal context during naturalistic stimulus presentation, we compared each neuron’s d’ score for face selectivity to the magnitude of its same context goodness-of-fit (Figures 5b) and its between context goodness-of-fit values (Figures 5c). The results show that while face selective neurons did not maintain their selectivity during the presentation of movies (Figure 5b, black data), those having a higher level of face selectivity maintained somewhat higher selectivity under during the early phase of the snippets (Figure 5b, blue data). Additionally, the results from the between context goodness-of-fits (Figure 5c) indicate a weak relationship, between the face selectivity d’ value and context r² value in the early and late response periods. Thus, while the majority neurons were disrupted by temporal discontinuity regardless of their category selectivity, the early responses of face selective neurons showed a slightly higher preservation of selectivity during the snippets than other neurons.

Discussion

In the current study, we found that the responses of neurons in anterior face-selective regions of the macaque inferior temporal cortex differ depending on temporal context. Neurons responded differently to the same movie content when it was presented continuously versus when it was divided into brief snippets and shown out of context, in randomized presentation. The disruption in visual responses was particularly prominent in the period shortly following the onset of the snippet. Results were similar during free viewing and controlled fixation. In the following sections, we discuss how visual operations may differ between the continuous, free viewing of a movie and the brief, randomized presentation of its content. We also discuss the role of perceptual history in the establishment of spatial and temporal context during normal vision, as well as the bearing of these results on the activity of face-selective neurons.

Visual selectivity altered by abrupt onset

The most striking effect we observed following the presentation of snippets and static frames extracted from the movie was during the initial phase of the response. During the first 200 ms of activity following the onset of the stimulus, the neural response profile to the content extracted from the movie had virtually no correspondence to the same neurons’ responses to the same content appearing in the movie itself. Is this result to be expected? In some ways, the answer may be yes, since onset responses have long been shown to cause nonspecific transients throughout the visual system^76–78. However, nonspecific transients were not the only factor reshaping neural selectivity in the early snippet responses observed in the present study. For some neurons, the response to snippet content, including the onset, were markedly decreased compared to the same content in the full movie (e.g., response at 150.5 s in Fig. 2a). Likewise, the change in selectivity cannot be attributed to long response latencies. The average response latency for both populations (combined = 118ms; AF = 115 ms; AM = 120 ms), as calculated relative to the static frame onset (see Methods), preceded the plateau observed in the goodness-of-fit time courses (Figures 3d, 4d). The low correspondence of the responses during the snippet onset presentation instead suggests that the visual continuity itself is important.

Visual continuity is an aspect of natural experience that has been absent in most experimental assays of neural selectivity, but which evidently has a significant influence on neural responses. One interpretation of the differences observed between continuous and discrete presentation conditions invokes neural adaptation, or fatigue, which can affect neural processing at all levels of the visual system. Adaptation can have many manifestations, including alterations in stimulus feature tuning, and is usually studied through continued or repetitive presentation of stimuli^79,80,81. In our study, the most straightforward way adaptation might affect responses is to diminish the magnitude of responses during continuous presentation compared to the discrete flashed presentation^82,83. However, this was not the case. Instead, we found the magnitude changes were stronger overall during the visually continuous presentation (see Figure S5a & S5b). This finding, combined with the clear differences in stimulus selectivity in the two conditions, does not lead to a straightforward explanation based on neural fatigue. At the same time, it is likely that some form of adaptation contributes to the difference. To systematically investigate this topic, future studies will need to vary the temporal dynamics in the presentation of movie segments and to employ carefully designed stimuli whose specific adaptational influences on subsequent presentations can be assessed.

Although the neural populations recorded in this study were from two fMRI-determined face patches, our study did not aim specifically to investigate category selectivity, nor its disruption with abrupt presentation. Our use of natural movie stimuli differs from the more conventional use of isolated objects in that the images depicted scenes containing multiple individuals and objects within a complex spatial layout information⁵². Because the movies were not designed to balance visual features or movie content equally, they are not well suited, for example, to systematically compare the effects of temporal continuity for faces versus other categories. Thus, more work is required to understand whether, for example, the observed effects of temporal context are related to the complex scene content or whether, for example, similar disruptions in tuning would be observed for continuous viewing of stimuli presented in isolated form or in the absence of spatial context.

Nonetheless, it bears emphasis that our understanding of high-level feature and category selectivity derives almost exclusively from the presentation of flashed, isolated stimuli. Thus, if the response selectivity observed during flashed presentation does not represent the visual preferences of neurons under more continuous conditions, current notions of feature coding and category representation in the brain may be inaccurate.

Temporal integration of category selective neurons

Beyond the striking changes in the early responses, there were also smaller but significant changes that emerged from the temporal reordering of scene content. These changes were observed in both the free viewing and controlled fixation conditions. As most studies to date have intentionally randomized stimuli that have no particular temporal relationship, relatively little is known about whether feature-selective neurons in the temporal cortex are sensitive to temporal or behavioral context. Some work, however, has suggested that where an animal is looking^9,84,85 or what an animal has recently seen¹³ can strongly affect the responses of IT neurons. Such neurons have been shown to modify their responses based on contextual information, usually in the form of a cue or memory component^9,86–93. Neurons in the superior temporal sulcus are associated with motion stimuli^{12,14,16,94–96}, in particular biological motion, for which temporal continuity may be particularly important. Recently, it has been shown that selective neurons in macaque IT can acquire selectivity for temporal context when isolated stimuli are played in a learned temporal sequence^22,23,97. Increasingly, researchers are using naturalistic stimuli as a complementary approach to conventional stimulus presentation (for a review, see Leopold and Park³¹).

Our results provide some insight into time scales over which neurons integrate visual information by systematically varying the amount of temporal contextual information that was presented, in the form of brief movie snippets lasting 800 ms, 250 ms, 100 ms snippets, and static frames presented for 100 ms, in each case extracted from a longer movie. We found that a subset of neurons within both the AF and AM face patches were significantly affected by the temporal reordering of movie content, even in the late snippet period, suggesting that they integrate some types of visual information at least over time scales of seconds.

In the ventral stream, few previous studies that have similarly addressed temporal integration, albeit with somewhat different methods. Hasson et al³⁴ used fMRI in a naturalistic viewing paradigm somewhat similar to the one we implemented to investigate the length of temporal receptive fields throughout the brain. They found that temporal receptive windows, the length of coherent temporal context necessary to reliably stimulate a portion of the brain, increased systematically as information moved from early visual cortex through the visual system into the prefrontal cortex. In the human temporal cortex, they found 4–8 second spans of coherent temporal information were required create reliable and repeatable visual responses. While there are differences in methodology and experimental design, these findings match our results, likewise suggesting that neurons within the temporal lobe may incorporated greater than one second worth of contextual information in determining their response to a given stimulus^{8,31,48,76–78}.

Conclusions

It is interesting to consider that, during periods of visual free viewing, inferior temporal cortical neurons, including those face patch neurons reported here, enter into a mode of operation marked by sequential target selection, continuity of visual content and attentional processes, and anticipation of events. In this exploratory mode, and in contrast to discrete modes of image presentation, basic visual responses may be modified by neuromodulatory inputs, along with a range of brain areas involved in memory, spatial perception, motor planning, and executive function. As experimental work continues to explore the frontiers of natural visual experience, our portrait of visual brain organization may need to evolve and expand. A central feature of natural experience that is explicitly removed from most experimental approaches is that of temporal continuity. The present study demonstrates that, for populations of well-studied feature selective neurons in the macaque high-level visual cortex, two aspects of temporal continuity, namely visual continuity, the uninterrupted viewing of a stimulus, and temporal context, the orderly progression of visual events in a scene, critically determines how the brain interprets its visual input at a given moment in time.

STAR Methods

RESOURCE AVAILABILITY

Lead contact

Further information and requests for data can be directed to either the lead contact, Brian E Russ (brian.russ@nki.rfmh.org), or the senior author, David A Leopold (leopoldd@mail.nih.gov).

Materials availability

The current study did not generate new unique reagents or materials.

Data and code availability

• The data reported in this manuscript will be shared upon request by the lead contact or senior author.

• Analysis code for the comparison of temporal context effects have been deposited on Github(https://github.com/beruss/Temporal_Context; doi:10.5281/zenodo.7419223).

• The lead contact will provide any additional information required to reanalyze the data reported upon request.

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Five rhesus macaques (macaca mulatta) participated in the current experiments, including 2 females (ages 4–13 years at the time of the study) and 3 males (ages 4–10 years). All procedures were approved by the Animal Care and Use Committee of the National Institute of Mental Health, and followed the guidelines of the United States National Institutes of Health.

METHOD DETAILS

Surgical procedures

In an initial surgery, each animal was implanted with a fiberglass head-post, for immobilizing the head during recording sessions. After behavioral training, described below, microwire bundle electrodes (Microprobes) and adjustable microdrives were implanted into the face patches during a second surgical procedure (see Figure 1a). The electrodes, microdrives, and surgical procedures are described in previous publications^59,60. In three subjects (two female, two male) the electrodes were directed to the anterior fundus face patch (AF): Subject T received bilateral 32 channel electrodes bundles; Subject R received a 64 channel bundle in the right hemisphere; and Subject S and Subject SR received a 64 channel bundle in the left hemisphere. In two other subjects (both male), electrodes were directed to the anterior medial (AM) face patches: Subject M received one 64 channel microwire bundle in the left AM; Subject D was implanted bilaterally with 128 channel microwire bundles, with the left bundle located within AM and the right bundle just adjacent to AM.

Following electrode implantation, the subjects participated in a series of behavioral tasks. During this period, the subjects’ access to liquid was controlled, such that they were motivated to receive liquid reward in exchange for participating in the experimental tasks. In the case that they did not receive sufficient liquid reward in a given session, they were supplemented with liquids in their home cage. Hydration and weights were monitored throughout the experimental period to ensure the animals’ health and well-being.

Visual Stimuli and Behavioral Monitoring

Subjects viewed a range of intact movies, randomized snippets and static frames extracted from the movies, and flashed categorical images. Image presentation and gaze fixation monitoring were under computer control using a combination of custom written QNX⁹ and Psychtoolbox⁹⁸ or a MonkeyLogic¹⁰⁰ code. Neurophysiological signals were recorded using TDT hardware and software. The subjects’ horizontal and vertical eye position was continuously monitored using a 60 Hz Eyelink (SR Research) system, which converted position to analog voltages that were passed to the neurophysiological and behavioral recording software. At the start of each session, the subject’s eye position was centered and calibrated using a nine-position grid with 6° visual angle separation. This calibration procedure was rerun repeatedly throughout each session to ensure good calibration during all experimental trials. Blocks of intact movie presentation, randomized snippet presentation, and categorical image presentation were randomly interleaved (Figure 1c) and took the following form.

Intact Movie Presentation.

The monkeys repeatedly viewed multiple repetitions of a 5 min movie containing a wide range of interactions, mostly between other monkeys^8,52,53. The movies were presented on an LCD screen with a horizontal dimension of 15 dva. While there was no fixation, subjects were required to maintaining their gaze within the movie frame, for which they received a drop of juice reward each 2 – 2.5 s.

Randomized Snippet Presentation.

The content from the movie above was broken down into short movie snippets and static frames, which the monkey viewed in randomized order. The original 5m movie was first divided into three hundred 1 s segments, which were subsequently used to generate snippets of different lengths. The snippet durations of 800, 250, and 100 ms spanned the center frame of each 1 s movie segment. For example, the 800 ms snippet from the 151^st second would present the frames of the movie from 150.10 to 150.90 s, whereas the 100 ms snippet from the same segment would display the frames from 150.45 to 150.55 s. The static frame condition presented the center frame for 100 ms. For each snippet length and the static frame condition, all 300 stimuli were presented repeatedly and in randomized order (Figure 1b). Other aspects of the stimulus were the same as in the intact movie condition, other than the 200 ms blank black screen now inserted between each presentation. As above, the subjects were permitted view the entire snippet content within the brief presentation period and were rewarded with a drop of juice and a 1.0 s break after completing approximately 2 s of trials, depending slightly on the snippet lengths. For an inappropriate fixation break, the animal would forfeit its juice for that period and incur an additional 750 ms timeout followed by repetition of the previous stimulus. For each block of the snippet task, the stimulus duration was held constant until all 300 stimuli were presented.

Categorical Image Presentation.

In these blocks, the monkeys serially viewed brief image presentations drawing from four categories, with 40 images from each category. The image categories were monkey faces, human faces, objects, natural scenes. The stimuli were randomized and presented at 12 dva for 100 ms with a 200 ms inter-stimulus interval. As above, the subjects were permitted to direct their gaze anywhere within the image. If gaze was maintained in the window for seven successive images, the subject received a liquid reward and a 1000 ms inter-trial break. An inappropriate fixation break led to forfeit of reward and an additional 750 timeout. This condition served the dual role of evaluating category selectivity and establishing a “fingerprint” for tracking the identity of single neurons across sessions⁶⁰.

QUANTIFICATION AND STATISTICAL ANALYSIS

Longitudinal Neural Recording

Given the large number of conditions and presentations required in the current study, our recording paradigm hinged on the capacity to accurately record the same neurons across multiple sessions. This longitudinal tracking of individual cells required particular care in spike sorting and rules for determining the continuous isolation of neurons across sessions.

Spike Sorting.

Following each session of recording, channels were cleaned and then submitted to an automatic spike sorting algorithm. The cleaning procedure, run separately on each bank of 16 channels sharing a connector, involved regressing out the first principal component across channels. This procedure was important in our experience to combat certain types of movement and chewing-related artifacts that were shared across channels. Following this cleaning procedure, the automatic sorting program wave_clus^99,101 was applied to identify and sort action potentials on individual channels. Candidate spike waveforms were first marked as any voltages exceeding four standard deviations of the noise. These candidate waveforms were then sorted automatically using PCA space, rendering a sequence of timestamps for between 0 and 4 usable waveforms from each microwire channel.

Tracking Neurons Across Days.

Verifying the isolation of a given neuron across days required a combined consideration of spike waveform, as determined by the automated spike sorting algorithm^99,101, and an independent comparison of the response fingerprint, or selectivity to the flashed images (see Figure S1 for example of isolation and fingerprinting). A cell was determined to be the same as that isolated on a previous day if it (1) was recorded from the same channel, (2) had the same fingerprint signature of response selectivity, and (3) had a similar waveform. Of these three criteria, the third was the only one that was subject to flexibility, since it is well known that small positional changes in the relationship between an electrode and a neuron can change the action potential shape. However, any neuron recorded from a different electrode or having a different selectivity fingerprint across days were considered to be a different neuron. Applying these criteria, we excluded from analysis neurons that were held for fewer than 75% of the sessions, and therefore had insufficient numbers of trials for our statistical analyses. All visually responsive neurons, defined as those giving statistically significant responses in any one of the conditions, were investigated for the effects of temporal continuity, with a separate analysis investigating the specific factor of face selectivity.

Data Analysis

The main data analysis focused on comparing the neural responses to the intact movie with those elicited by the same visual content presented during the randomized 800 ms snippets. Based on pilot analysis, we divided this analysis into two distinct time windows (Figure 1d). The first analysis window analyzed spiking from 100–300 ms following the onset of the snippet, targeting the contribution of the initial response. The second analysis window then focused on the delayed, or extended, response between 300–900 ms.

For each time window, we compared the mean spike count over multiple presentations of the 300 snippets to the corresponding mean spike counts elicited by the corresponding frames in the intact movie. In the main analysis, we compared the two conditions using a linear regression, from which we derived the goodness of fit (r²) to assess the effects of context (see Figure 1e). We also reordered the randomized snippet responses into their original sequence in order to visualize any temporal disruption (see Figure 1b).

For the main analysis, the r² parameter provided insight into the disruption in visual selectivity introduced by randomized snippet presentation. For example, r² values of 1.0 would indicate that a neuron exhibits identical responses to the same visual content across the continuous versus randomized snippet presentation conditions, whereas an r² value of 0.0 would correspond to a complete disruption of the stimulus selectivity, where the neural responses to the specific content were uncorrelated. For intermediate values, the r² indicates the tightness of the stimulus selectivity relationship across conditions, or response variance explained.

In addition to the 800 ms snippets, we also analyzed neural responses to equivalent snippets of 250 and 100 ms, as well as static images of the center frame presented for 100 ms. In those cases, the presentation times were too short to allow for a separate consideration of selectivity before and after the initial stimulus onset, thus a standard response window of 50 to 250 ms was used for evaluation of the onset response.

We calculated a gaze position dependent goodness-of-fit for the early and late period responses of the 800ms snippets. For each epoch, we then evaluated the dispersion of eye positions across presentations based on the Euclidean distance to the mean of the within context movie. This created a set of neural responses that were median split into clustered, or near, and spread-out, or far-away, fixations, as defined by a trials eye positions proximity to the other presentations of the same scene. Randomized snippet trials were then sorted based on the spatial proximity of the mean gaze to that observed during continuous viewing of the same content. This allowed us to divide the snippet responses into epochs exhibiting near and far-away gaze positions relative to that observed during the continuous movie. From this data, we calculated goodness-of-fit of neural responses separately for the snippet trials in which the gaze did or did not approximate that observed during the continuous movie.

We used the same response windows for each clip to calculate a Context Index. Here we define the Context Index as the difference between the within context response and the out-of-context response divided by the sum of both context’s responses. Therefore, if a cell responded strongly to the within context condition and not at all the out-of-context response, the Context Index would be equal to 1, showing a strong preference for the within context condition. However, if the response to both conditions was nearly equal, the Context Index would equal 0, or no preference for either condition.

Neuronal latencies were estimated from their responses to the static frames using the following procedures. We first calculated the average and standard deviation of the firing rate during the 50ms prior to stimulus onset. Then for each of the 300 snippets, we calculated a response onset to their average spike rates. Using a sliding window, we found the moment were spiking activity changed (increased or decreased) by 3std of the baseline firing rate and was maintained for a minimum of 15ms. We then averaged the obtained latencies for all 300 snippets to find the mean latency for a given neuron.

KEY RESOURCES TABLE

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and Algorithms
MATLAB	Mathworks	https://uk.mathworks.com/products/matlab.html
Analysis Code	Custom Matlab pipeline	https://github.com/beruss/Temporal_Context doi:10.5281/zenodo.7419223
Psychtoolbox	Brainard, D.H. (1997)98	http://psychtoolbox.org
Wave_clus	Chaure, F.J., et al (2018)99	https://github.com/csn-le/wave_clus
NIMH MonkeyLogic	Hwang J. et al (2019)100	https://monkeylogic.nimh.nih.gov/index.html

Highlights.

1. Face patch neuronal responses depend on temporal continuity

2. Neural selectivity is disrupted by randomly presenting snippets and static frames

3. Onset responses to dynamic movie snippets are uncorrelated with original movie

4. Post-onset responses are moderately disrupted by temporal context.

Inclusion and Diversity

We support inclusive, diverse, and equitable conduct of research.