Simple explanations before complex theories: Alternative interpretations of Sirotin and Das’ observations

Abstract

We make a few additional points regarding our discussion with Sirotin and Das’ 2009 Nature paper and their 2010 Neuroimage response to our commentary. While we find their data interesting in itself, we remain concerned with how the data are interpreted by the authors. We discuss two categories of methodological issues that limit the conclusions one can draw from their results. (1) The measures of fit quality between the optical and electrical data: kernel shape variation, variance of predicted/measured signals, and R², interact with each other and are confounded by the fact that one condition has a lower signal magnitude and therefore, lower signal-to-noise-ratio (SNR). (2) Hemodynamic responses to distinct events will be incorrectly or inefficiently estimated if the hemodynamic responses overlap across periodic trials that are not jittered and have an inter-trial interval less than 15 seconds. Most importantly, the overlapping responses across trials might cause transient effects that look similar to the anticipatory effects presented by Sirotin and Das. While their study demonstrates a potentially useful way to probe neurovascular coupling, we believe the current results have little practical relevance for interpreting hemodynamic measures of neural activity such as those used in fMRI. We conclude by making several suggestions for future analyses, which might help elucidate the mechanisms behind these observations and lead to a better understanding of how these observations relate to hemodynamic based measures of neural activation.

Their response to our commentary continues a useful discussion (Sirotin and Das in press). Just about everyone, including Sirotin and Das, agree that information remains to be uncovered regarding the relationship between neuronal activity and hemodynamics. It is very unlikely that hemodynamics in visual cortex show a focal, task-related response without some underlying neural system change. We believe that Sirotin and Das simply failed to identify such a change in the local LFP or spiking data. One of the more complex explanations for these observations, as highlighted in the title of Sirotin and Das’ Nature paper, is that local neural activity does not predict the hemodynamic signals observed in the trial-related signal. In this case, localized vasoregulation must be occurring through an unknown mechanism that isn’t preceded by locally altered LFP or spiking signals. This disconnect between neural and hemodynamic activity was also the primary reason for citation in at least 22 of the 52 peer reviewed articles that referenced it, most of which state that these results call the neurophysiological basis of the fMRI BOLD signal into question.^* Even though Sirotin and Das write in their commentary, “We emphatically do not want to conclude that the trial-related signal is ‘not neuronal’ or is ‘independent of neuronal activity’ as imputed by some of the commentaries on our work,” this conclusion is both in the title of their article and, unfortunately, is a core part of the message scientists are taking away from this publication.

We think it is worth the effort to continue looking for other explanations that better fit within our current understanding of neurovascular coupling. In our first commentary, we presented evidence suggesting that this relationship can be identified in the LFP data (Handwerker & Bandettini in press). In the commentary that follows here, we discuss potential weaknesses in task design and analysis. Even if it becomes clear we are studying a previously unobserved mechanism for neuro-vascular coupling, an understanding of the weakness of current methods is vital for improved paradigms and analysis for studying this effect in the future.

Examination of fitting measures

In their response, Sirotin and Das are correct that we should have noted that analyses based on fits of hemodynamic data to the spiking or LFP convolved with a kernel were presented at other frequencies and that supplemental Figure 7 included fitted time courses at other frequencies in their original article (Sirotin and Das 2009). We nevertheless don’t feel that these informatively address the issues of neurovascular coupling in their study. Our rationale, briefly mentioned in the prior commentary and expanded here, is as follows.

Most of Sirotin and Das’ interpretations rest on a basic approach. They take some aspect of the neural activity, like an LFP frequency band, and convolve it with a gamma shaped kernel. They alter the shape of the kernel to find the best possible fit between the kernel-convolved neural data and the hemodynamic data. When the fit has a large R² and the fitted kernel shape is consistent across runs, they say the electrical data predicts the hemodynamic data. A low R² and more variability in kernel shape means neural activity doesn’t predict hemodynamics. This type of logic only holds if other aspects of the signals, like their signal-to-noise ratios (SNR), are consistent across runs. In our original commentary, we noted that the standard deviation of the hemodynamic response during the dark condition was 23% of the standard deviation during the stimuli condition for one presented trial. If one assumes that the data from both conditions have equal amounts of noise, but a 23% smaller signal magnitude, then the SNR decreases by a factor of 4.35. This, in itself, weakens their conclusions.

To further show this effect, we ran a simulation to demonstrate how this would affect the presented statistics. We created a series of hemodynamic responses and fit those responses to kernels using information from the Nature paper and personal correspondence with Sirotin and Das. (26 trials with a 10s inter-trial interval (ITI) followed by 17 20s ITI trials, kernel template had a 2.7s time-to-valley, fitting done with fminsearch in Matlab). We added Gaussian noise to create a “low noise” condition where the fitted time course had a mean R² of 0.5 across 1000 iterations of the simulation. Their stimulated condition often had R²≈0.5. For the “high noise” condition, we scaled the noise by 4.35 to match the SNR decrease for the dark condition. This resulted in a mean R² of 0.05. In their response to our commentary, Sirotin and Das note that the R² fit using the high gamma signal was 0.53 in the stimulated condition and 0.06 in the dark condition (Sirotin and Das in press). While we do not claim the observed differences are purely an artifact of SNR differences, this demonstrates that one cannot interpret R² differences between conditions without also accounting for SNR differences.

SNR also effects kernel shape consistency. If the simulated data use a kernel that is similar to the template, the fitted kernel is usually close to the simulated kernel. To demonstrate how noise affects fits that aren’t close to the template, we ran the same simulation using a kernel (7s time-to-valley) that was far from the template (2.7s time-to-valley), but still in a physiologically reasonable range. While the mean fits across 1000 iterations were fairly good, the standard deviation of the difference between the true response shape and the fitted kernel was 0.07s for the low noise condition and 0.85s for the high noise condition. In addition, in the low noise condition, the fitted gamma kernel and kernel used in the simulated data both had negative magnitudes. With high noise, the fitted magnitudes were positive in 4% of the trials. This simulation has less kernel shape variation in both the low and high noise conditions than observed by Sirotin and Das (For example, Figure 2c of their Nature paper). Still, it does show how lower SNR, as we know exists in the dark condition (even with identical trials and only Gaussian noise), results in nontrivially more variability in kernel fits.

The final measure Sirotin and Das use to demonstrate how neural data can predict hemodynamics in the stimulated, but not dark conditions is the ratio of the variance between the predicted and measured signals. The variance of the measured signal is just the variance of the hemodynamic time series. The variance of the predicted signal is the variance of the neural data convolved with the kernel. This is a function of the variances of the electrical recordings and kernel separately. The fact that R² is lower and kernel shapes vary more in the dark condition, possibly due to different SNR levels, will directly affect these ratios and limit their interpretive utility.

Challenges of using a periodic stimulus pattern

In their Neuroimage commentary, Sirotin and Das were critical of our approach of constructing “an exactly periodic sequence by taking our mean signal and replicating it multiple times.” This is a risk for any periodic design, particularly if the hemodynamic response duration is longer than the inter-trial interval (ITI). If the stimulus pattern is periodic, the responses will reach a steady state that appears to show a consistent response within each trial. In Figure 1 of their Neuroimage commentary, Sirotin & Das claim the periodicity isn’t an issue in their data because response magnitudes for neuronal and hemodynamic data vary across trials. A modest amount of magnitude variation is simply another source of noise in the model. If there were enough magnitude variation between the stimulus and response magnitudes to clearly violate response periodicity it would raise questions whether the presented stimuli are the primary drivers of the neural responses. Since data with longer than 10s ITIs in Sirotin and Das’ 2009 Nature and PNAS papers show responses above baseline for more than 10s, this is a potential area of concern for analyses of trials with short ITIs.

Figure 1.

Data from 3 inter-trial intervals that were presented in Supplemental Figure 8B & C in Sirotin and Das 2009. (A) The presented time courses were placed on top of each other to make it possible to compare the shapes. (B, C, & D) The data were shifted so that the data and kernel had matching trial onsets. They show the 8, 12, and 20s ITIs respectively.

In Figure 1A of this commentary, we overlaid the hemodynamic responses from different ITIs shown in Supplementary Figure 8 B & C in their Nature paper. The hemodynamic responses peak earlier and with a lower magnitude for the shorter trials, which either means that the magnitudes of the responses are different, despite identical visual stimuli, or the lagged hemodynamic responses are interfering with each across the shorter periodic trials. In Figure 1 B-D we overlayed the hemodynamic data on top of their kernel estimates. Even without the 2-3s lag difference between the data and model, the 8s and 12s data have shorter peaks than the model. This is a sign that the final return to baseline of the hemodynamic response is visibly influencing the time course of the subsequent trial. While Sirotin and Das presented these data as an example of the quality of the kernel fits, it's clear that the presented fits do a poor job at modeling these trials.

Sirotin and Das observed that the shape of the hemodynamic response always adjusted to the periodic trial duration. We examined how hemodynamic responses overlapping across trials might create this trial-related effect using the data from Figure 3 in their Nature paper. We assumed that a 20s ITI fully models the hemodynamic response and thus convolved the response in Figure 3C with an 8s ITI periodic trial design. Figure 2A in this commentary shows the resulting hemodynamic response in steady state and compares it to the measured response from the 8s ITI. While there is a 3.8s phase shift between the two responses, similar to what was seen in Figure 1, the 20s ITI response at an 8s ITI steady state is almost identical in shape to the 8s ITI response. While Sirotin and Das look at their data and conclude that the hemodynamic response adjusts its shape to fit the trial duration (i.e. the trial-related hemodynamic signal), we show that a similar effect can occur when the hemodynamic response is longer than the periodic trial duration.

Figure 2.

These figures use the 8s and 20s ISI time series extracted from Figure 3B & C in Sirotin & Das Nature 2009. (A) The 8s ITI response is replicated multiple times and presented with the 20s ITI response convolved with a series of trials every 8s. This 20s ITI response is also shifted to better compare the shape with the 8s ITI signal. The single-trial 20s ITI is also included for comparison. (B) The 20s ITI response is convolved with trials every 8s and then every 20s just like Figure 3F in the Nature paper. The arrow points to a similar valley that Sirotin and Das considered part of the anticipatory response. (C) The 20s ITI response is convolved with trials every 20s and then every 8s just like Figure 3F in the Nature paper.

We then looked at the core evidence of the anticipatory response in Figure 3F-G of the Nature paper. The presented evidence is that the trial-related signal takes an extra trial to transition to a new pattern. Since the trial-related signal continues even after the final trial, Sirotin and Das reason that the anticipatory response is connected to the trial duration. In Figure 2B, we modeled the same 20s ITI response from that Figure and transitioned from an 8s to 20s ITI. We get a similar response pattern to Figure 3F of the Nature paper with the first 20s ITI containing a signal drop and increase as if it were still in an 8s ITI. Figure 2C shows a transition from a 20s to 8s ITI and it also takes an extra trial to reach the new steady state. Based on these observations, we do not think the presented evidence strongly supports the existence of an anticipatory hemodynamic signal since such a response is impossible to disambiguate from hemodynamic effects resulting from transitions between two stimulus timing intervals in the same time course.

Suggestions for future analyses

While we have been critical in these commentaries, we believe that Sirton and Das have an important new stimulus paradigm. They present a task that shows a measurable hemodynamic response while our common neural measures might be below an easily detectable threshold. By assuming that a hemodynamic response is a marker of some type of neural activity, this is a great paradigm to improve neural measurements and analyses to best see that activity and better probe the subtle complexities of neurovascular coupling. While, here we focus on one high profile paper that has suggested an inconsistent neuronal – BOLD relationship, we would like the reader to take away a broader perspective that every reported variation in BOLD dynamics relative to neuronal responses does not throw into question all the studies that have been or will be performed using BOLD, but rather has the potential to open up new avenues into exploration of unique neurovascular dynamics that might vary with task, region, subject, or population.

So, as last suggestions, we would offer the following: 1. That researchers stay as close as possible to the raw data as possible – not basing fundamental conclusions on the accuracy of a fit to a hemodynamic kernel. For example one could compare the ratio of variances of the neural and hemodynamic recordings across conditions, like we did in our first Neuroimage commentary rather than working off a fit to an imperfect model. 2. That researchers consider SNR of each measure so that a null relationship is not just a function of a noisy measurement of one aspect of the data (in this case the two conditions different signal magnitudes). For periodic data, a rough SNR estimate is as simple as dividing the mean trial signal by the residuals for each condition. 3. That task design is constructed such that conclusions are not drawn from artificial transients or shape alternations in the signal. If the trial-related hemodynamic observations remain for transitions between 20 and 30s ITIs, it would greatly strengthen the evidence for their existence.

In conclusion, we believe that more careful analyses and task designs would help confirm whether the differences in neurovascular coupling across conditions reported by Sirotin and Das can be reliably produced and are representative of a new vaso-regulation mechanism. Future studies will nevertheless certainly determine which observations and measurements of Sirotin and Das have practical impact for the interpretation of brain activation maps using hemodynamic measures.