Skip to main content
PLOS ONE logoLink to PLOS ONE
. 2024 May 6;19(5):e0302660. doi: 10.1371/journal.pone.0302660

Lessons learned from an fMRI-guided rTMS study on performance in a numerical Stroop task

Lysianne Beynel 1,*,#, Hannah Gura 1,2,#, Zeynab Rezaee 1, Ekaete C Ekpo 1, Zhi-De Deng 1, Janet O Joseph 1,3, Paul Taylor 4, Bruce Luber 1, Sarah H Lisanby 1
Editor: Desmond J Oathes5
PMCID: PMC11073721  PMID: 38709724

Abstract

The Stroop task is a well-established tool to investigate the influence of competing visual categories on decision making. Neuroimaging as well as rTMS studies have demonstrated the involvement of parietal structures, particularly the intraparietal sulcus (IPS), in this task. Given its reliability, the numerical Stroop task was used to compare the effects of different TMS targeting approaches by Sack and colleagues (Sack AT 2009), who elegantly demonstrated the superiority of individualized fMRI targeting. We performed the present study to test whether fMRI-guided rTMS effects on numerical Stroop task performance could still be observed while using more advanced techniques that have emerged in the last decade (e.g., electrical sham, robotic coil holder system, etc.). To do so we used a traditional reaction time analysis and we performed, post-hoc, a more advanced comprehensive drift diffusion modeling approach. Fifteen participants performed the numerical Stroop task while active or sham 10 Hz rTMS was applied over the region of the right intraparietal sulcus (IPS) showing the strongest functional activation in the Incongruent > Congruent contrast. This target was determined based on individualized fMRI data collected during a separate session. Contrary to our assumption, the classical reaction time analysis did not show any superiority of active rTMS over sham, probably due to confounds such as potential cumulative rTMS effects, and the effect of practice. However, the modeling approach revealed a robust effect of rTMS on the drift rate variable, suggesting differential processing of congruent and incongruent properties in perceptual decision-making, and more generally, illustrating that more advanced computational analysis of performance can elucidate the effects of rTMS on the brain where simpler methods may not.

Introduction

The Stroop task is used to investigate the effects of competing visual categories on decision making. The earliest version of the Stroop task [1] employed words printed in various colors, and participants were asked to name the colors of the ink that the words were printed in as quickly as possible. When the ink was the same color as the word (the congruent case), naming responses were faster, while when the ink color was different (the incongruent case), responses were slower. These facilitatory and interference effects in performance are quite reliable in their general replicability across healthy adults and in their stability in effect size (after a short initial learning period) within individuals. This reliability has made the Stroop task a useful tool for investigating the cognitive processes involved with its effects. For example, automaticity in stimulus recognition, processing speed, and selective attention, as well as in neuropsychological testing in patients with deficits in such processing. The reliability of this task has also enabled neuroimaging investigations into the neural structures underlying Stroop effects, which have implicated the roles of the parietal cortex and prefrontal regions such as the anterior cingulate cortex [2, 3]. Studies employing online repetitive transcranial magnetic stimulation (rTMS) during task performance have sought to establish causal relationships between these neural structures thought to subserve Stroop effects.

In one such study, a counting Stroop task was used, in which subjects were asked to count the number of words shown on a screen, comparing screens using number words (the interference condition) and screens using neutral words [4]. Short (4 pulse) trains of rTMS given at 10 Hz starting 200 ms after stimulus onset to the anterior cingulate cortex caused the eradication of the slowing effect of the interfering words compared to the neutral words, while significant slowing was shown with stimulation to an active control site in posterior midline cortex. Another set of studies used a numerical Stroop task [5, 6] in which two digits were concurrently shown, and the subject was to identify the larger number in magnitude (i.e., 4>2). To induce a Stroop effect, the physical size of each number was manipulated. In the neutral condition, both numbers had the same font size. However, in the congruent condition, the number larger in magnitude had the larger font size; and in the incongruent condition the number smaller in magnitude was in larger font. Both studies (performed by the same group), applied a short rTMS train (3 pulses) at 10 Hz, beginning 220 ms after stimulus onset over the right intraparietal sulcus (IPS). Both resulted in significant reduction of the Stroop facilitation/interference effects on reaction time compared to sham stimulation. Overall, there was a consistency across these three studies using rTMS to modify performance in Stroop tasks: in their application of online rTMS, their effects on Stroop performance, and in the interpretation of their results. In terms of the latter, in Cohen Kadosh et al. [6] it was suggested that rTMS reduced the Stroop interference effect by disrupting right parietal automated processing associated with the competing but irrelevant stimulus dimension which would normally slow the decision process involved in completing the task. This suggestion could also apply to Sack et al. [5] as well as Hayward et al. [4] all of which could be classified under a general mechanism of performance enhancement caused by “addition by subtraction”, where a normally competing but interfering process is disrupted by rTMS [7].

The primary goal of the Sack et al. [5] study was to provide a direct comparison of four different rTMS targeting approaches used at that time: 1) scalp-based measurement, 2) anatomical MRI locations, 3) average group coordinates from fMRI activations using the task being studied, and 4) individualized targeting based on task-related fMRI. The interindividual reliability of the numerical Stroop task they successfully used in Cohen Kadosh et al. [6] made it a good candidate task for the targeting comparison. Sack et al. [5] concluded that the most effective approach was individualized targeting based on the task-related fMRI. This was an important result for the brain stimulation field in general, as one of the main moderators of rTMS efficacy is the targeting approach [8]. More than a decade after this elegant demonstration of the importance of individualized fMRI targeting, experimental TMS methodologies have advanced, with for example the development of robotic coil holders that allow for more precise and stable positioning of the coil relative to the stimulation target, and an electrical sham technique that mimics the TMS-induced sensations, and thus potentially better blinds subjects to experimental conditions. In the present study, we tested whether the addition of these techniques combined with individual fMRI-based targeting approach endorsed by Sack and colleagues could improve rTMS effect on numerical Stroop performance, with the expectation based on Cohen Kadosh et al. [6] and Sack et al. [5] that rTMS applied over the right IPS would decrease the interference of incongruent stimuli and thus lessen of the reaction time in that condition. Additionally, contrary to the prior study from Sack et al., participants completed the task multiple times preceding rTMS application (one block of trials before the MRI session, 4 blocks during MRI, and one block in the TMS session prior to TMS), to stabilize performance and minimize variability due to practice effects. This approach enabled us to probe the influence of task practice on behavioral performance and fMRI activations.

Finally, we decided post-hoc to perform behavioral modeling approach using a drift diffusion model to assess rTMS effects within a decision-making context. This model, by considering the distribution of reaction times rather than just central tendencies, offers more insight in decision making processes. By elucidating the dynamics of evidence accumulation over time, and by distinguishing between decision and non-decision processes, the model provides a more nuanced understanding of ongoing cognitive mechanisms. Drift diffusion models have demonstrated effectiveness in modeling performance data derived from simple tasks with short reaction times (< 1 s), as in the present case. In addition, they have been linked to neural behavior on a number of scales, from single cells to large populations (e.g., using EEG and fMRI) [9]. The drift diffusion models have also been successfully used to investigate the neuronal substrates for numerical representation using TMS [10]. Therefore, it was expected that this modeling approach would reveal more subtle rTMS effects, as recommended by Hartwigsen et al. [11].

Material and methods

Participants

Nineteen healthy volunteers (14 female, 5 male) with a mean age of 39 ± 14 (SD) years participated in this 2-day study pre-registered on CT.gov (NCT00024635). All participants were right-handed and identified as White (n = 10), Black (n = 4), Asian (n = 3), or Hispanic (n = 2). Participants were recruited from the Washington, DC, metropolitan area via flyers, listserv emails, advertisements, and the NIH Office of Patient Recruitment. All participants gave written informed consent approved by the NIMH Institutional Review Board (#18-M-0015) to be screened with psychiatric, physical, and neurological examinations, urine drug screens, and pregnancy tests for women of childbearing capacity. Volunteers were excluded if they had a history of current or past Axis I psychiatric disorders (including substance abuse/dependence) as determined by the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-NP), a history of neurological disease, or seizure risk factors. Participants were compensated for their participation.

Experimental design and numerical Stroop task

Participants came in for two experimental sessions. The first visit included consenting, an initial numerical Stroop task practice session, and structural and functional neuroimaging in combination with the Stroop task. The second visit included an additional Stroop task practice followed by active and sham rTMS, with the order counterbalanced between participants (Fig 1). The numerical Stroop task was coded using the PsychoPy Builder interface [12]. In this task, two single-digit numbers were presented on the screen (between 1 and 8 numbers apart) with a center-to-center horizontal visual angle of 7.7 degrees. The numbers varied in physical size (vertical visual angle of 1.4 degrees for the small numbers or 2.9 for the larger ones). Three conditions were tested: Congruent, in which the numerically larger digit was also larger in size; Incongruent, in which the numerically smaller digit was larger in size; Neutral, in which both digits presented were the same small size. Participants were asked to indicate with a left or right button press as quickly and accurately as possible the number with the numerically larger magnitude, irrespective of size. Reaction time and accuracy were collected.

Fig 1. Consort diagram and experimental design.

Fig 1

Visit 1

After consenting to the study, participants completed a first training block of the numerical Stroop task with 48 trials (24 congruent, 24 incongruent; no neutral trials were included as they were not used during the fMRI acquisition). Feedback was provided after each trial with the word “correct” or “oops” on the screen to ensure participants understood the task, and overall accuracy was provided on the screen at the end of the task. Participants were then scanned using a 3T gradient echo scanner (General Electrics) equipped with a 32-channel head coil. A structural T1-weighted MP-RAGE image was first acquired (FOV = 25.6 cm2, voxel dimension = 1 mm isotropic, TE = Min Full echo, TI =1100 ms, bandwidth = 25 Hz/Pixel, flip angel = 7 degrees). Four EPI sequences were then acquired with an oblique orientation defined with the AC-PC axis (FOV = 72*72*52, voxel dimension = 3 mm isotropic, TE = 30ms, TR = 3000 ms, flip angle = 70 degrees) during which participants performed four more blocks of the Stroop task. Feedback (accuracy and average reaction time) was provided at the end of each block. Stimuli were back-projected onto a screen located at the foot of the MRI bed using an LCD projector. Subjects viewed the screen via a mirror system located in the head coil and the start of each run was electronically synchronized with the MRI acquisition computer. BOLD signal analyses were then performed (see MRI targeting section) to identify a target for TMS.

MRI targeting

AFNI (version 22.3.05) [13] was used to process MRI data. First, @SSwarper was used for skull stripping and estimating nonlinear alignment (via 3dQwarp; [14]) of the anatomical image from native space to Montreal Neurological Institute (MNI) stereotaxic space [15]. Then afni_proc.py was used to setup a full pipeline for the fMRI analysis of each participant, including the automatic generation of a quality control HTML for evaluating the data and processing steps [16]. The full afni_proc.py command is provided in GITHUB (https://github.com/afni/apaper_rtms_fmri_stroop) but we briefly describe the steps here. The first 2 TRs (6 seconds) to allow stabilization of the magnetic field were discarded from the analysis. All images were corrected for slice acquisition timing, motion corrected by registration to the minimum outlier volume, and spatially smoothed with a 4mm full-width-half-maximum smoothing kernel. EPI-anatomical alignment was performed using the lpc+ZZ cost function [17] with local EPI unifizing for additional stability, and these datasets were checked for left-right consistency [18]. To reduce effects of participant motion, volumes with large motion (Enorm > 0.3 mm between successive time points) were censored.

A duration modulation regression basis was used, and separate events were modeled for Congruent and Incongruent trials. The onset for each was defined by the beginning of each event, and duration for each was defined by the extent of the participant’s reaction times. Incorrect trials in which participants did not answer correctly (for either Congruent or Incongruent stimuli) were modeled identically but as a separate stimulus class, and missed trials were not modeled. The contrast between Incongruent > Congruent trial was generated. After regression modeling, several steps were then performed for QC evaluation using the APQC HTML, such as: checking the alignment between the anatomical, EPI and the template image; checking that the stimuli were properly assigned between each stimulus class; checking how many data points were censored because of motion. One participant was excluded via this QC evaluation because of motion (with 27.5% of the data censored due to motion); looking at stimulus statistical maps for validity. The final Incongruent > Congruent statistical maps and the intraparietal sulcus (IPS) mask from Neurosynth (https://neurosynth.org/) were then mapped back into subject original space; and along with the original T1 these three results were loaded into the neuronavigation system (BrainSight, Rogue Research, Canada); and the area of the statistical peak within the right IPS mask for each participant was chosen as the rTMS target.

Visit 2

During visit 2, participants conducted a training block followed by four blocks of the numerical Stroop task in conjunction with active or sham rTMS. Active and sham stimulation were applied on the same day in random order, counterbalanced between participants (e.g., 2 blocks of active followed by 2 blocks of sham or vice-versa). Training and stimulation blocks contained 72 trials (24 congruent, 24 incongruent, and 24 neutral trials, replicating Sack’s number of stimuli), with feedback (accuracy and average reaction time) provided at the end of the block.

Stimulation was applied at 10-Hz over the individualized IPS target at 60% maximum stimulator output (MSO) as per Sack et al [5]. During each trial, a triplet of pulses was applied at 220, 320, and 420 ms after the onset of numbers presentation. Sham stimulation was applied using the same coil in placebo mode, which produced clicking sounds and somatosensory sensations via electrical stimulation with scalp electrodes similar to the active mode, but without a significant electric-field (E-field) induced in the brain [19]. Each subject was informed that 2 types of stimulation would be applied, and that we needed to define the intensity. For active rTMS, resting motor threshold (rMT) was assessed using motor threshold assessment software tool MTAT [20] over the right motor cortex and used as a potential covariate. For sham stimulation, subjects were asked if they felt each pulse. If not, the sham intensity was increased. If the pulse was felt, participants were asked if the sensation was tolerable. The lowest tolerable intensity was chosen for stimulation. An A/P-B65 coil powered by a MagVenture MagPro X100 stimulator (MagVenture, Farum Denmark) was used, and coil placement was guided by a computerized frameless stereotaxic system (Brainsight, Rogue Research, Montreal Canada). Coil position and orientation was maintained precisely over the target with the Axilum Robotics TMS-Cobot (Axilum Robotics, Schiltigheim, France).

Statistical analysis

The analysis is divided in three sections: the first part relies on classical analysis of reaction time and accuracy; the second section uses a drift diffusion modeling approach to test the rTMS effect with a more subtle decision-making context. The final section is exploratory and seeks to identify potential predictors of the rTMS effect. Analyses of accuracy and reaction time of correct trials were performed with JASP (Version 0.17.2, Amsterdam Netherlands). In the classical analysis, three analyses were performed: the first linear mixed model investigated the impact of task practice on behavioral performance using Congruency (congruent, incongruent) and Timepoint (1 block before MRI, 4 separate MRI Blocks, 1 block before TMS) as fixed effects variables, and Participants as a random effect variable. The second analysis used a linear mixed model to investigate the rTMS effect on task performance using participants as a random effect and Stimulation (active, sham), Congruency (congruent, incongruent, neutral) and Order of stimulation (sham-first, active-first) as fixed effects. Information about the pre-processing of behavioral data is provided in the results section. The third analysis was conducted to replicate Sack et al.’s finding on the size congruency effect (SCE), the difference between reaction time for Incongruent and Congruent trials, to test whether active rTMS would decrease the SCE compared to sham stimulation. We also investigated whether rTMS impacted the facilitatory and inhibitory component of the SCE. For each of these analyses, we performed an ANOVA with Stimulation (active, sham) as a within-subject factor, and Order of stimulation (active first, or sham first) as a between-subject factor.

While changes in reaction time and/or accuracy are the most commonly used variables to investigate rTMS effects on a behavioral task, it has been demonstrated that they only provide a limited statistical sensitivity, which might not pick up slight changes in response strategies, for instance conflicting response tendencies in Stroop tasks (see [11] for a review). To overcome this limitation, the second part of the analysis uses a drift diffusion model [21], in which the reaction time distributions for correct and incorrect responses were fit into a model comprised of the sensitivity to the relevant stimulus (drift rate), the decision threshold (boundary parameter, i.e., the amount of information needed to trigger the button press), and the non-decision time (i.e., the duration of information processing before the decision process and the time taken to execute the motor command). This approach allows assessment of rTMS effects within a more subtle decision-making context. Drift diffusion models were fit to trial-by-trial measures of reaction times and response type. The drift rate, boundary separation, and nondecision time parameters were estimated separately across congruency and TMS conditions. 5% of all trials were assumed to be outliers and modeled under a different process.

Finally, a linear regression model was performed to explore the impact of participant’s age, stimulation intensity, distance to scalp, fMRI activation at the stimulated target location, distance to Sack et al.’s group fMRI target, and number of days between the MRI and the TMS session, as potential predictors of rTMS effect, and to test the potential impact of numerical distance between stimuli, a factor that is known to strongly impact reaction time.

Results

Four subjects were withdrawn during the protocol due to the time commitment (n = 2), movement during the MRI (n = 1), and hair thickness that was incompatible with accurate electrode placement (n = 1). Data from one additional subject was excluded due to excessively long reaction times, which were greater than 2 standard deviations above the group mean for four out of six conditions. Therefore, analyses were performed on 14 participants. All data below are presented as mean ± standard error.

Classical analysis

Practice effect

To investigate the impact of practice on task performance, we applied a linear mixed effect model to analyze correct reaction time (Fig 2A). Before performing the analyses, outliers and missed trials were removed from the data. First, missed trials, in which participants did not respond, were removed from both for reaction time and accuracy data (4.2% of the trials). Next, for the analysis of reaction time only, incorrect trials, in which participants answered incorrectly were removed (1.9%); then trials with reaction times larger than 2.5 standard deviations above the mean were removed (2.3%). We did not run any model on the accuracy data, as there was not enough variability in the data because accuracy was near ceiling at all timepoint (overall group accuracy was 98.1 ± 0.3%, see Fig 2B for accuracy values and change across sessions). Results below focus on correct reaction time.

Fig 2. Behavioral performance in the scanner: A) reaction times and B) accuracy in the six training blocks, for congruent trials in blue and incongruent trials in yellow.

Fig 2

We assessed the effects of Congruency and Timepoint on correct reaction times during practice using a linear mixed model with Congruency and Timepoint as fixed effect variables, and Participant as random effect. The linear mixed model did not show any interaction between Congruency and Timepoint, however we found the expected main effect of Congruency (F(1,12.58) = 94.698, p < .001) with congruent trials yielding faster reaction times (533 ± 16 ms) than incongruent trials (607 ms ± 14 ms). A main effect of Timepoint (F(5, 14.68) = 23.830, p < 0.001) was also observed; and post-hoc contrasts (Bonferroni corrected) revealed that while performance in the pre-MRI practice block was not significantly different from the first MRI block (p > 0.05); reaction times at each of the timepoints thereafter were faster than the initial reaction time collected before the MRI (p < 0.05 for all pair-wise comparisons). Reaction times at all timepoints after MRI block 1 were also significantly faster than for MRI block 1 (p < 0.05 for all pair-wise comparisons). No significant pairwise differences in RT between other blocks were observed (Fig 2A).

In addition, as our group of participants was older (39 years old on average) than the participants in Sack’s study (24 years old on average), we investigated whether there was any correlation between age and reaction times in Congruent and Incongruent trials collapsed across the six practice blocks. Results did not reveal any significant correlations between them (Congruent trials: r = 0.26; p = 0.40; Incongruent trials: r = 0.45, p = 0.13), suggesting that age did not impact performance.

rTMS effect

The same cleaning approach was performed on the rTMS data except that an additional first step was added—the trials in which the TMS coil was far away from target (>= 3mm, often due to participant movement) were removed from the data (2.1% of trials). Then, trials in which participants did not answer were excluded (1.3% of remaining trials). Finally, for reaction time analysis, trials in which participants did not answer correctly (12%) and trials with reaction times larger than 2.5 standard deviations above the mean (3% of remaining trials) were removed. The overall group accuracy was 88%, however since no main effects or interaction were found between the three variables, the results below focus on reaction times only. A linear mixed model was performed with participants as a random effect, and Stimulation, Congruency and Order as fixed effects. The model revealed a main effect of Congruency (F(2, 25.30) = 47.56, p < 0.001), with incongruent trials slowing down participants (492 ± 15 ms) compared to congruent (439 ± 12 ms) and neutral trials (457 ms ± 12 ms) (p < 0.01 for each pair-wise comparison). The model also revealed a significant interaction with Stimulation, Congruency and Order (F(2, 44.43) = 3.68; p = 0.03). However, decomposition of this interaction with post-hoc Bonferroni-corrected contrasts did not reveal any significant difference between active and sham rTMS.

Size congruency effect (SCE)

To replicate the findings of Sack et al. [5], we calculated the SCE by measuring the reaction time (RT) difference between congruent and incongruent trials [SCE = RT (incongruent)—RT (congruent)]. We excluded data from an additional participant whose size congruency values exceeded 2.5 standard deviations above the group mean, analyses were therefore performed on 13 participants. Contrary to expectation, the repeated measures ANOVA did not reveal a main effect of Stimulation (F(1,11) < 1, p = 0.40: Active = 51 ± 16 ms; Sham = 46 ± 9 ms), Order (F(1,11) < 1, p = 0.58) or interaction between Order and Stimulation (F(1,11) < 1, p = 0.58), suggesting that contrary to Sack et al., rTMS in our study did not impact the size congruency effect (Fig 3).

Fig 3. rTMS effect of size congruency effect and its components.

Fig 3

A) Size Congruency effect, B) Facilitatory component and C) Interference component showing a significant interaction between Stimulation and Order of stimulation.

Following the methodology employed by Sack et al., we also calculated the facilitatory component as [RT (neutral)—RT (congruent)] and the interference component computed as [RT (incongruent)—RT (neutral)]. For the facilitatory component, neither the interaction between Stimulation and Order, nor the main effects reached statistical significance. However, for the interference component, a significant interaction between Stimulation and Order was observed (F(1, 11) = 9.39; p = 0.011, η2 = 0.21). However, post hoc analysis did not reveal any significant differences among pairwise comparisons (Active-first group: Active = 47 ± 26 ms, Sham = 21 ± 16 ms; Sham-first group: Active = 24 ± 17 ms, Sham = 36 ± 17 ms). The absence of broader significant findings may be attributable to an insufficient sample size, which could diminish the power of our analysis.

Computational analysis of rTMS effects with a drift diffusion model

The drift diffusion model revealed a significant interaction between Congruency and Stimulation (F(2,70)=10.9, p<.001, Fig 4). Post hoc comparisons revealed that while no difference were observed for incongruent (t(14)=0.90, p=.39) and neutral trials (t(14)=0.89, p=.39), the drift rate was significantly higher with active rTMS compared to sham stimulation for the congruent trials (t(14)=6.5, p<.001). While we would have expected rTMS to the right IPS to increase the drift rate for both congruent and incongruent trials, given its role in automatic spatial magnitude processing; the current finding suggests that within the decision process, at least in a well-practiced state, the mechanism underlying the integration of congruent properties differs and can be more easily distinguished from that underlying the integration of incongruent properties.

Fig 4. Mean drift rate for each congruency and each stimulation condition.

Fig 4

Exploratory analyses

In the first analysis, we performed a linear regression analysis with a backward selection method to test the impact of several covariates that could impact rTMS effect (participant’s age, functional activations at the stimulated target, scalp-to-cortex distance, distance to Sack’s targets, stimulation intensity expressed as a percentage of the resting motor threshold, and number of days between MRI acquisition and TMS session) on rTMS effect for calculated as follow:

rTMSeffect%=ReactionTimewithActiverTMSReactionTimewithShamrTMSReactionTimewithShamrTMS×100

This method allowed to iteratively remove non-significant variables from the model, and thus to identify the most relevant predictors of rTMS effect while controlling for potential confounding variables. The final model showed that only stimulation intensity predicted the rTMS effect (F(1,13) = 13.09, p = 0.004, adjusted R2 = 48.2%). This suggests that participants who received the strongest stimulation intensity where the ones benefitting the most from rTMS (showing the fastest reaction times with active rTMS). We note that on average, in our sample, an intensity of 60% MSO led to a strong variability in stimulation intensity, which ranged from 94% to 188% rMT (mean = 128.9 ± 5.7% rMT, see S1 Table for individual participants data). None of the other variables: participant’s age, scalp-to-cortex distance, distance to Sack’s target, and number of days between fMRI acquisition and TMS session, or fMRI activation at the stimulated target predicted the rTMS effect. Contrary to our assumption analysis of the fMRI activation at the group level, did not show the expected right IPS activation. Instead, the analyses of BOLD signal revealed a cluster of activation (t > 2.10, with 23 voxels) in the left IPS and in subcortical structures such as the bilateral corpus callosum, and the left cingulum (see Fig 5A and S2 Table for the list of clusters and their overlap with the Glasser atlas, generated by the AFNI whereami function).

Fig 5. fMRI results, TMS targets, and correlations.

Fig 5

A) Group fMRI activation in the Incongruent > Congruent contrast. Colors show the effect estimate value. Each voxel is thresholded by its t-stat values and the transparent thresholding was applied [22] (Threshold: t = 2.11, bi-sided; p < .05) on a coronal and an axial view and on a 3D rendering (left is subject left). B) TMS target for each individual on an MNI brain, and C) correlation between stimulation intensity and rTMS effect.

To evaluate whether the lack of rTMS effect may have been due to targeting issues resulting from a change in fMRI activation across blocks, which may have resulted from practice, learning, and/or fatigue, we investigated how activations differed between the first and last pairs of blocks. Our results showed that while activation in the bilateral IPS was strong during the first two blocks, IPS activity vanished during the last two blocks (Fig 6). Given that these changes were observed while accuracy remained quite high and constant across all blocks, and while reaction time continued to improve (Fig 2A). We suspect this change is due to a learning effect and hypothesize two potential mechanisms: 1) the IPS was highly involved in the earlier learning stages of the task and became less necessary with practice, with possible activity shifts to other regions; 2) the number of IPS neurons necessary for task performance decreased with the optimization caused by practice, with subsequent diminishment of signal. If correct, these hypotheses indicate the need for a better understanding of the local and network changes associated with task practice and learning effects to estimate the best targeting parameters. In the present case we potentially would have seen stronger rTMS effects if we only acquired two blocks of fMRI, and this may have served as a better target for rTMS if the latter of the two reasons stated above for the activation decrease was true.

Fig 6. fMRI activation in the Incongruent > Congruent contrast in the first pair and last pairs of fMRI acquisition.

Fig 6

Colors show the effect estimate value for: a) Last blocks > First blocks, b) the First blocks only, and c) the Last blocks only. Each voxel is thresholded by its t-stat values and the transparent thresholding was applied [22] (Threshold: t = 2.16, bi-sided; p < .05) on a sagittal view.

The second analysis investigated the effect of numerical distance between stimuli. Indeed, it has been demonstrated in the literature that performance in the Stroop task is inversely correlated with numerical distance; more precisely, if the distance between two numbers is large (e.g. between 1 and 9: distance of 8) reaction times are faster than if the distance is small (e.g., between 2 and 3: distance of 1) [23]. Since our study was not designed to test this effect, the distribution of the trials in each distance prevented us from testing the effect of each distance on performance. Therefore, we decided to categorize trials into small distance (from 1-3 digits) and large distance (from 4-8 digits) groups and performed an ANOVA with Numerical Distance (Small and Large), Stimulation (Active and Sham). The analysis revealed the expected main effect of Numerical Distance F(1,13) = 51.83, p < 0.001) with faster reaction times for large (448 ± 12 ms) compared to small distances (471 ± 13 ms). The main effect of Simulation Type was not significant (F(1,13) = 3.46; p = 0.09), and interestingly no interaction was found between Stimulation Type and Numerical Distance (F(1,13) = 1.14; p = 0.31), suggesting that while Numerical Distance had a strong impact on reaction time, the effects of rTMS were not impacted by it.

Discussion

Cohen Kadosh et al. [6] and Sack et al. [5] reported that using short bursts of 10 Hz rTMS to right IPS during the performance of a numerical Stroop task caused a significant decrease in the difference between incongruent and congruent RTs. We were not able to replicate these findings with a larger sample size, seeing no main effect of rTMS on RT or on the SCE. As discussed below, this failure to replicate may have to do with methodological differences between their studies and ours. However, we did observe a significant effect of right parietal rTMS on task performance, with thirteen of fourteen subjects showing an increase with active stimulation to the drift rate variable in a drift diffusion model of the RT data.

Use of more sophisticated approaches to modeling performance data stands a much greater chance of revealing effects of rTMS on behavior [11]. In the present case, rather than just looking at mean RT effects, use of a drift diffusion model incorporates the full RT distributions of both correct and incorrect responses, accommodating speed/accuracy trade-offs and bias effects, and by assuming that information accumulates at a constant rate, is able to estimate that rate [9]. This drift rate can be thought of as an estimate of the signal-to-noise ratio in the perceptual decision process.

Drift diffusion modeling has previously been used successfully with rTMS. For example, 1 Hz rTMS to dorsolateral prefrontal cortex, which might be expected to disrupt ongoing decision processing occurring there, was reported to decrease the drift rate found modeling data from the performance of an object discrimination task [24]. In the context of numerical Stroop task performance, rTMS to right IPL might be expected to increase the drift rate for both congruent and incongruent trials. This is based on the role of right parietal cortex in automatic spatial magnitude processing (in the case of the numerical Stroop task, a comparison of physical sizes) that Cohen-Kadosh et al. [6] posited to explain their TMS findings. Right IPL rTMS should disrupt the automatic processing of number font size, leading to the subtracting out of its irrelevant contribution to the ongoing decision process. The decision process, having one less channel of useless information (i.e., noise) to integrate, is thus made more efficient, which should be reflected in an increased drift rate. In this interpretation for the cause of a change in drift rate with rTMS, the congruency condition should not matter. However, here only the congruent trials, but not the incongruent, showed this effect. In the context of numerical Stroop task performance, rTMS to right IPL might be expected to increase the drift rate for both congruent and incongruent trials. This is based on the role of right parietal cortex in automatic spatial magnitude processing (in the case of the numerical Stroop task, a comparison of physical sizes) that Cohen-Kadosh et al. [6] posited to explain their TMS findings. Right IPL rTMS should disrupt the automatic processing of number font size, leading to the subtracting out of its irrelevant contribution to the ongoing decision process. The decision process, having one less channel of useless information (i.e., noise) to integrate, is thus made more efficient, which should be reflected in an increased drift rate. In this interpretation for the cause of a change in drift rate with rTMS, the congruency condition should not matter. However, here only the congruent trials, but not the incongruent, showed this effect. A primary methodological difference between the Sack et al. [5] and Cohen-Kadosh et al. [6] studies and the present one is that subjects were relatively unpracticed on the task in the former but well-practiced in the latter, and the main difference in results was that an rTMS effect was seen directly in the RT data of the former studies but not the latter. In terms of the present drift rate finding, these differences suggest that within the decision process, at least in a well-practiced state, the mechanism underlying the integration of congruent properties differs and can be more easily distinguished from that underlying the integration of incongruent properties. With practice, the effect of right parietal rTMS on automatic processing seen in the earlier studies may change such that it only occurs in the congruent state, while the decision process in the incongruent condition may rely more heavily on other brain regions, such as anterior cingulate cortex. A further study comparing the effects of rTMS on subjects that are task-naïve vs. practiced would be needed to verify this. Moreover, the diffusion model results and speculations based on them result need to be taken cautiously as the analysis was done post hoc and the study was not designed optimally to evaluate the drift. For example, it only included 48 trials per condition, while it has been demonstrated that about hundred trials are needed to estimate this parameter robustly [25]. In addition, other models might be tested: for example, the drift diffusion model used here fits performance data to a single processing stage, while a dual-stage model, involving an early evidence accumulation phase followed by a later decision phase might exhibit a more robust fit to the data than a single-stage model [26]. Future studies might also want to conduct a drift modeling approach on single trial fMRI to delineate the neurocircuitry associated with practice and congruency effects.

A primary methodological difference between the Sack et al. [5] and Cohen-Kadosh et al. [6] studies and the present one is that subjects were relatively unpracticed on the task in the former but well-practiced in the latter, and the main difference in results was that an rTMS effect was seen directly in the RT data of the former studies but not the latter. In terms of the present drift rate finding, these differences suggest that within the decision process, at least in a well-practiced state, the mechanism underlying the integration of congruent properties differs and can be more easily distinguished from that underlying the integration of incongruent properties. A further study comparing the effects of rTMS on subjects that are task-naïve vs. practiced would be needed to verify this. Success in such a study would result in a toolbox of manipulations involving states of practice, active and sham TMS applied at sites involved in task processing (e.g., left and right IPL, DLPFC), congruent and incongruent visual properties, and use of drift diffusion modeling. Use of this toolbox could provide the means for a more extensive investigation into the neural mechanisms of perceptual decision making.

On replicating an rTMS effect observed in earlier studies

In this study, we tested whether fMRI-guided rTMS effects on numerical Stroop task found in Cohen Kadosh et al. [6] and Sack et al [5] could still be observed while using more advanced techniques that have emerged in the last decade. In general, we adhered to their reported methods, and when possible, we substituted procedures that corresponded to more recent experimental standards (for example by employing electrical sham stimulation). We expected that the substitution for some of the older techniques used in the earlier papers would potentially improve the effect size they obtained. However, contrary to these expectations, we did not find better results, and instead failed to show any superiority of active rTMS overs sham stimulation. While it is clear that reproducing an experiment and its outcome in a different lab is never simple or even necessarily straight-forward, it is a useful exercise to examine the reasons that the replication of rTMS effect might not have occurred. Below we provide a list of the changes we made compared to the original studies and their possible consequences on the outcome of this study, as well as considerations for future rTMS studies using the numerical Stroop task to improve rigor and reproducibility. These are grouped into features involving the task itself, the study design, and rTMS delivery.

Issues concerning handling of the Stroop task in experimental procedures

Practice effect

After a brief introductory period, the difference in RT between congruent and incongruent conditions is maintained in the numerical Stroop task, even after extensive training (see, for example, Fig 1), which makes the task a good candidate for use in a repeated measures design in rTMS experiments looking at stimulation effects across congruency conditions. However, first subjects must be introduced to and practice the task. In this study, we trained participants with one block of 48 task trials before the MRI (followed by performance of 4 blocks of the task while in the scanner), and then a full block of 72 practice trials in the second session, prior to rTMS, which led to a relatively stabilized reaction time for the experimental conditions. In contrast, Sack et al. [5] only familiarized participants with the task for a few trials until it was clear the participants understood the task instructions. This difference in practice may have contributed to the differences seen in the effects of TMS between the studies. It should be noted, however, that the fMRI-targeted condition in Sack et al. received more practice on the task (because of performing additional blocks of the task while in the scanner) than their other targeting methods. This discrepancy in the degree of practice on the task across targeting conditions in Sack et al. calls into question whether there is definitive evidence that individualized fMRI-guidance is the most effective way to target rTMS to modulate performance on the numerical Stroop. To make such a definitive statement, it would be important to equate the degree of practice on the task across the targeting conditions.

It may very well be that a brain in an unpracticed state, while it is still optimizing the processing needed to carry out the task, is more vulnerable to external stimulation than a brain in a practiced state, where the task processing is more stable and set and resilient to disruption. Further, our interpretation of the Sack et al. [5] and Cohen Kadosh et al. [6] rTMS effect on the numerical Stroop task performance was related to an addition-by-subtraction mechanism, where performance enhancement occurred because rTMS disrupted the interference of the irrelevant stimulus dimension (font size) in the decision process. However, our study using well-practiced subjects found that the application of rTMS did not lead to significant improvements in RT in the incongruent condition, suggesting that there may be a ceiling effect in terms of the potential benefits of rTMS to right IPS in enhancing RT in that condition of the Stroop task. Future studies examining the effect of rTMS on the numerical Stroop task in both practiced and unpracticed states may be warranted and could examine whether the locus of the effect shifts in practiced subjects to another location in the prefrontal-parietal executive network involved in the task.

Our findings, and their contrast with the previous studies using the numerical Stroop, also highlight the importance of considering individual performance levels, training protocols, and potential ceiling effects when investigating the effects of TMS in cognitive tasks such as the numerical Stroop task. Further research is needed to explore the optimal conditions and target populations where rTMS interventions may have the greatest impact on task performance. Incorporating a staircase procedure in future studies may offer a viable solution to mitigate the confounding effects of training [27]. By dynamically adjusting the task difficulty based on individual performance, a staircase approach can help maintain a consistent level of challenge and control for potential learning effects. This can enhance the accuracy and reliability of assessing the impact of TMS on task performance and provide valuable insights into the underlying mechanisms of cognitive processes. Further research is needed to explore the feasibility and effectiveness of implementing a staircase procedure in this context.

Use of performance feedback

One specific manipulation we implemented in our task presentation was to provide the participants with feedback regarding their performance, which was not employed in the earlier studies. In our study, we decided to provide trial-by-trial feedback for the two practice blocks to facilitate learning, and feedback on average performance over a whole block of trials for each of the MRI and TMS blocks. Our decision was informed by Liu et al. [28] who used a Hebbian reweighting model to demonstrate that both types of feedback induce significant, equal learning.

Accounting for the numerical distance between the digit stimuli

Neither the present study nor Sack et al. [5] or Cohen Kadosh et al. [6] considered the interaction of the congruent/incongruent Stroop effect and the varying effect on performance caused by the “distance effect” in the contrast between the cardinality of the two stimulus digits presented in a given trial of the task, which potentially can add to variability in performance across individuals and work to obscure the sought after rTMS effect. Generally, the “distance effect” has been found to be an inverse relationship of RT and the cardinality difference between two numbers. For example, Moyer & Landauer (1967) [23]observed that participants were faster when the distance between numbers was greater (e.g., 2 and 7 versus 2 and 4), and that larger numbers (e.g., 8) were statistically more likely to be correct answers due to fewer single digits being greater than that number (e.g., 1-7 < 8<9). Methods to control for the distance effect include using number pairs with a set numerical ratio of 0.3 (e.g., 6, 2) [29], and presenting set pairs with differences of 1 (e.g., 3-4) and 5 (e.g., 4-9) to control the difference and number of times each digit was presented [30]. However, these controls limit the variability of trials and may lead to artificially low reaction times. While not accounted for in our study, the distance between the digit pairs used as stimuli should be balanced such that these effects are expected to be similar between active and sham conditions. Future studies using the numerical Stroop task should consider these limitations.

Issues involving experimental design

A possible cumulative rTMS effect

The five subjects in the group using individualized fMRI targeting in Sack et al. [5] received sham and active rTMS conditions in the same session, and as a group showed a significant active-sham difference in numerical Stroop performance. In contrast, in the present study the participants who first received active rTMS followed by sham rTMS did not show the significant reaction time improvement in incongruent trials that was seen in the group that began with sham rTMS (in post hoc testing following a Stimulation x Congruency x Order interaction). While this could be an accident of sampling in groups with small sample size, it may reflect, instead, a potential carryover effect of rTMS. Online stimulation is expected to modulate neural activity in an acute fashion, providing insight into the timing of neural processing that underlies behavior, and there is an underlying assumption of independence between trials in the effect of rTMS. However, very few studies have tested whether these short-term effects could accumulate over time. To our knowledge only two studies, including one from our group, have tested this assumption and concluded there is a lack of evidence for such cumulative rTMS effects [31, 32]. However, the stimulation parameters were different between each of these studies (10Hz during 3 seconds at 110% rMT over the DLPFC and superior parietal lobule; 5Hz during 4 seconds at 120% rMT over the DLPFC). These differences prevent direct comparison with each other or with this specific protocol. A potential way to assess this effect would be to interleave trials with and without stimulation. Overall, in terms of experimental design, we would recommend performing active and sham stimulation with sufficient time between the two conditions (e.g., perform them on different days) to mitigate any potential confound with accumulating effects.

Electrical sham

Sack et al. [5] used a placebo coil for sham stimulation, in which the sound of the TMS pulse and the pressure of the coil on the head are reproduced, but the sensation of the pulse is not. We used an electrical sham approach that uses electrodes applied to the participant’s head underneath the TMS coil, that delivers a weak electrical current that mimics the somatosensory stimulation induced by active rTMS. We used a titration approach where we asked participants to report whether the sensation was tolerable. While this approach appears to be an improvement, it is also associated with disadvantages. For example, because of hair thickness on the parietal cortex, it was sometimes difficult to keeping sham electrodes firmly adhered to the scalp, resulting in sham TMS pulses not felt by some participants, even at the maximum stimulation intensity. It is also possible that the residual E-fields induced in the brain by sham electrodes lead to potential neuromodulatory effects on behavioral performance [19]. Further, to protect subject blinding, we followed the recommendations from Smith, & Peterchev (2018) and told the participants that two types of stimulation would be applied. However, since our study did not include any blinding questionnaire, we cannot conclude on whether participants felt any difference. To address these limitations, future studies could include post TMS questions on this topic. Since Sack et al. [5], did not use this sham methodology, potential differences in performance caused by the addition of electrical stimulation could be another reason contributing towards the failed replication.

Issues involving TMS delivery

Target selection

In selecting a TMS target through the use an individual fMRI, the individual’s peak activation in a contrast of interest within a circumscribed cortical region is chosen. But the actual size of that region is somewhat challenging to delimit a priori, as it is generally based on previous coordinates in a group analysis but should also consider spatial variability across individuals. We chose to follow the description given in Sack et al. [5]: the individual location within the right IPS exhibiting the strongest BOLD-signal contrast for the SCE. To select the TMS target, we used the IPS mask from the Neurosynth website (https://neurosynth.org/), within which we estimated the location the peak activation for each participant. However, the size of this mask was quite large and the choice of target within this mask suffers from a large variability across subjects in the location of the target selected, probably due to the lack of additional anatomical constraints in our mask (Fig 2B). We also constrained our choice to be the most active location situated on superficial cortex, to induce the strongest E-field and potentially lead to the strongest TMS efficacy. This clearly differed from Sack et al., who did not introduce such a constraint, and in fact chose targets that were deeper in cortex (but, it turned out, grouped closer together). While Sack et al. did not describe the coil orientation chosen for each target, in the present study the coil orientation was chosen such that the second phase of the induced E-field was perpendicular and directed into the nearest sulcal wall for each individual subject. This was done visually rather than through E-field modeling and could have suffered some variability as a result. Overall, while there is general agreement in the field that, as concluded by Sack et al. [5], the use of individual fMRI activations for TMS targets is the best method for TMS target selection, given the low signal-to-noise levels in individual fMRI measures, there is still room for improvement.

A potential approach to overcome these limitations could be to rely on information from causally derived targets beyond the causal evidence provided by earlier TMS studies [5, 6]. For instance, studies have demonstrated that selecting targets from brain lesions or deep brain stimulation studies could potentially serve as effective TMS targets (see [33] for a review). While most causal targets have been derived from psychiatric symptoms such as depression or anxiety, examining behavioral deficits associated with focal brain injuries or deep brain stimulation can help selecting a TMS target for this specific executive function.

Regarding the group fMRI results, contrary to our expectations, while we were able to find significantly activated voxels for most of our participants (64%), our results at the group level do not indicate any significant activations in the right IPS in the Incongruent>Congruent contrast. Instead, we found some small clusters in the left IPS, and some larger clusters in subcortical structures such as the corpus callosum, and the cingulate cortex, the latter regions involved in conflict monitoring [34]. However, the lack of a group activation in the right IPS does not necessarily imply that it was not a good target. Indeed, it has been demonstrated that fMRI group analysis might not always detect the brain regions that are necessary at the individual level, and instead consider the individual topographic variability as noise [35] and that applying rTMS on subject-specific regions, even though variable, can lead to a significant rTMS effect while stimulating the group fMRI activation site does not [35].

A potential way to improve rTMS effects in this task might be to investigate the role of the left IPS during the Stroop task. To our knowledge, only two studies have tried this location and demonstrated either a stronger effect by applying TMS on the left IPS than the right IPS [36] or an effect only for the right hemisphere [6]. Another approach could be to conduct a connectivity analysis and to stimulate the superficial region showing the strongest connectivity with the corpus callosum. This has not been tested with this region and with this task, but this connectivity-based technique has previously demonstrated strong effects when rTMS was applied over the parietal cortex, to indirectly modulate the hippocampus during an episodic memory task [37] and it may be useful in the present context.

Stimulation intensity

We used the same dosing strategy of applying a fixed intensity across subjects that was used in the original Sack et al. study (60% maximum stimulator output). However, while the stimulators used in their study and the present one were the same models, the TMS coil we used (A/P-B65) was not the one used by Sack et al (B-70). The electric field generated by those two coils are slightly different [38], and that, combined with the more superficial location of our TMS targets compared to Sack et al., could also be a potential confound explaining the difference in outcomes. Further, individual differences in anatomy are something that should be taken into account when choosing intensity, given the correlation we found between the intensity expressed in relation to each individual’s motor threshold and the size of the rTMS effect observed. The correlation suggests that using higher intensity might increase the rTMS effect.

Number of pulses and stimulation timing

Like Sack et al. [5], we delivered three pulses of 10Hz rTMS at 220, 320, and 420 ms post-stimulus onset. This timing was based on past event-related potentials (ERPs) studies that found ERP components correlating perceptual processing of congruity and numerical distance over the parietal electrodes between 300-480 ms [39]. Szucs and colleagues sought to determine whether the facilitation and interference effects of relevant/irrelevant information occurred during perceptual or response processing stages of the numerical Stroop task. To do so, they had participants conduct two sub-tasks, either selecting the value greater in numeric value or physical size. Citing early work discussing the bidirectional congruency effects in the numerical Stroop task [40] and finding similar early ERP timing in both sub-tasks, they concluded that numerical and size dimensions of the task were processed in parallel regardless of relevance. With faster responses in the size task, they identified the facilitation and interference of the physical size on the numerical value determination. More recently, Huang et al. [41], investigated the interaction between task-relevant and irrelevant features on task performance, and found results contradicting Szucs & Soltez [38]. Huang et al. (2020) evaluated a temporally longer range of ERPs including the late positive complex (LPC). They found more positive responses in congruent than incongruent in early (320-520 ms) and late (520-720 ms) LPCs, with early also showing more positive responses in congruent vs neutral trials. Therefore, it is possible that the 220-420 ms was not the optimal timing for this task, and future studies might want to investigate different timing or add more pulses to cover a larger section of the neural processing.

Summary of replication findings. In the present study, we tried to reproduce the rTMS effect on numerical Stroop found in Sack et al. [5] and Cohen Kadosh et al. [6], with some newer methods that are now available in the field, such as robotic coil holder and electrical sham, hoping to find a similar or even stronger rTMS effect by combining those approaches with a larger sample size. In designing our experiment, we did not initially perform a power analysis on the optimal number of participants needed to observe significant difference between active and sham rTMS, relying on the findings of Cohen-Kadosh et al., and Sack et al., who each found a significant rTMS effect with 5 participants. While we tried to emulate their methods, the changes we did make in the experimental design may have potentially lowered the effect size, and we may have needed to employ a larger sample. There were some differences in the procedures used in the earlier and present studies. In terms of the numerical Stroop task, we practiced our subjects to a greater degree, additionally using feedback to accelerate their learning. While we succeeded in removing the confound of learning, we may have altered the neural processing involved, changing the sensitivity of right IPS to rTMS in the context of the Stroop task. Regarding experimental design, we differed in the addition of electrical stimulation to the sham condition, and we may also have found a cumulative effect of active/sham condition order that may have masked the performance effect we sought. Additionally, there were differences in how targeting choices using subject-specific fMRI data were handled that may have had profound effects on our results. We further identified several issues that were common to all the studies, involving stimulus choices, unbalanced distances between the digit stimuli presented, and choices of rTMS parameters such as intensity and stimulus timing, which could impact rTMS effects on the numerical Stroop task.

Conclusions

In two earlier studies the use of rTMS to right IPS during performance of a numerical Stroop task resulted in strong RT changes reflecting a diminishment of the interference of irrelevant but automatically processed visual properties [5, 6]. While we did not directly replicate those findings, application of a drift diffusion model to performance data demonstrated a strong effect of rTMS on the drift rate variable in the decision process. This analysis revealed possible differences in how congruent and incongruent properties are integrated during perceptual decision making. To compare rTMS targeting methods during a cognitive task, we first must be able to validate and replicate the rTMS effect on the task. Our results show that replicating rTMS effects remains challenging and we highlight steps that can be taken to improve the rigor and reproducibility in rTMS studies of cognition. For example, the differences and issues in replicating rTMS studies that we discussed point to the need for more extensive experimental planning and piloting, as well as more documentation of methodology (such as degree of practice on the task). Moreover, the sensitivity to changes of the effect of rTMS on the numerical Stroop task and its resistance to replication calls into question conclusions that can be drawn from it, and thus indirectly, conclusions on the best method for rTMS targeting. This calls for further work establishing best methods for TMS targeting. It also suggests more generally the need for further development of procedures to establish what constitutes a brain/behavior “TMS effect”, both in terms of the modeling of performance data and on the cognitive processes involved, as well as engagement of brain networks supporting that performance. The application of more sophisticated behavioral performance analysis such as drift diffusion modeling may help parse out more subtle effects of rTMS on brain networks involved in the task.

Supporting information

S1 Table. Summary of stimulation target location and stimulation intensity for each participant.

Stimulation coordinates are in LPI coordinates system, t-scores at the stimulated target, stimulation intensity expressed as a percentage of the resting motor threshold, Euclidean distance to Sack et al. average target, and scalp-to-cortex distance for each participant. Highlighted in blue are the lowest values and in red the highest values. Participant is italics font was the outlier not included in the analyses.

(DOCX)

pone.0302660.s001.docx (15.8KB, docx)
S2 Table. Overlap between activated clusters in the Incongruent > Congruent contrast and their overlap with the Glasser atlas.

(DOCX)

pone.0302660.s002.docx (15.1KB, docx)

Acknowledgments

We would like to thank Dr. Eudora Jones, Dr. Fahad Mukhtar, and Dr. William Regenold for providing medical coverage for each session; as well as Cristina Abboud, Megan Hynd, Elyssa Feuer, Paul Rohde, and Pei Robbins for their help with participants’ screening.

This work utilized the computational resources of the NIH HPC Biowulf cluster (http://hpc.nih.gov).

Data Availability

Behavioral data are available from the Figshare database: https://figshare.com/articles/dataset/NumericalStroopTMS_xlsx/25130669.

Funding Statement

Beynel L., Gura, H., Rezaee, Z., Ekpo, E., Joseph, J., Deng, Z-D., Luber, B., and Lisanby, S.H. are supported by the NIMH Intramural Research Program (ZIAMH002955). Taylor, P. was supported by the NIMH Intramural Research Program (ZICMH002888) of the NIH/HHS, USA. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1.Stroop RJ. Studies of Interference In Serial Verbal Reactions. J Exp Psychol. 1935. [Google Scholar]
  • 2.Milham MP, Banich MT. Anterior cingulate cortex: An fMRI analysis of conflict specificity and functional differentiation. Hum Brain Mapp. 2005;25: 328–335. doi: 10.1002/hbm.20110 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Kaufmann L, Koppelstaetter F, Delazer M, Siedentopf C, Rhomberg P, Golaszewski S, et al. Neural correlates of distance and congruity effects in a numerical Stroop task: An event-related fMRI study. Neuroimage. 2005;25: 888–898. doi: 10.1016/j.neuroimage.2004.12.041 [DOI] [PubMed] [Google Scholar]
  • 4.Hayward G, Goodwin GM, Harmer CJ. The role of the anterior cingulate cortex in the counting Stroop task. Exp Brain Res. 2004;154: 355–358. doi: 10.1007/s00221-003-1665-4 [DOI] [PubMed] [Google Scholar]
  • 5.Sack AT, Kadosh RC, Schuhmann T, Moerel M, Walsh V, Goebel R. Optimizing functional accuracy of TMS in cognitive studies: A comparison of methods. J Cogn Neurosci. 2009;21: 207–221. doi: 10.1162/jocn.2009.21126 [DOI] [PubMed] [Google Scholar]
  • 6.Cohen Kadosh R, Cohen Kadosh K, Schuhmann T, Kaas A, Goebel R, Henik A, et al. Virtual Dyscalculia Induced by Parietal-Lobe TMS Impairs Automatic Magnitude Processing. Current Biology. 2007;17: 689–693. doi: 10.1016/j.cub.2007.02.056 [DOI] [PubMed] [Google Scholar]
  • 7.Luber B, Lisanby SH. NeuroImage Enhancement of human cognitive performance using transcranial magnetic stimulation (TMS). Neuroimage. 2014;85: 961–970. doi: 10.1016/j.neuroimage.2013.06.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Beynel L, Appelbaum LG, Luber B, Crowell CA, Hilbig SA, Lim W, et al. Effects of online repetitive transcranial magnetic stimulation (rTMS) on cognitive processing: A meta-analysis and recommendations for future studies. Neurosci Biobehav Rev. 2019;107: 47–58. doi: 10.1016/j.neubiorev.2019.08.018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Ratcliff R, Smith PL, Brown SD, McKoon G. Diffusion Decision Model: Current Issues and History. Trends Cogn Sci. 2016;20: 260–281. doi: 10.1016/j.tics.2016.01.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Kadosh RC, Muggleton N, Silvanto J, Walsh V. Double dissociation of format-dependent and number-specific neurons in human parietal cortex. Cerebral Cortex. 2010;20: 2166–2171. doi: 10.1093/cercor/bhp273 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Hartwigsen G, Bergmann TO, Herz DM, Angstmann S, Karabanov A, Raffin E, et al. Modeling the effects of noninvasive transcranial brain stimulation at the biophysical, network, and cognitive Level. Progress in Brain Research. Elsevier B.V.; 2015. pp. 261–287. doi: 10.1016/bs.pbr.2015.06.014 [DOI] [PubMed] [Google Scholar]
  • 12.Peirce J, Gray JR, Simpson S, MacAskill M, Höchenberger R, Sogo H, et al. PsychoPy2: Experiments in behavior made easy. Behav Res Methods. 2019;51: 195–203. doi: 10.3758/s13428-018-01193-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Cox RW. AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance Neuroimages. COMPUTERS AND BIOMEDICAL RESEARCH. 1996. doi: 10.1006/cbmr.1996.0014 [DOI] [PubMed] [Google Scholar]
  • 14.Cox RW, Glen DR. Nonlinear warping in AFNI. Poster presented at the 19th Annual Meeting of the Organization for Human Brain Mapping. 2013. [Google Scholar]
  • 15.Fonov V, Evans AC, Botteron K, Almli CR, McKinstry RC, Collins DL. Unbiased average age-appropriate atlases for pediatric studies. Neuroimage. 2011;54: 313–327. doi: 10.1016/j.neuroimage.2010.07.033 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Reynolds RC, Taylor PA, Glen DR. Quality control practices in FMRI analysis: Philosophy, methods and examples using AFNI. Front Neurosci. 2023;16. doi: 10.3389/fnins.2022.1073800 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Saad ZS, Glen DR, Chen G, Beauchamp MS, Desai R, Cox RW. A new method for improving functional-to-structural MRI alignment using local Pearson correlation. Neuroimage. 2009;44: 839–848. doi: 10.1016/j.neuroimage.2008.09.037 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Glen DR, Taylor PA, Buchsbaum BR, Cox RW, Reynolds RC. Beware (Surprisingly Common) Left-Right Flips in Your MRI Data: An Efficient and Robust Method to Check MRI Dataset Consistency Using AFNI. Front Neuroinform. 2020;14. doi: 10.3389/fninf.2020.00018 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Smith JE, Peterchev V A. Electric field measurement of two commercial active/sham coils for transcranial magnetic stimulation. J Neural Eng. 2018;15. doi: 10.1088/1741-2552/aace89 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Chapter Awiszus F. 2 TMS and threshold hunting. Supplements to Clinical Neurophysiology. Elsevier B.V.; 2003. doi: 10.1016/S1567-424X(09)70205-3 [DOI] [PubMed] [Google Scholar]
  • 21.Wagenmakers E-J, Van Der Maas HLJ, Grasman RPPP. THEORETICAL AND REVIEW ARTICLES An EZ-diffusion model for response time and accuracy. Psychon Bull Rev. 2007. [DOI] [PubMed] [Google Scholar]
  • 22.Taylor PA, Reynolds RC, Calhoun V, Gonzalez-Castillo J, Handwerker DA, Bandettini PA, et al. Highlight results, don’t hide them: Enhance interpretation, reduce biases and improve reproducibility. Neuroimage. 2023;274. doi: 10.1016/j.neuroimage.2023.120138 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Moyer RS, Landauer TK. Time required for Judgements of Numerical Inequality. Nature. 1967;215: 1519–1520. doi: 10.1038/2151519a0 [DOI] [PubMed] [Google Scholar]
  • 24.Philiastides MG, Auksztulewicz R, Heekeren HR, Blankenburg F. Causal role of dorsolateral prefrontal cortex in human perceptual decision making. Current Biology. 2011;21: 980–983. doi: 10.1016/j.cub.2011.04.034 [DOI] [PubMed] [Google Scholar]
  • 25.Lerche V, Voss A, Nagler M. How many trials are required for parameter estimation in diffusion modeling? A comparison of different optimization criteria. Behav Res Methods. 2017;49: 513–537. doi: 10.3758/s13428-016-0740-2 [DOI] [PubMed] [Google Scholar]
  • 26.Hübner R, Steinhauser M, Lehle C. A dual-stage two-phase model of selective attention. Psychol Rev. 2010;117: 759–784. doi: 10.1037/a0019471 [DOI] [PubMed] [Google Scholar]
  • 27.Beynel L, Davis SW, Crowell CA, Hilbig SA, Lim W, Nguyen D, et al. Online repetitive transcranial magnetic stimulation during working memory in younger and older adults: A randomized within-subject comparison. PLoS One. 2019;14. doi: 10.1371/journal.pone.0213707 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Liu J, Dosher B, Lu ZL. Modeling trial by trial and block feedback in perceptual learning. Vision Res. 2014;99: 46–56. doi: 10.1016/j.visres.2014.01.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Leibovich T, Diesendruck L, Rubinsten O, Henik A, Libertus M, Hopkins J. The importance of being relevant : modulation of magnitude representations. Front Psychol. 2013;4: 1–10. doi: 10.3389/fpsyg.2013.00369 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Kadosh RC, Kadosh KC, Henik A. When brightness counts: The neuronal correlate of numerical-luminance interference. Cerebral Cortex. 2008;18: 337–343. doi: 10.1093/cercor/bhm058 [DOI] [PubMed] [Google Scholar]
  • 31.Hamidi M, Johson JS, Feredoes E, Postle BR. Does high-frequency repetitive transcranial magnetic stimulation produce residual and/or cumulative effects within an experimental session? Brain Topogr. 2011;23: 355–367. doi: 10.1007/s10548-010-0153-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Beynel L, Powers JP, Appelbaum LG. Effects of repetitive transcranial magnetic stimulation on resting-state connectivity: A systematic review. Neuroimage. 2020;211. doi: 10.1016/j.neuroimage.2020.116596 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Siddiqi SH, Fox MD. Targeting symptom-specific networks with TMS. Biol Psychiatry. 2023. doi: 10.1016/j.biopsych.2023.11.011 [DOI] [PubMed] [Google Scholar]
  • 34.Botvinick MM, Braver TS, Barch DM, Carter CS, Cohen JD. Conflict Monitoring and Cognitive Control Despite the importance of these efforts to characterize the func-tion of cognitive control, most of them share an important limita-tion in scope. Most current theories focus nearly exclusively on the. Cohen & Servan-Schreiber. 2001;108: 624–652. [Google Scholar]
  • 35.Feredoes E, Tononi G, Postle BR. The neural bases of the short-term storage of verbal information are anatomically variable across individuals. Journal of Neuroscience. 2007;27: 11003–11008. doi: 10.1523/JNEUROSCI.1573-07.2007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Cappelletti M, Muggleton N, Walsh V. Quantity without numbers and numbers without quantity in the parietal cortex. Neuroimage. 2009;46: 522–529. doi: 10.1016/j.neuroimage.2009.02.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Wang JX, Rogers LM, Gross EZ, Ryals AJ, Dokucu ME, Brandstatt KL, et al. Targeted enhancement of cortical-hippocampal brain networks and associative memory. Science (1979). 2014;345: 1054–1057. doi: 10.1126/science.1252900 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Deng Z De Lisanby SH, Peterchev A V. Electric field depth-focality tradeoff in transcranial magnetic stimulation: Simulation comparison of 50 coil designs. Brain Stimul. 2013;6: 1–13. doi: 10.1016/j.brs.2012.02.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Szucs D, Soltez F. Event-related potentials dissociate facilitation and interference effects in the numerical Stroop paradigm. Neuropsychologia. 2007;45: 3190–3202. doi: 10.1016/j.neuropsychologia.2007.06.013 [DOI] [PubMed] [Google Scholar]
  • 40.Macleod CM. Haifa Century of Research on the Stroop Effect: An Integrative Review. Psychol Bull. 1991. [DOI] [PubMed] [Google Scholar]
  • 41.Huang HW, Nascimben M, Wang YY, Fong DY, Tzeng OJL, Huang CM. Which digit is larger? Brain responses to number and size interactions in a numerical Stroop task. Psychophysiology. 2020;58: 1–13. doi: 10.1111/psyp.13744 [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Desmond J Oathes

1 Feb 2024

PONE-D-23-36866What is the best way to target rTMS?  Lessons learned from a replication attempt on fMRI-guided rTMS modulation of the numerical Stroop task.PLOS ONE

Dear Dr. Beynel,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

The reviewers clearly see value in the work but also there is some consistency that the title is misleading given no head-to-head comparison of targeting approaches. The reviewers make a number of other excellent points, requests for additional analyses, and suggestions for framing that will further improve the manuscript.

==============================

Please submit your revised manuscript by Mar 15 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Desmond J. Oathes

Academic Editor

PLOS ONE

Journal requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. Thank you for stating the following financial disclosure: 

 [Beynel L., Gura, H., Rezaee, Z., Ekpo, E., Joseph, J., Deng, Z-D., Luber, B., and Lisanby, S.H. are supported by the NIMH Intramural Research Program (ZIAMH002955).

Taylor, P. was supported by the NIMH Intramural Research Program (ZICMH002888) of the NIH/HHS, USA.].  

Please state what role the funders took in the study.  If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."" 

If this statement is not correct you must amend it as needed. 

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

3. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: " ext-link-type="uri" xlink:type="simple">https://www.youtube.com/watch?v=_xcclfuvtxQ".

4. Thank you for stating the following in the Competing Interests section: 

[The opinions expressed in this article are the authors' and do not reflect the views of the National Institutes of Health, the Department of Health and Human Services, or the United States government.  Dr. Deng is inventor on patents and patent applications on electrical and magnetic brain stimulation therapy systems held by the National Institutes of Health (NIH), Columbia University, and University of New Mexico. Dr. Lisanby is inventor on patents and patent applications on electrical and magnetic brain stimulation therapy systems held by the NIH and Columbia University, with no remuneration.]. 

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: ""This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. 

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

5. Thank you for uploading your study's underlying data set. Unfortunately, the repository you have noted in your Data Availability statement does not qualify as an acceptable data repository according to PLOS's standards.

At this time, please upload the minimal data set necessary to replicate your study's findings to a stable, public repository (such as figshare or Dryad) and provide us with the relevant URLs, DOIs, or accession numbers that may be used to access these data. For a list of recommended repositories and additional information on PLOS standards for data deposition, please see https://journals.plos.org/plosone/s/recommended-repositories.

6. Please include the reference section of your manuscript.

7. We note that Figure(s) 4A and 4B in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure(s) 4A and 4B to publish the content specifically under the CC BY 4.0 license. 

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an ""Other"" file with your submission. 

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I really enjoyed reading this paper. The authors did a fantastic and rally went the extra mile in terms of methodology and data analyses. Chapeau! I also love that they were inspired by our original article from 2009, and aimed at replicating our comparison of different TMS coil positioning approaches.

The discussion is comprehensive and addresses many factors that may have contributed to the result(s) and may explain differences between their and our own findings from 2009.

There is so much food for thought in there that I absolutely applaud such as the question of replicability and robustness of TMS-induced behavioral performance effects and their dependence on, e.g., degree of practice on the task at hand, or experimental design choices such as between versus within-subject comparisons etc. This is all wonderful.

My only and main concern here is that the title, abstract and introduction, in fact the overall framing of this paper, is misleading and unjustified. The title literally promises to address the question “What is the best way to target rTMS?”

And then the abstract and introduction frames this as a replication of our comparison of different targeting approaches from 2009. But, in sharp contrast to this framing , in the current study no comparison of targeting approaches is presented. In fact, only fMRI-guided TMS neuronavigation is used to test for effects on performances in the numerical stroop task. The current study only compares active and sham stimulation and fails to replicate the behavioral RT effects reported in earlier TMS studies using the same task. There is no comparison of TMS coil positioning approaches and the question in the title “What is the best way to target rTMS?” is not addressed by the study. I understand that replicating the behavioral effect is a step towards a replication study of the localization differences reported by us, but now readers will get the impression that they already did that and failed.

The authors seem to suggest that the failed replication of the behavioral effect using fMRI-guided TMS casts doubt on the localization / TMS targetting effect. I think this conclusion is very questionable.

The main gist and focus of our study in 2009 were to compare the relative differences between different TMS targeting approaches. It was not meant to claim that you need only 5 subjects or 15 subjects or whatever number of subjects to find a significant behavioral finding in the numerical stroop test when using this or that targeting approach. The design and overall extremely low sample size was completely unsuited for that (5 subjects, between subject design). We were also not interested in the numerical stroop task per se. This task just served as a tool, a means to compare the performance differences induced by the various TMS targeting approaches we compared with each other, showing relative differences. The numerical stroop task we used was brought to us by our visiting researcher, Dr. Roi Cohen Kadosh, who was PI of a research line using this task in the fMRI.

In related publications, e.g. Cohen Kadosh 2007, this team reported right parietal activity changed during the execution of this numerical stroop task and behavioral effects when stimulating those individual right parietal activity clusters using TMS.

I understand that the authors originally wanted to replicate our targeting approach study but got stopped at step 1, when hoping to first find the expected behavioral changes when using the supposedly best targeting approach: individual fMRI-guided TMS. Since they could not find any behavioral effects here, they now publish this still in the framing of an intended and failed replication of our TMS targeting study. But in fact, they failed to find the behavioral effects using one targeting approach. This is something completely different.

In fact, when looking at their results in more detail, the issue of replicating previous work already starts with the fMRI data itself. Unlike previous work by Cohen Kadosh and others, the fMRI data of the current does not demonstrate clear individual right parietal activity clusters to underlie the execution of this task. In fact, in many cases, their fMRI data, if really used to localize the individual hotspot of this task for TMS targeting, would have suggested areas in the left parietal cortex.

So, ironically, because they aimed to replicate our study of 2009, they wanted to stick to the right IPS (because this is where we fund our fMRI hotspot in each participant for this task), but by doing so, they actually violated the entire principle of individual fMRI-guided TMS targeting which bases the TMS positioning on the individual fMRI hotspot, data-driven, not hypothesis- or replication-driven.

The should have listed t the individual fMRI findings and stimulate at the fMRI0based hotspot, and maybe, if they then would have found a significant behavioral effect, the could have compared this to other targeting approaches. This would have been closer to replicating our study than sticking to a right IPS cluster because of an a priori principle. Redo the analyses, without masks or restrictions and do the TMS based on the individual hotspot, no matter whether right or left IPS. Otherwise, you are not capitalizing on the power of individual fMRI-guided TMS.

In sum, great study, many lessons learned, great discussion, but title, and framing as a replication of our study from 2009 is misleading, and the study is really not a replication of our work as it differs in too many methodological aspects and doesn’t even include any comparison of different TMS targeting approaches. This can easily be fixed by changing title and reframing abstract and introduction.

But I want to again congratulate the authors for conducting this study, I want to encourage them to continue on this path, and express my admiration for their high scientific standards and methodological rigor.

Best wishes,

Alexander Sack

Reviewer #2: The present study attempted to replicate prior findings showing that short bursts of high frequency TMS delivered to the right intraparietal sulcus disrupt the Stroop effect (i.e., Cohen Kadosh et al., 2007; Sack et al., 2009). Well conducted replication efforts like this one are rare, at least as can be determined by the published literature. However, I have several questions, comments, and suggested revisions that I think could strengthen the paper.

(1) Was a power analysis performed? If not, the authors should justify their sample size and acknowledge the lack of a power analysis as a limitation. I am skeptical of the authors’ claim that “rTMS effects on task performance are typically seen using sample sizes of thirteen and up”. This claim was made on the basis of their review of a heterogeneous behavioral TMS literature using a variety of TMS protocols and cognitive tasks. Surely, protocol and task differences impact power considerations.

(2) Was the decision to perform drift diffusion models made post-hoc? If so, consistent with the spirit of reproducibility that motivated the present study, it should probably be described that way. I was able to find general study details on clinical trials.gov but did not see a description of planned analyses.

(3) The Discussion includes material that should probably be outlined in the Introduction and better detailed in Methods. For example, drift diffusion models are mentioned only briefly in the Introduction. It would be useful to briefly outline why this approach was used versus others. More importantly, as described in detail in the Discussion section, there are several ways in which the present study differs from the studies that it sought to replicate. It would be useful to preview what motivated these differences in the Introduction and cover them in more detail in Methods.

(4) I think it would be worth testing to see if the ANOVA homoscedasticity assumption is violated, as the congruent/incongruent Stroop conditions may have different variances. If this assumption is not met, mixed models would be a more appropriate way to analyze these data.

(5) It may be worth mentioning (although I think this should be up to the authors) that causally derived targets may be better candidates than fMRI correlates, as these correlates may be spurious or reflect compensatory processes. TMS accessible cortical areas that are strongly connected to TMS/DBS sites or lesions that alter Stroop performance or Stroop relevant processes may be particularly promising.

(6) Could the authors please include a Methods figure detailing study procedures by visit?

(7) I am curious about the authors’ speculations on their drift rate finding that TMS affected congruent versus incongruent trials. This seems worth briefly commenting on in the Discussion section. However, I leave this up to the authors if they feel the speculations are too speculative.

Reviewer #3: This paper conducts a replication study of the facilitatory effects of rTMS on numerical Stroop task performance. They employed personalized targeting of the right intraparietal sulcus, as determined by peak activation. Although they were not able to replicate the reaction time results, they did find an increase in the drift rate in the congruent trials. The failure to replicate was attributed to various methodological differences, including task and stimulation administration.

Overall, the paper is clearly and concisely written and the included figures illustrate the results in a simple and elegant way. I also applaud the authors for a replication attempt due to the profound lack of these studies. My main point of criticism is the employed statistical methods. The use of more robust statistical tools could give more confidence in the results by appropriately modeling the experimental structure. I am aware that this suggestion brings the scope of this study slightly past replication, and note that it already involves an innovative use of DDM. I believe the authors can enhance their already meaningful contribution to the literature.

Abstract:

- No comments

Introduction:

- #112: Please further motivate the decision to include the drift diffusion model (DDM) in the analyses. While I am aware that these models are highly influential and agree that they are interesting in this context, the theoretical reason to use it was not clear. I believe there are compelling reasons to do so, but they should be stated even if the authors did not have any clear or directional hypotheses.

- I also think that while the Stroop task is quite old and famous, the deck is stacked against some studies due to the relatively small and variable behavioral effect. The authors do discuss these issues, but I do think a bit more could be stated about using traditional TMS sequences to find small and noisy effects in general. For instance, is there a confidence interval in which we would expect the current and previous study’s results to fall?

Methods:

- Was motion confound regression applied to the BOLD time series? If so, please indicate which motion parameters were used.

- Did the authors consider that the observations are not independent, as many samples are derived from the same participant? The use of a linear model, such as an ANOVA may not be appropriate in this case and may warrant the use of more advanced statistical methods such as a linear mixed effects model.

- For the DDM, although 5% of the trials were modelled with a different distribution, were outlier reaction times ( 200ms) removed from the analyses? It is likely that these trials reflect a different underlying cognitive process than what is trying to be measured.

- There are concerns that the number of trials used for each of the DDMs may have been too low, which may have affected the precision of parameter estimations. Please refer to Lerche et al. 2017 and comment with regards to the current study.

Results:

- As the lack of a size congruency effect was one of the main negative findings in the replication, please include a figure for the demonstrating the lack of a size congruency effect.

Discussion:

- There have been multiple studies involving DDM in the context of attentional and cognitive control that have required modifications to the DDM to account for these types of data (Hübner et al., 2010). Please refer to these works in the limitations section.

- Although the authors use the DDM results to discount the role of the right IPL in automatic spatial magnitude processing, do they have any cognitive or neurobiological explanation for why the drift rate may have increased in only the congruent condition?

- The choice of fMRI preprocessing parameters can change the findings of otherwise identical studies. Please comment on the differences in preprocessing schemes and how they may have affected your results.

References:

Lerche, V., Voss, A., Nagler, M. (2017). How many trials are required for parameter estimation in diffusion modeling? A comparison of different optimization criteria. Behavior Research Methods, 49(2), 513–537. https://doi.org/10.3758/s13428-016-0740-2

Hübner, R., Steinhauser, M., Lehle, C. (2010). A dual-stage two-phase model of selective attention. Psychological Review, 117(3), 759–784. https://doi.org/10.1037/a0019471

Minor:

- #213: Was the intention “highest tolerable intensity”?

Reviewer #4: General comments:

Overall this is a valuable study for the cognitive TMS field. It highlights the many factors that influence whether a TMS effect on behaviour will be observed. The discussion is particularly strong in this regard.

The study is primarily posed, however, as a replication of Sack et al. (2008), looking at the efficacy of individualised fMRI targeting to modulate numerical Stroop task performance. The aim of the Sack study was to compare different targeting approaches, whereas here the aim was to replicate one of the results from Sack et al. My first suggestion is therefore to change the title of the paper, as the current study is not addressing the targeting issue directly. (If replication of the targeting approach was the main aim of the present study, then this would have been better achieved by comparing against other targeting approaches, as per the original study.) While the finding of targeting approach (individual v group fMRI clusters for example) might not have been directly replicated, this issue has come up in other publication (e.g., Feredoes et al., 2007, J Neurosci), and many have used an individualised fMRI approach and shown individual variability around a specific target. Alongside findings from e-field modelling, it’s not clear this is such an issue these days.

While replications, especially in the cognitive TMS field, are always extremely valuable, the ‘message’ of this study is more related to the number of considerations that a researcher must face when designing and conducting study on behaviour. I would therefore suggest re-framing the paper along these lines, through an attempt to replicate what could be regarded as a robust finding, given the reliability of the Stroop effect and individualised targeting. I feel that this would be of interest to the field, given that the targeting of cognitive processes with TMS, while conceptually simple, remains challenging, and the targeting approach is only one consideration when it comes to maximising TMS efficacy.

Additional comments:

- How were the contrast maps thresholded? The t-values of the stimulated targets in Table 1 seem on the low side?

- Could some more detail be provided on how the IPS target was selected? The methods states that the ‘statistical peak’ within the right IPS was chosen (line 191), but what if there was more than one peak? Was it the cluster with the highest t-value? Were there any further anatomical constraints beyond being within the Neurosynth mask? From Figure 4 it does look like the individual targets are anatomically variable, which is not an issue by itself, but some of the targets look like they are in the precuneus, and several others are in the inferior parietal cortex.

- Given the multiple parameters that comprise a TMS study, would it be worthwhile trying to combine as many of these as possible (since they were recorded) to see if they can predict an individual’s results? Currently the values from Table 1 were assessed as single correlations with behaviour. Could they instead be predictors in a regression model? (The distance to the Sack group coordinate might not be useful to include, however, as they did not demonstrate a significant effect of TMS when this was targeted.) Such an analysis would also fit with the aim of the paper, which is to highlight the need for more sophisticated analyses, but also it would fit with our understanding of individual differences in response to TMS.

- For the group-level fMRI analysis, the lack of an effect in right IPS is not necessarily an issue, and would speak to the anatomical variability of this region. As mentioned above, this issue has been covered by others such as Feredoes et al. It may be worth highlighting this particular point in the discussion when mentioning the potential of left IPS as a better target.

Reviewer #5: I am reviewing “What is the best way to target TMS? Lessons learned from a replication attempt on fMRI-guided rTMS modulation of the numerical Stroop task.” by Beynel et al. In this study, the authors sought to replicate results from the influential Sack et al. 2009 paper, which showed the relative advantage and increase in statistical power obtained when using individualized vs group-MRI based (or scalp based) TMS targeting for improving performance in a numeric Stroop task. The current study largely failed to replicate Sack et al., and thoughtfully acknowledged a few reasons why that might be, while making recommendations for future studies. I appreciated the overall effort and study, and thought the manuscript was thoughtful and well written. However, some of the results, design and analysis choices, raised questions and a few concerns for me, which are delineated below:

Major

1. Most strikingly, from the way the intro was framed, I was under the impression that the authors were going to directly compared contrast different TMS targeting strategies (e.g. anatomical vs. functional based) within subjects (in contrast with the between-subjects approach used by Sack et al.) However, the study design only included the individualized TMS targeting condition, as opposed to the critical comparison of e.g. MRI/group- based coordinates vs. individualized, functional-based coordinates. While this point might appear moot in the context of the current (null) results, I still think this point (i.e., the primary study aim, which is not exactly a replication of Sack et al., which would have required the aforementioned explicit targeting strategy comparison) should be recontextualized and clarified in the Intro (and title).

2. A critical aspect of the results that should be taken into account explicitly across all analysis (and it currently isn’t) is the 3-way interaction with TMS order (sham first vs control first). If I followed the results correctly, the result obtained with the group who had sham first is precisely the predicted result. Given that TMS (sham vs. active) order clearly matters here, all subsequent analysis should include order in the model as either a covariate or perhaps more ideally as an interacting factor (including the SCE analysis!).

3. Relatedly, how does overall Stroop performance compare between Sack et al. the current study? I am finding that RT differences for Congruent vs. Incongruent were slightly larger in the Sack study (104ms, potentially providing a better range to modulate those effects) vs the current study (71ms?), but I am not finding the average accuracy data in the Sack study... That would be important to know and compare across the samples.

4. And how does the age of the participants compare? (I was struck by the relatively older age of the current sample, 39 y old). I am revisiting the Sack paper, and seeing that the mean age was 24 y old in that study… This point relates back to the performance point above, particularly if performance differs across the 2 samples…

- Does age correlate with Stroop performance and/or with the magnitude of TMS effects found here?

5. How far apart were the MRI and TMS sessions….? Does the interval matter in terms of the magnitude of the effects?

6. Accuracy seemed quite a bit lower for the TMS day (88%) compared to the fMRI day (98%). Was that a significant difference between those days? If so, is it possible that TMS is impacting accuracy (and/or differing by TMS site)?

7. I was wondering about whether there is a chance that the ‘sham’ condition in this study was less of a sham than what was expected…. which is a point the authors themselves raised in the Discussion. Is there a way to more directly get at this question, e.g., by comparing stroop performance (accuracy or RT) in the sham condition vs. the baseline fMRI day?

8. I was not entirely clear on the interpretation of Figure 3, particularly regarding the specificity to the Congruent condition. Can that interpretation be more clearly spelled out, both as a short summary in the Results section, as well as in the Discussion section?

Minor

1. I found Figures 2 3 hard to parse as the Stroop effect is most easily visualized by comparing Congruent vs. Incongruent conditions directly, which are currently plotted far away from one another; wouldn’t plotting Congruent vs. Incongruent side-by-side, and Sham vs. TMS (etc) on the X axis, make those plots easier to read?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Alexander Sack

Reviewer #2: No

Reviewer #3: Yes: John D. Medaglia

Reviewer #4: No

Reviewer #5: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: PLOS_One_Review.docx

pone.0302660.s003.docx (15.1KB, docx)
PLoS One. 2024 May 6;19(5):e0302660. doi: 10.1371/journal.pone.0302660.r002

Author response to Decision Letter 0


28 Feb 2024

¬¬Journal requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

Response: The manuscript has been edited to follow Plos one style templates.

2. Thank you for stating the following financial disclosure:

[Beynel L., Gura, H., Rezaee, Z., Ekpo, E., Joseph, J., Deng, Z-D., Luber, B., and Lisanby, S.H. are supported by the NIMH Intramural Research Program (ZIAMH002955).

Taylor, P. was supported by the NIMH Intramural Research Program (ZICMH002888) of the NIH/HHS, USA.].

Please state what role the funders took in the study. If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.""

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

Response: This information has been updated in our cover letter: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript

3. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ".

Response: The corresponding author ORCID ID has been added to the Editorial Manager

4. Thank you for stating the following in the Competing Interests section:

[The opinions expressed in this article are the authors' and do not reflect the views of the National Institutes of Health, the Department of Health and Human Services, or the United States government. Dr. Deng is inventor on patents and patent applications on electrical and magnetic brain stimulation therapy systems held by the National Institutes of Health (NIH), Columbia University, and University of New Mexico. Dr. Lisanby is inventor on patents and patent applications on electrical and magnetic brain stimulation therapy systems held by the NIH and Columbia University, with no remuneration.]. Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: ""This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Response: We have updated the competing interest’s statement in our cover letter.

5. Thank you for uploading your study's underlying data set. Unfortunately, the repository you have noted in your Data Availability statement does not qualify as an acceptable data repository according to PLOS's standards. At this time, please upload the minimal data set necessary to replicate your study's findings to a stable, public repository (such as figshare or Dryad) and provide us with the relevant URLs, DOIs, or accession numbers that may be used to access these data. For a list of recommended repositories and additional information on PLOS standards for data deposition, please see https://journals.plos.org/plosone/s/recommended-repositories.

Response: Data have been added to “figshare” repository with this URL: https://figshare.com/account/home#/projects/194468

6. Please include the reference section of your manuscript.

Response: References were at the end of the manuscript. We will add it separately in the document upload.

7. We note that Figure(s) 4A and 4B in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright. We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure(s) 4A and 4B to publish the content specifically under the CC BY 4.0 license. We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.” Please upload the completed Content Permission Form or other proof of granted permissions as an ""Other"" file with your submission. In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

[Note: HTML markup is below. Please do not edit.]

Response: Figure 4 is our own figure, no copyright needed.

Reviewer #1:

I really enjoyed reading this paper. The authors did a fantastic and rally went the extra mile in terms of methodology and data analyses. Chapeau! I also love that they were inspired by our original article from 2009 and aimed at replicating our comparison of different TMS coil positioning approaches. The discussion is comprehensive and addresses many factors that may have contributed to the result(s) and may explain differences between their and our own findings from 2009. There is so much food for thought in there that I absolutely applaud such as the question of replicability and robustness of TMS-induced behavioral performance effects and their dependence on, e.g., degree of practice on the task at hand, or experimental design choices such as between versus within-subject comparisons etc. This is all wonderful.

Response: Thank you very much (Merci!) Dr. Sack for those good words on our manuscript, we are glad that you found it interesting and, in the section below we provide a point-by-point response to your concerns.

1. My only and main concern here is that the title, abstract and introduction, in fact the overall framing of this paper, is misleading and unjustified. The title literally promises to address the question “What is the best way to target rTMS?” And then the abstract and introduction frames this as a replication of our comparison of different targeting approaches from 2009. But, in sharp contrast to this framing, in the current study no comparison of targeting approaches is presented. In fact, only fMRI-guided TMS neuronavigation is used to test for effects on performances in the numerical stroop task. The current study only compares active and sham stimulation and fails to replicate the behavioral RT effects reported in earlier TMS studies using the same task. There is no comparison of TMS coil positioning approaches and the question in the title “What is the best way to target rTMS?” is not addressed by the study. I understand that replicating the behavioral effect is a step towards a replication study of the localization differences reported by us, but now readers will get the impression that they already did that and failed.

Response: Thank you for raising this important concern. We acknowledge that our initial framing may have created a misleading impression regarding the scope of our study. While our ultimate goal with this research is to compare the efficacy of new targeting approaches, as you say we first wanted to ensure we could produce a reliable behavioral effect on the numerical Stroop task using TMS. Consequently, we decided to rewrite our manuscript so that the framing reflects the scope and methodology of our research.

The new title is: “Lessons learned from an fMRI-guided rTMS study on performance in a numerical Stroop task.”

The abstract now reads: “The Stroop task is a well-established tool to investigate the influence of competing visual categories on decision making. Neuroimaging as well as rTMS studies have demonstrated the involvement of parietal structures, particularly the intraparietal sulcus (IPS), in this task. Given its reliability, the numerical Stroop task was used to compare the effects of different TMS targeting approaches by Sack and colleagues, who elegantly demonstrated the superiority of individualized fMRI targeting. We performed the present study to test whether fMRI-guided rTMS effects on numerical Stroop task performance could still be observed while using more advanced techniques that have emerged in the last decade (e.g., electrical sham, robotic coil holder system, etc). To do so we used a traditional reaction time analysis and we performed, post-hoc, a more advanced comprehensive drift diffusion modeling approach. Fifteen participants performed the numerical Stroop task while active or sham 10 Hz rTMS was applied over the region of the right intraparietal sulcus (IPS) showing the strongest functional activation in the Incongruent Congruent contrast. This target was determined based on individualized fMRI data collected during a separate session. Contrary to our assumption, the classical reaction time analysis did not show any superiority of active rTMS over sham, probably due to confounds such as potential cumulative rTMS effects, and the effect of practice. However, the modeling approach revealed a robust effect of rTMS on the drift rate variable, suggesting differential processing of congruent and incongruent properties in perceptual decision-making, and, more generally, illustrating that more advanced computational analysis of performance data can elucidate the effects of rTMS on the brain where simpler methods may not.” (PAGE 2)

We modified the introduction as follows: “Sack et al. (2009) concluded that the most effective approach was individualized targeting based on the task-related fMRI. This was an important result for the brain stimulation field in general, as one of the main moderators of rTMS efficacy is the targeting approach (Beynel et al., 2019). More than a decade after this elegant demonstration of the importance of individualized fMRI targeting, experimental TMS methodologies have advanced, with for example the development of robotic coil holders that allow for more precise and stable positioning of the coil relative to the stimulation target, and an electrical sham technique that mimics the TMS-induced sensations, and thus potentially better blinds subjects to experimental conditions. In the present study, we tested whether the addition of these techniques combined with an individualized fMRI-based targeting approach endorsed by Sack and colleagues could improve the rTMS effect on numerical Stroop performance, with the expectation based on Cohen Kadosh et al. (2007) and Sack et al. (2009) that rTMS applied over the IPS would decrease the interference of incongruent stimuli and thus lessen of the reaction time slowing in that condition. Additionally, contrary to the prior study from Sack et al., participants completed the task multiple times preceding rTMS application (one block of trials before the MRI session, 4 blocks during MRI, and one block in the TMS session prior to TMS), to stabilize performance and minimize variability due to practice effects. This approach enabled us to probe the influence of task practice on behavioral performance and fMRI activations. (PAGE: 5-6)

2. The authors seem to suggest that the failed replication of the behavioral effect using fMRI-guided TMS casts doubt on the localization / TMS targeting effect. I think this conclusion is very questionable. The main gist and focus of our study in 2009 were to compare the relative differences between different TMS targeting approaches. It was not meant to claim that you need only 5 subjects or 15 subjects or whatever number of subjects to find a significant behavioral finding in the numerical stroop test when using this or that targeting approach. The design and overall extremely low sample size was completely unsuited for that (5 subjects, between subject design). We were also not interested in the numerical stroop task per se. This task just served as a tool, a means to compare the performance differences induced by the various TMS targeting approaches we compared with each other, showing relative differences. The numerical stroop task we used was brought to us by our visiting researcher, Dr. Roi Cohen Kadosh, who was PI of a research line using this task in the fMRI. In related publications, e.g. Cohen Kadosh 2007, this team reported right parietal activity changed during the execution of this numerical stroop task and behavioral effects when stimulating those individual right parietal activity clusters using TMS. I understand that the authors originally wanted to replicate our targeting approach study but got stopped at step 1, when hoping to first find the expected behavioral changes when using the supposedly best targeting approach: individual fMRI-guided TMS. Since they could not find any behavioral effects here, they now publish this still in the framing of an intended and failed replication of our TMS targeting study. But in fact, they failed to find the behavioral effects using one targeting approach. This is something completely different. In fact, when looking at their results in more detail, the issue of replicating previous work already starts with the fMRI data itself. Unlike previous work by Cohen Kadosh and others, the fMRI data of the current does not demonstrate clear individual right parietal activity clusters to underlie the execution of this task. In fact, in many cases, their fMRI data, if really used to localize the individual hotspot of this task for TMS targeting, would have suggested areas in the left parietal cortex. So, ironically, because they aimed to replicate our study of 2009, they wanted to stick to the right IPS (because this is where we fund our fMRI hotspot in each participant for this task), but by doing so, they actually violated the entire principle of individual fMRI-guided TMS targeting which bases the TMS positioning on the individual fMRI hot

Attachment

Submitted filename: Rebuttal_Letter.docx

pone.0302660.s004.docx (83.6KB, docx)

Decision Letter 1

Desmond J Oathes

10 Apr 2024

Lessons learned from an fMRI-guided rTMS study on performance in a numerical Stroop task

PONE-D-23-36866R1

Dear Dr. Beynel,

Congratulations, guys!

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Desmond J. Oathes

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #4: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors did a very good job in addressing my previous comments.

I just want to share some additional thoughts I have regarding their assumption that selecting the right IPS as an individual target for fMRI-guided TMS is the only logical choice here based on previous work (Cohen kadosh and sack et al) , and thus independent of the actual individual fMRI data collected in the present study ( apriori focus on right IPS).

The failed replication of this right IPS dominance (in fMRI already is importnat to acknowledge. But the manuscript still focusses very much on the replication attempt of the behavioural effects when stimulating right IPS during the numerical stroop task. it assumes that everything is according to plan up until the lack of finding / replicating this behavioural effects when following the procedure as described in Sack et al (1).

But the actual difference (or problem) already starts with the fMRI data and thus before the actual TMS study. Unlike Cohen Kadosh, the current fMRI data reveals something different, namely an initial bilateral, right dominant IPS activation during the task, followed by a practise-dependent shift / reduction of the right IPS activation. This means that in the actual TMS study, many of the tested subjects showed no right IPS activation during task execution (maybe because of practise , maybe because the right IPS functional dominance is in itself not a reliable and robust finding).

Before testing the reliability and replicability of TMS target approaches (ultimate goal of this study), and even before testing the replicability of behavioural effects of right IPS TMS in numerical scoop task (as framed now), one may just as well question and test the replicability of the assumed right parietal functional activation during task execution.

I am no expert in this field of research but if Cohen-Kadosh found bilateral parietal fMRI activation in their sample, with the TMS effect only observed when the right hemisphere was stimulated, isn't the first and maybe most interesting aspect in the current study that already at this level, the replication shows a much more nuanced picture? Maybe there is no replicable or robust right IPS dominance here, OR there is but only at the novice level of task execution , and with practise this right IPS activity disappears (is shifted to other regions, less needed, etc). This would be in line with similar research on the test retest reliability of BOLD fMRI). And this would elegantly explain the lack of TMS effects when still sticking to an a priori right IPS hotspot for individual TMS targeting even in the absence of an individual fMRI activation at the moment of the TMS.

In his context I really enjoyed this extra analyses " Further, we performed an additional analysis comparing activation in the first two fMRI blocks recorded during the task with the second two blocks, and demonstrated there was a dramatic drop in functional activation from an initially significant right IPS activation. This decrease could have resulted from practice, learning, and/or fatigue, and led to a non-significant activation when combining all 4 blocks. As this provides some insight into targeting issues that may have hindered us from finding an rTMS effect, we have included this additional analysis in the manuscript"

For me this deserves more attention as this indeed relates to the heart of the ultimate goal of individualised targeting. If an fMRI activation changes dynamically with practise (as is the case here), how do I select my individual hotspot for TMS targeting then? I think just following what previous studies have found even though the fMRI data is different (for what reason ever, practise, age, pathology) may not be the best choice. Please note that Cohen Kadosh and sack cannot count as 2 independent validations of right IPS activation using fMRI / TMS as these studies were not independent from each other.

This is what I meant when I suggested to rather target the individual fMRI hotspots of this task in this sample and at this level of practise, even if it differs from the right IPS.

Best wishes,

Alex

Reviewer #2: The authors did a good job responding to my and other Reviewer's commentary. This represents an important contribution to our developing field.

Reviewer #4: The authors have addressed my comments, and have revised the manuscript so that it is now a direct reflection of what the results show, and can impart to the field as a result.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Alexander T. Sack

Reviewer #2: No

Reviewer #4: No

**********

Acceptance letter

Desmond J Oathes

24 Apr 2024

PONE-D-23-36866R1

PLOS ONE

Dear Dr. Beynel,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Desmond J. Oathes

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Table. Summary of stimulation target location and stimulation intensity for each participant.

    Stimulation coordinates are in LPI coordinates system, t-scores at the stimulated target, stimulation intensity expressed as a percentage of the resting motor threshold, Euclidean distance to Sack et al. average target, and scalp-to-cortex distance for each participant. Highlighted in blue are the lowest values and in red the highest values. Participant is italics font was the outlier not included in the analyses.

    (DOCX)

    pone.0302660.s001.docx (15.8KB, docx)
    S2 Table. Overlap between activated clusters in the Incongruent > Congruent contrast and their overlap with the Glasser atlas.

    (DOCX)

    pone.0302660.s002.docx (15.1KB, docx)
    Attachment

    Submitted filename: PLOS_One_Review.docx

    pone.0302660.s003.docx (15.1KB, docx)
    Attachment

    Submitted filename: Rebuttal_Letter.docx

    pone.0302660.s004.docx (83.6KB, docx)

    Data Availability Statement

    Behavioral data are available from the Figshare database: https://figshare.com/articles/dataset/NumericalStroopTMS_xlsx/25130669.


    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES