Reporting matters: Brain mapping with transcranial magnetic stimulation

1.

Transcranial magnetic stimulation (TMS) allows researchers to noninvasively probe the human brain. In a recent issue of Human Brain Mapping, Massé‐Alarie, Bergin, Schneider, Schabrun, and Hodges (2017) used this technique to investigate the task‐specific organization of the primary motor cortex for control of human forearm muscles. Specifically, TMS was used to create cortical topographical maps of four forearm muscles at rest, and, in one of these muscles, during isometric wrist extension and isometric grip. The authors were interested in how these maps differ between muscles, how they overlap, and how they change with different motor tasks. Key to their approach was the use of indwelling fine‐wire electrodes to record motor evoked potentials elicited by magnetic stimulation, which revealed the size of cortical maps is grossly overestimated when evoked potentials are recorded from electrodes placed on the skin surface.

In their paper, Massé‐Alarie et al. (2017) set (statistical) significance at p < 0.05. Yet the authors interpret several p values above this threshold as statistical trends. Post hoc analyses were even performed for main effects that were not statistically significant:

“Although narrowly missing significance, there was a tendency for a main effect of Pairs of tasks for percentage of MEP peak overlap (F _{(1, 13)} = 3.30; p = 0.053), which was explained by a tendency towards a greater percentage peak overlap in Rest‐Ext (32.1 ± 9.6%) than Rest‐Grip (14.3 ± 8.2%; p = 0.02); Fig. 4B), and by a tendency toward a greater percent peak overlap for Grip‐Ext (26.8 ± 9.7%) compared to Rest‐Grip (p = 0.09).” p. 6126

“Finally, although nonsignificant, there was a tendency toward a main effect for muscle pairs for percentage of MEP peak overlap (F_{(2, 18)} = 3.23; p = 0.06). This was explained by a tendency toward a larger overlap between ECRB_fw‐surf (43.5 ± 8.7%) than ECRB_fw‐EDC (25.2 ± 7.5%; post hoc p = 0.03; Fig. 6C).” p. 6126‐6127

The authors later discuss these effects as if they were statistically significant:

“The findings that peak overlap was larger between ECRB_surf and ECRB_fw than for ECRB_fw and EDC implies that […].” p. 6130

Implicit to this type of interpretation is that statistical trends reflect real effects. However, additional data are more likely than not to turn a trend into a nonsignificant result (Wood, Freemantle, King, & Nazareth, 2014). Is this a problem? In this instance the spin is blatant and an astute reader can draw their own conclusions. However, spin in various forms is so common (Bero, 2018; Chiu, Grundy, & Bero, 2017; Héroux, 2016) that readers and reviewers may be wooed by such biased interpretations. Why do I say biased? The authors are lenient only in one direction. In their paper, Massé‐Alarie et al. (2017) report 11 p values that fall between 0.02 and 0.05. Why are these not reported as tending toward nonsignificance? Regardless of whether the p values were just above or just below the threshold of p = 0.05, an exact replication (i.e., same sample size and methods) of this study only has a 50% chance of reproducing these statistically significant (or near significant) effects (Button et al., 2013; Forstmeier, Wagenmakers, & Parker, 2017). As recently pointed out, p values are fickle (Cumming, 2014; Halsey, Curran‐Everett, Vowler, & Drummond, 2015), especially when sample size is small (Button et al., 2013; Higginson & Manufó, 2016). Thus, how confident should we be about the results of Massé‐Alarie et al. (2017)? Looking back, how confident should we be about our own work? This type of nuanced view is not common. But we need more of it; especially towards noninvasive brain stimulation where many published effects are simply not reproducible (Héroux, Taylor & Gandevia, 2015; Héroux, Loo, Taylor, & Gandevia, 2017).

Another reporting matter in the paper by Massé‐Alarie et al. (2017) is the all‐to‐common use of the standard error of the mean (SEM) to summarize data variability (Héroux, 2016; Héroux et al., 2017; Weissgerber, Milic, Winham, & Garovic, 2015). This is not what the SEM quantifies. But does it actually matter what measure is reported? Experts think so (Curran‐Everett & Benos, 2004), as do I. Here are some of the above results reported with standard deviations:

“[…] which was explained by a tendency towards a greater percentage peak overlap in Rest‐Ext (32.1 ± 35.9%) than Rest‐Grip (14.3 ± 31.1%; p = 0.02); Fig. 4B), and by a tendency toward a greater percent peak overlap for Grip‐Ext (26.8 ± 36.3%) compared to Rest‐Grip (p = 0.09).”

Given that percentages are bound between 0 and 100, what does 14.3 ± 31.1% actually mean? What does the underlying data look like? Reporting results with standard deviations or other appropriate measures of variability does not affect statistical tests—a significant result will remain a significant result—so let us not be afraid of them. Reporting results with standard deviations or other appropriate measures of variability provides the reader with a better sense of the underlying data, which is important to appropriately interpret study results and figures (Belia, Fidler, Williams, & Cumming, 2005; Curran‐Everett & Benos, 2004; Drummond & Vowler, 2011).

Exploratory research is important to identify new avenues of research and test new hypotheses, and the paper by Massé‐Alarie et al. (2017) raises many interesting questions on how the human primary motor cortex is organized. Nevertheless, I encourage the authors and others in the field to be mindful when reporting and interpreting study results, especially when sample sizes are relatively small. Let us heed the advice of experts and, as a field, strive toward publishing research that is less biased, and more reproducible and transparent.

Héroux ME. Reporting matters: Brain mapping with transcranial magnetic stimulation. Hum Brain Mapp. 2019;40:352–353. 10.1002/hbm.24371