Meta-analysis of biological variables’ impact on spinal motoneuron electrophysiology data

Abstract

Experimental, methodological, and biological variables must be accounted for statistically to maximize accuracy and comparability of published neuroscience data. However, accounting for all variables is nigh impossible. Thus we aimed to identify particularly influential variables within published neurological data, from cat, rat, and mouse studies, via a robust statistical process. Our goal was to develop tools to improve rigor in the collection and analysis of data. We strictly constrained experimental and methodological variables and then assessed four key biological variables within motoneuron research: species, age, sex, and cell type. We quantified intraexperimental and interexperimental variances in 11 commonly reported electrophysiological properties of spinal motoneurons. We first assessed variances without accounting for biological variables and then reassessed them while accounting for all four variables. We next assessed variances with all possible combinations of these four variables. We concluded that some motoneuron properties have low intraexperimental, but high interexperimental, variance; that individual motoneuron properties are impacted differently by biological variables; and that some unexplained variances still remain. We report here the optimal combinations of biological variables to reduce interexperimental variance for all 11 parameters. We also rank each parameter by intra- and interexperimental consistency. We expect these results to assist with design of experimental and analytical methods, and to support accuracy in simulations. Furthermore, although demonstrated on spinal motoneuron electrophysiology literature, our approach is applicable to biological data from all fields of neuroscience. This approach represents an important aid to experimental design, comparison of reported data, and reduction of unexplained variance in neuroscience data.

NEW & NOTEWORTHY Our meta-analysis shows the impact of species, age, sex, and cell type on lumbosacral motoneuron electrophysiological properties by thoroughly quantifying variances across literature for the first time. We quantify the variances of 11 motoneuron properties with consideration of biological variables, thus providing specific insights for motoneuron modelers and experimenters, and providing a general methodological template for the quantification of variance in neurological data with the consideration of any experimental, methodological, or biological variables of interest.

INTRODUCTION

Scientific experiments are designed to test hypotheses. Such design requires decisions that balance scientific, ethical, and budgetary requirements. These decisions include the consideration of broad experimental variables, specific methodological variables, and relevant biological variables. The variables researchers choose to address are critical to how explicitly their data can accept or refute the hypothesis, how many experimental groups are necessary, what level of precision must be used, and how much time and resources the experiment requires. The meta-analysis method herein quantifies variances in reported spinal motoneuron electrophysiological data and provides a template for deducing the appropriate inclusion/exclusion of experimental, methodological, and biological variables.

The first intracellular recordings in the nervous system were performed on spinal motoneurons (Brock et al. 1951, 1952a, 1952b; Woodbury and Patton 1952a, 1952b); thus electrophysiological data on spinal motoneurons are sufficiently available for statistical analysis. Also, the often highly deviating reported values make this data particularly interesting for our study. Motoneurons control motor function; thus motoneuron research is vital to understanding, characterizing, and replicating the way we move. This field is highly relevant to spinal cord injury, prosthetics, motor degenerative diseases, physical fitness, and aging. In this study, we focused on lumbosacral motoneurons, which primarily control hindlimb, tail, bladder, and bowel functions. Electrical properties determine the input/output function of motoneurons and, ultimately, the resulting motor command. Thus motoneurons are often categorized by their electrical properties and morphology. The smallest, slow (S-type) motoneurons are activated at low electrical thresholds and sustain prolonged and low-force muscle contractions, used for behaviors such as balance and posture. Larger, faster (F-type) motoneurons are activated at higher electrical thresholds and produce larger forces for shorter duration muscle contractions, used for behaviors such as sprinting and jumping. Furthermore, F-type motoneurons are sometimes subcategorized as fast, fatigable (FF), fast, intermediate (FI), and fast, fatigue resistant (FR). Motor function level also differs with age, between sexes, and among species. Therefore, motoneuron cell type (S vs. FR vs. FI vs. FF vs. F vs. nonfiring), age (neonatal vs. adult), sex (male vs. female vs. combined), and species (cat vs. rat vs. mouse) are four key biological variables that may affect electrophysiological data in motoneuron research. Thus we analyzed the effects of these four biological variables on the variance of the following commonly reported motoneuron electrical properties: input resistance (R_in), input conductance (G_in), rheobase, resting membrane potential (RMP), action potential amplitude (AP_amp), action potential duration (AP_dur), afterhyperpolarization amplitude (AHP_amp), AHP duration (AHP_dur), frequency-current gain (gain), time constant (tau), and membrane capacitance (C_m). These 11 electrical parameters were selected because they are commonly measured and reported experimentally, are often considered for replication in modeling, and are the fundamental parameters used to characterize basic motoneuron function.

Motoneuron electrophysiological experiments performed to measure these electrical parameters are very complex, with many potential sources of experimental variance, such as level of oxygenation, solution composition, pipette resistance, and pipette shape. Thus an analysis of variances in reported electrical properties both within and between studies can elucidate which variables must be controlled to ensure accurate, comparable, and unbiased measurements. We focused on a single animal preparation and specifically considered the effects of the four biological variables that we hypothesized to be large sources of variance. We assessed both their contribution to the variances and how much variance remained after we accounted for them. Our results show that the biological variables affect each parameter differently. Our results can also inform future studies as to which variables are relevant to specific experimental or modeling conditions. However, additional efforts must be made to determine additional sources of variance. Consistency and robustness among motoneuron data will improve comparability between experiments and help modelers replicate true motoneuron function more accurately. While our specific conclusions apply directly to motoneuron research, our approach and its purpose can apply to any area of neuroscience.

MATERIALS AND METHODS

Protocol and Registration

This review adheres to the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) standards. The review protocol was not registered but is thoroughly described herein. To minimize selection bias, studies for inclusion in this review were selected from a predefined, broad literature search followed by the application of a specific set of predefined inclusion/exclusion criteria as described henceforth. The search and selection process is illustrated in Fig. 1 and is based on the 2009 PRISMA flow diagram. Collected summary data and the corresponding biological factors can be found at https://doi.org/10.5281/zenodo.3720017.

Fig. 1.

Flow diagram of the identification, screening, and inclusion processes. Our original methodical search process (see materials and methods) returned 5,245 results. An alternative search process with the help of a library scientist (see materials and methods) returned 64 results. After duplicates were removed, we screened 5,283 studies for inclusion based on the title, abstract, or methods and were left with 331 studies. We examined the full text of these 331 studies and ultimately found 105 studies meeting our inclusion criteria and containing relevant data.

Data Collection

Information sources/eligibility and exclusion criteria.

For consistency, objectivity, and repeatability, a series of broad, methodical PubMed searches were performed between April 2018 and August 2018 with combinations of the following words:

• 1) Cat/rat/mouse,
• 2) Motoneuron, and
• 3) Input resistance/input conductance/rheobase/resting membrane potential/AHP.

For example, the first search was “cat motoneuron input resistance,” followed by “cat motoneuron input conductance,” etc. These searches returned a combined total of 5,245 articles to be screened for inclusion in this review. We also performed an independent, more detailed search with the aid of a library scientist which included alternative spellings of “motoneuron,” all 11 parameters of interest, and key terms such as “sharp,” “intracellular,” “lumbosacral,” etc. This search found 64 articles to be screened for review, 26 of which were duplicate results from the original searches. All search results were then filtered manually to exclude any studies out of our predefined scope, as based on information in the title, abstract, or methods. This resulted in 331 studies whose full text was then analyzed to verify inclusion criteria and to verify that the necessary summary data was reported. More specifically, we only included studies that used sharp electrodes to measure from lumbosacral spinal motoneurons in the cat, rat, or mouse. Only wildtype or control group data were recorded. Both in vivo and in vitro studies were included, provided that the tissue was not cultured. For each selected study, the species, age, sex, and motoneuron type were recorded if reported; then the mean, SD/SE, and sample size were recorded manually from each study when all three were reported for any of the 11 selected electrical properties. The collected means and SDs from each study are shown for each parameter in Fig. 2, along with the corresponding interexperimental consistency scores (see materials and methods, Interexperimental Consistency Score). The number of means found for each parameter can be found in Table 1. The data came from the following 105 studies: Bączyk et al. 2013; Bakels and Kernell 1993a, 1993b; Baranauskas and Nistri 1995; Baranauskas et al. 1995; Barrett and Crill 1974; Beaumont and Gardiner 2002; Beaumont et al. 2004; Beaumont et al. 2008; Bertrand and Cazalets 1998; Bories et al. 2007; Boyce et al. 2012; Bracci et al. 1996; Burke et al. 1982; Button et al. 2007, 2008; Carlen et al. 1980; Carp et al. 2008, 2010; Cazalets et al. 1996; Chase et al. 1985; Chopek et al. 2013; Cormery et al. 2000, 2005; Cotel et al. 2009; Deardorff et al. 2013; Delestrée et al. 2014; Dum and Kennedy 1980; Elbasiouny et al. 2010; Elliott and Wallis 1992, 1993; Engelhardt and Chase 1992; Fisher et al. 1994; Fisher and Nistri 1993; Flatman et al. 1982; Fleshman et al. 1981; Foehring et al. 1986, 1987; Forsythe and Redman 1988; Fulton and Walton 1986; Gardiner and Seburn 1997; Gonzalez and Collins 1997; Gustafsson et al. 1982; Gustafsson and Pinter 1984; Hall 1982; Heckman and Binder 1988; 1991; Heckman et al. 1994; Hochman et al. 1991; Huh et al. 2017; Huizar et al. 1975; Jiang and Heckman 2006; Jiang et al. 1991; Kernell 1966; Kernell and Zwaagstra 1981; Kjaerulff and Kiehn 1997; Kohlmeier et al. 1996; Krawitz et al. 2001; Krnjević et al. 1978; Krutki et al. 2014, 2015, 2017; Kuno et al. 1974; Liu et al. 1995; Manuel and Heckman 2011; Manuel et al. 2005, 2009; Marchetti et al. 2001, 2002; Martínez-Silva et al. 2018; Meehan et al. 2010; Mentis et al. 2007; Morales and Chase 1981; Morales et al. 1987a, 1987b, 1987c; Morris et al. 1985; Mrówczyński et al. 2010; Munson et al. 1986; Ostroumov et al. 2011; Paroschy and Shefchyk 2000; Perreault 2002; Petruska et al. 2007; Pflieger et al. 2002; Pinter et al. 1983; Pinter and Vanden Noven 1989; Powers et al. 2000; Rivera-Arconada and Lopez-Garcia 2005; Rotterman et al. 2014; Sasaki 1991; Seebach et al. 1999; Seebach and Mendell 1996; Turkanis and Karler 1986; Turkin et al. 2010; Ulfhake and Kellerth 1984; Vinay et al. 2000; Vinay and Clarac 1999; Wu et al. 2012; Xi et al. 1997; Yamuy et al. 2004; Zengel et al. 1983, 1985; Zhang and Krnjević 1987, 1988; Ziskind-Conhaim 1988).

Fig. 2.

Summary statistics from literature. Reported means are spread along the x-axes from smallest to largest. The y-axes show the reported values with relevant units and standard deviation (SD) bars. Interexperimental consistency scores are stated for each graph in inset boxes and can be interpreted visually by the vertical spread of values on each graph. A parameter with high interexperimental consistency score (such as resting membrane potential, RMP) would indicate that a large proportion of the means of that parameter from different studies are spread within a small range of values (i.e., a small vertical spread in y-values). A parameter with low interexperimental consistency score (such as input conductance, G_in) would indicate that a large proportion of the means of that parameter from different studies are spread within a large range of values (i.e., a large vertical spread in y-values). AHP_amp, afterhyperpolarization amplitude; AHP_dur, afterhyperpolarization duration; AP_amp, action potential amplitude; AP_dur, action potential duration; C_m, membrane capacitance; gain, frequency-current gain; R_in, input resistance; tau, time constant.

Table 1.

Optimal biological variables and numbers of means

Parameter	Best Variable Combination	Not Considered Due to Absence of Data	Nmeans (CoV < 0.5)	Nmeans (All)
Rin	ATc		91	146
Gin	SAX	Tc	11	13
Rheobase	SATc		55	112
RMP	XTc		101	102
APamp	Tc		71	72
APdur	S or X		7	9
AHPamp	XTg		50	76
AHPdur	SATg		48	49
Gain	SA	Tc	23	38
Tau	AXTc		40	45
Cm	X	Tg, Tc	6	7

Effects of 4 key biological variables were considered: species (S), age (A), sex (X), and cell type (T); cell typing methods for grouped (Tg) and categorized (Tc) fast motoneurons are indicated. The best combination of variables is that which results in the maximum consistency score out of every possible combination. Variables that were not able to be considered for each parameter due to lack of data in the variable groups are listed. Means used to form Table 1 were only those with a reliable coefficient of variance (CoV); the number of means used for each parameter is indicated as N_means (CoV < 0.05). The total number of means recorded for each parameter (regardless of CoV) is also indicated, as N_means (All). AHP_amp, afterhyperpolarization amplitude; AHP_dur, afterhyperpolarization duration; AP_amp, action potential amplitude; AP_dur, action potential duration; C_m, membrane capacitance; gain, frequency-current gain; G_in, input conductance; R_in, input resistance; RMP, resting membrane potential; tau, time constant.

Risk of bias in individual studies.

Possible bias within studies was not an exclusion criterion when studies were selected for review, but the effects of potential biases are reduced by analyzing only control data from studies with a wide range of treatment/disease/injury groups. We noted for each selected study whether biases were addressed or blinding was conducted.

Risk of bias across studies.

We also accounted for bias across studies by considering the number of studies that did or did not report relevant biological variables (see results).

While great care was taken for accuracy and thoroughness in the described data collection process, we do not claim to have found every applicable study, nor do we claim to have found every reported relevant biological variable. Such completeness is unlikely, due to the wide variety of manuscript styles and organization of methods. However, the sample size we have collected is large enough to accurately represent the current body of literature on motoneurons within the given scope. Nonetheless, conclusions drawn from underreported parameters in the literature (such as C_m, G_in, and AP_dur) should be interpreted with caution.

Intraexperimental Consistency Score

Intraexperimental consistency was based on standard deviation (SD). When a selected study reported standard error (SE), SD was calculated according to Eq. 1:

$$SD=SE*\sqrt{\left|mean\right|}$$ (1)

The coefficient of variation (CoV) was next calculated according to the following equation:

$$CoV=SD/\left|mean\right|$$ (2)

A high CoV (in which the standard deviation is large relative to the mean) indicates a statistically unreliable mean. What value constitutes a high CoV depends on the specific data set and is subject to interpretation. Our upper quartile of all CoVs was ~0.5, so we considered this our minimum high CoV. Thus the intraexperimental consistency score for a given parameter was calculated as the percentage of studies that have a CoV < 0.5 for that parameter, as shown in Eq. 3:

$$N_{CoV<0.5}/N_{total}$$ (3)

Simulation of Raw Data for Tukey’s Statistical Analysis

A t test could compare one study’s reported value of a given parameter to that of another study using only the summary statistics (i.e., reported mean, reported standard deviation, and reported n value). However, Tukey’s comparisons are preferred in this kind of analysis. Tukey’s test assesses the overall variance of the parameter among all studies rather than considering only the variance between two studies compared at a given time, thus lowering the type 1 error (Bewick et al. 2004). Tukey’s test requires the raw data from which the summary statistics were derived for all studies being compared. However, we only had the published summary statistics (i.e., reported mean, reported standard deviation, and reported n value). Therefore, we simulated the raw data for each parameter using a built-in MATLAB function. This generated normally distributed data sets from each collected summary statistic. Specifically, we generated a raw data set with the reported sample size centered and scattered around the reported mean in a manner similar to the reported standard deviation, using the following equation:

$$RawDATA=mean+SD*randn\left(1,n\right)$$ (4)

The error between the published mean and the mean of the generated data was then calculated using the following equation:

$$\%erro{r}_{mean}=\left|mea{n}_{generated}-mea{n}_{reported}\right|/mea{n}_{reported}\times 100$$ (5)

Similarly, the error between the published SD and the SD of the generated data was calculated using the following equation:

$$\%erro{r}_{SD}=\left|S{D}_{generated}-S{D}_{reported}\right|/S{D}_{reported}\times 100$$ (6)

A generated raw data set was only accepted when the combined percent error in mean and SD was less than 1%, as follows:

$$\%erro{r}_{mean}+\%erro{r}_{SD}\le 1\%$$ (7)

In this way, we were able to randomly generate raw data sets for each parameter with means, SD, and sample sizes that accurately represented the reported summary statistics in literature, thereby allowing us to run Tukey’s tests to compare parameter values across studies.

Interexperimental Consistency Score

To calculate an interexperimental consistency score for a given parameter, Tukey’s comparisons were conducted among the simulated raw data of all studies. Given that the Tukey’s test returns a P value for every comparison of two studies, the interexperimental consistency score for each parameter was calculated as the percentage of study comparisons that are not different at an α of 0.05, as follows:

$$(\mathit{\text{no. of comparisons with}}\hspace{0.17em}P>0.05)/\left(\mathit{\text{total no. of comparisons}}\right)$$ (8)

The collected means and SDs from each study are shown for each parameter in Fig. 2, along with the corresponding interexperimental consistency scores (consistency scores calculated by a single Tukey’s test for each parameter with no consideration of biological variables).

Biological Variable Groups Analyzed via Tukey’s Tests

In this step of the analysis, we grouped the data sets by all possible combinations of biological variables (Table 2) and recalculated the interexperimental consistency scores. These combinations of biological variables were used to perform several smaller Tukey’s tests, in which only data sets of like biological variables were compared. Species was grouped into mouse, rat, or cat. Age was split as either neonatal (age ≤ postnatal day 20) or adult. Sex was categorized as male, female, combination of male and female, or not reported. If a study reported that both male and female animals were used (and did not report separate means for male and female), then those means were placed in the “combined” sex group. Cell type was grouped two different ways: 1) S, F, nonfiring, and not reported (NR) and 2) S, FR, FI, FF, F, nonfiring, and NR. These typing methods were abbreviated Tg and Tc, respectively, for cell type with grouped F and cell type with categorized F. While it is very useful to sort motoneurons into these types, there is a great deal of overlap in their electrical properties, rather than clear, discrete borders between types. Therefore, we only included typing if it was explicitly stated. Although conservative, this approach prioritizes accuracy.

Table 2.

Biological variable combinations

No Groups	SX	SAT
S	ST	SXT
A	AX	AXT
X	AT	SAXT
T	XT
SA	SAX

A, age; S, species; T, cell type; X, sex.

RESULTS

Data Collection

Risk of bias within studies.

Of the 105 studies included (see materials and methods, Data Collection), 16 studies addressed possible sampling bias and only 3 studies mentioned experimenter blinding in any way.

Risk of bias across studies.

Additionally, 67 studies did not report sex used, 27 studies did not report age used, and 75 studies did not report cell type examined. Reporting of relevant biological variables is key to ensuring unbiased comparisons of data across the literature. Thus, while we aim to determine which biological variables impact parameter variances, our results may be biased by lack of reporting.

Risk of bias in this meta-analysis.

It should be noted that G_in, AP_dur, and C_m have a relatively small number of reported means found for analysis (N = 13, 9, and 7, respectively; see Table 1). Therefore, conclusions drawn for these parameters should be interpreted with caution. Nonetheless, we focused on the effects of the four selected key biological variables by selecting a single methodological and experimental protocol for analysis (see materials and methods). To visualize which parameters have higher or lower consistencies relative to other parameters and how interexperimental consistencies change when accounting for biological variables, we reported the intra- and interexperimental consistency scores by ranking the 11 parameters from most to least consistent, and then explored the effects of the biological variables on interexperimental consistencies.

Intraexperimental Variability

For a given parameter, the percentage of studies that have CoV <0.5 was used as the intraexperimental consistency score (see materials and methods). These consistency scores allowed us to rank the 11 electrical parameters from most to least consistent (Fig. 3A). Our analysis shows that these motoneuron electrical parameters have a wide range of intraexperimental consistency in reported literature. There are three electrical parameters with very high intraexperimental consistency scores (scores > 0.95): RMP, AP_amp, and AHP_dur. This indicates that experimental mean values of these parameters do not exhibit large variations within a given study group; i.e., they rarely have excessively high standard deviations relative to their means. Conversely, rheobase appears to have the lowest intraexperimental consistency score (score < 0.5), indicating that large variations in the values of this parameter frequently occur within a given study group. Additionally, gain and R_in also have relatively low intraexperimental consistency scores (scores < 0.7).

Fig. 3.

Parameters ranked by consistency score with no consideration of biological variables. A: intraexperimental consistency, indicating the portion of studies with coefficient of variance (CoV) < 0.5, is ranked from high to low. B: interexperimental consistency rank. Interexperimental consistency scores are a measure of the portion of studies that are not statistically significantly different from each other (P > 0.05). The parameters are ordered on the y-axis according to their rank when all data are considered. This rank corresponds to the black bars and to the scores given in Fig. 2. Removing high-CoV studies slightly changes interexperimental consistency scores, which are shown by the gray bars (the new rank of scores represented by gray bars is given by gray numbers to the left of each parameter listed on the y-axis). AHP_amp, afterhyperpolarization amplitude; AHP_dur, afterhyperpolarization duration; AP_amp, action potential amplitude; AP_dur, action potential duration; C_m, membrane capacitance; gain, frequency-current gain; G_in, input conductance; R_in, input resistance; RMP, resting membrane potential; tau, time constant.

Interexperimental Variability

Because the Tukey’s test returns a P value for every comparison of two studies, the interexperimental consistency score for each parameter was calculated as the percentage of study comparisons that are not different at an α of 0.05 (Eq. 8; see materials and methods). The interexperimental consistency scores were then used to rank the 11 electrical parameters from most to least consistent (Fig. 3B). Interestingly, our analysis shows that motoneuron electrical parameters also have a wide range of interexperimental consistency. Furthermore, some parameters with high intraexperimental consistency scores had low interexperimental consistency scores. For instance, G_in, C_m, and AHP_dur have intraexperimental consistency scores > 0.8 but interexperimental consistency scores < 0.4. This indicates that while these parameters are consistent within studies, their average values vary considerably among studies. This could be because these parameters are biologically stable (intraexperimentally consistent) but sensitive to variations in methods of electrical measurement or statistical analysis (interexperimentally variable). Other parameters demonstrated more straightforward results: RMP, AP_amp, and tau have both high intra- and interexperimental consistency scores (compare Fig. 3, A and B).

Figure 3B (black bars) includes all collected data regardless of the CoV value. However, we posited that removing data with the most statistically unreliable means (CoV > 0.5) would provide a stricter assessment of interexperimental variability by eliminating data with substantial undetermined sources of intraexperimental variability that could confound the Tukey’s analysis for comparing means. This elimination is justified for examining interexperimental variance because biological factors only vary between experiments (not within one experiment). Thus eliminating high-CoV data prevents inflated interexperimental consistency scores due to high CoVs. In other words, a high interexperimental consistency score with high-CoV data included could indicate a highly consistent parameter across literature or a parameter with many high CoVs whose means, therefore, could not be statistically differentiated from other means due to intraexperimental variance. Thus removing high CoVs ensures that interexperimental consistency scores truly reflect variance between studies and are not biased by intraexperimental variance. As expected, removing data with CoV > 0.5 resulted in lower interexperimental consistency scores (in Fig. 3B, gray bars have lower scores than black bars). Because removing these studies lowered the consistency scores, the removed studies must have had means that were typically not statistically different from other means. Whether these means were statistically not different due to interexperimental factors or their high intraexperimental variance is undetectable. Therefore, statistically comparing high-CoV data would make our interexperimental analysis less statistically powerful by increasing type 2 error and could therefore bias the apparent effects of biological variables, which is of primary interest for our study.

Thus, after removal of data with CoV > 0.5, we re-ranked the parameters by interexperimental consistency in Fig. 3B (gray numbers left of y-axis). Note that rank is not greatly impacted by removing these data: rheobase, AHP_amp, R_in, and gain exchange places in the intermediate ranks. Because the removal of high-CoV data only slightly impacted the relative consistency of the parameters while eliminating the possibly confounding, unexplained intraexperimental variability, we will continue to only include data with CoV ≤ 0.5 for the remainder of the analysis. This more focused analysis gives the clearest picture of interexperimental variance and allows us to focus on the effects of the biological variables of interest.

Grouping by Biological Variables

In Figs. 2 and 3, we compared all data for each parameter with no consideration of biological variables. Given that properties of motoneurons are known to differ, like other neurons, by species and cell type, between sexes, and over age ranges, we hypothesized that accounting for these biological variables would provide a more accurate assessment of the interexperimental consistency of these parameters in literature. Thus we considered the effects of these four key biological variables: species, age, sex, and cell type. Figure 4A shows the effect on the interexperimental consistency ranking when we accounted for all four biological variables (see materials and methods). There were some minor changes in rank (compare gray number rank in Fig. 3B with top-down rank in Fig. 4A), but RMP, AP_dur, and tau remained in the top four; R_in and gain remained mid-ranked; and AHP_amp, AHP_dur, C_m, and G_in remained in the bottom five. If the four biological variables were responsible for most of the interexperimental variance, accounting for all of them should increase the consistency scores toward 1. However, we see in Fig. 4A that some parameters’ consistency scores increased (from gray bars to red bars), whereas others’ decreased. In fact, we found that accounting for fewer biological variables, as opposed to all four, increased the interexperimental consistency score for all parameters (Fig. 4B; all black bars are longer than red bars). This suggests that all four biological variables do not necessarily influence each cell parameter. In fact, by testing all possible combinations of biological variables (see combinations in materials and methods), we found that each parameter had its own optimal combination of biological variables that yielded the highest interexperimental consistency score (shown in Table 1 and Fig. 4B, black bars). In other words, the influence of biological variables is individual to each parameter. Again, minor changes were seen in the rank order of parameters when we used the best combination of biological variables (compare Fig. 4, A and B). RMP, AP_dur, rheobase, and tau remained in the top four (indicated in Fig. 4 with dark blue labels); R_in, gain, and AP_amp remained mid-rank (indicated in Fig. 4 with light blue labels); and AHP_amp, AHP_dur, C_m, and G_in remained in the bottom four (indicated in Fig. 4 with magenta labels).

Fig. 4.

Interexperimental consistency rankings only including data with coefficient of variance (CoV) < 0.5, with Tukey’s analysis applied to varying conditions of biological variables. Interexperimental consistency scores are a measure of the portion of studies that are not statistically significantly different from each other (P > 0.05). A: interexperimental consistency scores with no biological variables (gray bars) are compared with interexperimental consistency scores calculated by grouping all 4 biological variables of interest (species, age, sex, and cell type, SAXT; red bars) with Tukey’s test. Parameters are ranked on the y-axis by the grouped scores (red bars) from most to least consistent. B: interexperimental consistency scores are compared among scores with 1) no biological variables (gray bars), 2) all biological variables (red bars), and 3) optimized biological variables (black bars). Parameters are ranked on the y-axis by the optimized scores (black bars) from most to least consistent. Parameters labeled in dark blue are the top 4 most consistent for all interexperimental rankings without high CoV and when biological variables are considered (compare with A). Likewise, parameters labeled in light blue are always mid rank, and parameters labeled in magenta are always low rank. AHP_amp, afterhyperpolarization amplitude; AHP_dur, afterhyperpolarization duration; AP_amp, action potential amplitude; AP_dur, action potential duration; C_m, membrane capacitance; gain, frequency-current gain; G_in, input conductance; R_in, input resistance; RMP, resting membrane potential; tau, time constant.

Our analysis also shows, across the parameters examined, that all four biological variables commonly affect parameter consistency in literature, but no parameter needs all four variables accounted for to be most consistent. Table 1 should be interpreted with careful consideration of analysis limitations due to unevenly reported biological variables. Sometimes every variable combination could not be included for a given parameter because there was no reported data for comparison. For example, cell type is the least commonly included variable in the experimental design of motoneuron studies, and so it has the least available data to study its effects. Also, there is limited data for age analysis because there is little reported data on neonates for the selected animal preparation. Similarly, sex sometimes has limited data for analysis because it is commonly unreported in literature.

Although using the best combination of biological variables for each parameter increased the interexperimental consistency scores (Fig. 4B), there remained notable variance (scores indicated by many black bars in Fig. 4B are still well <1). This suggests there are additional elements underlying this variance.

DISCUSSION

The present study provides a novel, comprehensive meta-analysis to quantify variances in reported biological data in literature. The analysis was applied to lumbosacral motoneuron electrophysiological properties as a case study. However, our approach is general and is applicable to other areas of neuroscience. We show that some motoneuron properties have low intraexperimental but high interexperimental variance, and that individual motoneuron properties are affected differently by species, age, sex, and cell type. That is, each parameter is affected by its own set of key biological variables (see Table 1). Specifically, calculating interexperimental variance with consideration of each parameter’s optimal combination of biological variables resulted in higher consistency scores than calculating with all biological variables. Additionally, no one biological variable appeared to improve motoneuron electrical parameter consistencies more than the others (i.e., species improved 5 of the 11 parameters, age and sex improved 6 parameters, and cell type improved 7 parameters; Table 1). Importantly, even after the optimal combination of biological variables was accounted for, substantial interexperimental variance remained. Thus we conclude that there are still important, unidentified sources of variance, other than these biological variables, that must be explored.

Meta-Analysis of Summary Statistics: Assumptions and Limitations

One could argue that without access to the raw data and information on the data distribution, meta-analysis of summary statistics from literature could be prone to inaccuracies. Requesting raw data from the authors of 105 studies, some of which are as old as 1974, is not feasible, because a number of authors could no longer be reached. Therefore, we developed a simulation approach to replicate the raw data of a given parameter from the summary statistics with <1% error (see Eqs. 4–7 in materials and methods for detail). This allowed us to conduct paired Tukey’s comparisons among the simulated raw data of all studies to calculate the interexperimental consistency score for that parameter (computed as the percentage of study comparisons that are not statistically different; see Eq. 8). The only assumption of this simulation approach is that the raw data has normal distribution. To test the validity of this assumption, we have analyzed the 105 studies included in our analysis to infer information on the raw data distribution. Our analysis showed that 50 of the 105 papers have explicitly tested for and confirmed normality and/or used parametric statistics, indicating normality. Only 6 of the 105 studies reported using a nonparametric statistical test. The remaining studies either did not perform statistics on their reported mean data (31 of 105 papers) or did not specify what kind of test was performed (18 of 105 papers), making it difficult to infer information on data distribution. Additionally, from the 50 papers that explicitly confirmed normality and/or used parametric statistics, we found that all 11 parameters examined in our analysis have been either found to be or statistically treated as normally distributed by at least one or more of these papers. Together, this shows that the assumption of normality is valid for a large extent and is well supported by the literature.

Our approach simulates the reported mean and standard deviation values within 1% error, and the sample size is exact to the experimental value. Therefore, the statistics will treat the simulated data the same as the original data. The Tukey’s test we used for paired comparisons between each two studies is also consistent with the normality assumption and considers the overall variance of the parameter among all studies, rather than the variance between only the two studies compared at a given time (as done in Student t tests). This lowers the type 1 error (Bewick et al. 2004). Additional evidence that the simulation of raw data for Tukey’s test on these parameters was warranted is that our analysis captured known within-experiment relationships among parameters (e.g., Table 1 shows that rheobase, R_in, AHP_amp, and AHP_dur are most consistent when data are grouped by cell type, as physiologically expected in different motoneuron types). With these findings taken together, in the authors’ opinion, this simulation approach is rigorous enough to prove useful to this meta-analysis.

While the type of preparation (in vitro vs. in vivo) is a pertinent dichotomy to include as a variable in this analysis, we found that the difference between in vitro and in vivo is highly influenced by other biological variables, primarily species and age. For instance, almost all studies meeting the criteria for our analysis in the mouse used the in vitro preparation (i.e., recordings obtained from the spinal cord in a dish), whereas almost all studies in the cat used the in vivo preparation (i.e., recordings obtained from an intact cord in the animal). There was a mix for the rat experiments, but the in vivo preparation was usually used in adult animals, whereas the in vitro preparation was used in the neonates. Thus, given these trends in the literature, it is difficult to separate the effects of the preparation from the effects of age and species, and the biological variables were our focus.

Intra- and Interexperimental Consistency of Motoneuron Properties

The intraexperimental consistency score assesses how variable a given parameter is within one study, whereas the interexperimental consistency score assesses how variable the means of that parameter are among different studies. Our analysis shows that RMP and AP_amp have both very high intra- and interexperimental consistency scores (i.e., their values do not change much within one study or among different studies; Fig. 3, A and B). This high intra- and interexperimental consistency could be attributed to the typical use of these two parameters as discard criteria during data collection (i.e., recordings are typically obtained from motoneurons whose RMP < 60 mV and AP_amp > 70 mV). Parameters such as rheobase, R_in, and gain, on the other hand, have low intraexperimental consistency scores (i.e., their values vary considerably within a given experiment; Fig. 3A), which could reflect the association of these parameters with motoneuron types, which in reality form a continuum rather than discrete categories.

For studying a given parameter for modeling or experimental applications, Table 1 provides modelers and experimenters with a guide as to which biological variables should definitely be accounted for and suggests which variables should be included and reported in future studies. For example, Table 1 shows that gain will be most consistent among studies that use the same species and age. However, categorized typing (Tc) could not be considered for gain because of the lack of reported data. Therefore, future studies should report separate gain means for motoneurons of different categorized types and explore whether there are differences between types that may explain gain’s remaining interexperimental variance. Furthermore, it is likely unnecessary to group or compare gain according to sex, because we have shown that not including sex as a variable actually increased gain’s consistency across literature. This information provides a useful guide to the literature on these parameters and could inform experimentalists in the design of motoneuron experiments (i.e., what biological variables they need to report, those they need to account for, and those they could ignore). This information could also guide modelers in the development and validation of accurate motoneuron simulations (i.e., how strictly a given parameter needs to be matched to experimental data).

Insights for Experimenters

Experimenters face many challenges when collecting biological data, especially given that animal models have inherent biological variability. Collected measurements must be validated, and when those measurements are highly variable, interpreting and validating the data is particularly difficult. Investigators must consider whether that variability arises from a natural biological function or a confounding variable. To conclude that a given parameter is inherently variable, comparison with data on that parameter from the literature is needed. Furthermore, meta-analyses such as we present here can both clarify these issues and/or help us to discover new questions to investigate. For instance, G_in, C_m, and AHP_dur have high intraexperimental, but low interexperimental, consistency scores. That is, these parameters are consistent within studies, but their average values vary considerably among studies. This could be because these parameters are biologically stable (intraexperimentally consistent) but sensitive to variations in methods of measurement or statistical analysis (interexperimentally variable). Ultimately, additional meta-analysis studies are needed to elucidate these issues, the ultimate goal being to improve accuracy and comparability of data across the literature so that it can be interpreted correctly.

Interestingly, the three parameters with the lowest intraexperimental consistency reported (Fig. 3A) are R_in, gain, and rheobase—parameters often used to assess motoneuron excitability. This perhaps contributes to the discrepancies in reported hypo- vs. hyperexcitability of motoneurons in fields such as amyotrophic lateral sclerosis research (Delestrée et al. 2014; Jiang et al. 2017; Martínez-Silva et al. 2018) and illustrates the importance of meta-analyses in advancing the field. Additionally, RMP, AP_dur, and tau displayed the largest interexperimental consistency (Figs. 3B, 4A, and 4B). Thus experimenters should expect their own measurements of these parameters to closely match those in literature, when gathered under the same conditions.

An additional challenge for experimenters is to identify biological variables that significantly influence the parameters relevant to their research questions so that they may account for these variables in their experimental design and/or data analysis. This meta-analysis provides a guide for identifying those variables. For example, rheobase was eighth in interexperimental consistency (Fig. 3B, gray number rank) when no biological variables were accounted for. However, rheobase was highly consistent when biological variables were accounted for (Fig. 4, A and B). Thus experimenters should be sure to consider the appropriate biological variables (species, age, and cell type; see Table 1) when comparing or collecting rheobase data. In addition to improving accuracy of data, identifying and including only the appropriate biological variables can reduce the number of animals needed for an experiment and thus meet time and resource constraints. Our analysis of four biological variables that are key to motoneuron behaviors (Fig. 4) demonstrated that accounting for all four biological variables was less effective than specifically accounting for only those that impact that given parameter. Thus experimenters need to account for biological variables that specifically impact their parameters of primary interest when designing a study. Table 1 provides a useful guide to which biological variables influence each cell parameter examined in the present study.

Furthermore, even when biological variable sets were optimized for each parameter, our analysis suggested that additional factors beyond these four biological variables must be identified to explain the entirety of the interexperimental variance. Our analysis was designed to minimize experimental variables and some methodological variables by focusing on a single animal preparation and recording technique, with the goal of assessing the four selected key biological variables. However, we cannot claim to have eliminated all experimental or methodological confounds. Thus future analyses to explore the effects of additional relevant experimental, methodological, or biological variables would be highly useful to experimenters in the field. Consistent reporting of these variables by experimenters would also increase the effectiveness of such meta-analyses.

Insights for Modelers

Modelers face several challenges when developing computational models of biological systems or cells. One such challenge is determining which parameters the model must match to experimental means very strictly and which could or should be matched less strictly. The intraexperimental consistency scores provided (Fig. 3A) can inform the degree of accuracy and the appropriate range of variance for simulation of a given parameter. The parameters that displayed the highest intraexperimental consistency are RMP, AP_amp, and AHP_dur (Fig. 3A). Accordingly, in motoneuron simulations, these parameters should match their experimental means as accurately as possible (i.e., with less variability around the mean value). However, parameters with low intraexperimental consistency (e.g., rheobase, R_in, and gain; Fig. 3A) should be matched with more variance in their values around their experimental means to mimic experimental data. In fact, built-in variance could be especially important if multiple meta-analyses support that such variance is a natural feature of that parameter, and not due to biological, experimental, or methodological confounds.

Rigorous and independent verification of model parameters is another challenge for modelers. Ideally, the parameters of a validated model should match the experimental values of a source study (an experimental data set used for model development) and independent studies (one or more experimental data sets not used in model development). However, this is usually not the case because of the differences in reported values between experimental studies (Allen and Elbasiouny 2018). This study identified parameters with high interexperimental consistency scores before and after all and each parameter’s best biological variables were considered (e.g., AP_dur, RMP, and tau; see Figs. 3B and 4). These parameters are highly consistent and ought to be matched accurately by models across many experimental studies in the verification process. Conversely, parameters identified in this study with low interexperimental consistency scores (e.g., C_m and G_in) would be expected to vary among studies without reflecting negatively on the verification process. Further meta-analyses on these parameters would be needed to understand and identify the remaining sources of this inconsistency across studies.

A third, and major, challenge is accounting for relevant biological variables when selecting which experimental studies are relevant to the given simulation. Table 1 provides information on the impact of each of the four key biological variables on each parameter. This additional knowledge can help modelers determine the most relevant primary experimental data sets for constraining their simulations. For instance, experimental studies that account for the most impactful biological variables on the modeler’s parameters of interest should be chosen as the primary data sets for simulation. Interestingly, because our data show that each parameter is affected differently by biological variables, each parameter in the model might require its own set of primary experimental data.

Conclusion

In conclusion, we quantified the intra- and interexperimental variances for a set of motoneuron parameters that are frequently measured to elucidate the function of motoneurons under specific conditions. Understanding which of these useful parameters are more or less consistent within one study (i.e., intraexperimental consistency) or among different studies (i.e., interexperimental consistency), or which are impacted by species, age, sex, or cell type, is expected to be an important aid to reducing or interpreting conflicting data in the literature. This knowledge could also offer valuable guidance in the design of motoneuron experiments and in the development and validation of accurate motoneuron simulations. Finally, as our methodology is broadly applicable within neuroscience, it can assist with the daunting task of comprehensive meta-analysis of the field.

GRANTS

Funding for this work was provided by National Institute of Neurological Disorders and Stroke Grant NS091836 and National Academy of Sciences (NAS) and United States Agency for International Development (USAID) NAS Subaward 2000009148 (to S. M. Elbasiouny).

DISCLAIMERS

Any opinions, findings, conclusions, or recommendations expressed in this article are those of the authors alone, and do not necessarily reflect the views of USAID or NAS.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

M.M.H., J.M.A., and S.M.E. conceived and designed research; M.M.H. and J.M.A. performed experiments; M.M.H. and J.M.A. analyzed data; M.M.H., J.M.A., and S.M.E. interpreted results of experiments; M.M.H. prepared figures; M.M.H. and S.M.E. drafted manuscript; M.M.H. and S.M.E. edited and revised manuscript; M.M.H., J.M.A., and S.M.E. approved final version of manuscript.