Using Multilevel Models for the Analysis of Event-Related Potentials

Abstract

Multilevel modeling (MLM) is becoming increasingly accessible and popular in the analysis of event-related potentials (ERPs). In this article, we review the benefits of MLM for analyzing psychophysiological data, which often contains repeated observations within participants, and introduce some of the decision-making points in the analytic process, including how to set up the data set, specify the model, conduct hypothesis tests, and visualize the model estimates. We highlight how the use of MLM can extend the types of theoretical questions that can be answered using ERPs, including investigations of how ERPs vary meaningfully across trials within a testing session. We also address reporting practices and provide tools to calculate effect sizes and simulate power curves. Ultimately, we hope this review contributes to emerging best practices for the use of MLM with psychophysiological data.

Psychophysiologists have long recognized that the multivariate and densely repeated-measures nature of their data call for special approaches to data analysis (e.g., Games, 1976; Keselman & Rogan, 1980; Vasey & Thayer, 1987; Wilson, 1967). Over the past half-century most researchers have continued to use traditional approaches for analysis of psychophysiological data, including the use of repeated measures ANOVA to test differences in mean amplitude or latency of traditionally quantified event-related potential (ERP) components (see Jennings & Allen, 2017; Luck, 2014). In the early years of the field, this practice was likely driven by the lack of available alternatives or feasible means to carry them out. However, since statistical software packages for conducting complex data analyses—and desktop-type computers with which to run them—became available in the early 1980s, new analytic approaches have been developed that require more intensive computational resources for fitting models, including sophisticated approaches that do not rely on quantifying a particular ERP component at a single moment in time (e.g., Kiebel & Friston, 2004; Litvak et al., 2011; Pernet et al., 2011). However, it is beyond the scope of this article to describe all data analytic advancements for ERPs. Instead, we focus on one popular approach—multilevel modeling (MLM), alternatively called hierarchical linear modeling, mixed linear modeling, mixed effects modeling, and mixed effect regression—that has emerged for analyzing traditionally quantified ERP components.

MLM is appropriate for any data that is structured such that observations are recorded within naturally occurring groups, like schools. In the realm of ERPs, multiple observations are grouped within individuals. A number of previous articles have advocated for the use of MLM with psychophysiological data, including ERPs (see Bagiella et al., 2000; Boisgontier & Cheval, 2016; Goedert et al., 2013; Kristjansson et al., 2007; Krueger & Tian, 2004; Page-Gould, 2017; Tibon & Levy, 2015; Tremblay & Newman, 2015; Volpert- Esmond et al., 2018; Vossen et al., 2011). The purpose of this article is to provide a gentle orientation to psychophysiologists who are interested in learning more about how to apply MLMs to their ERP data, and to provide suggestions for best practices to increase the reproducibility of these analyses and orient researchers to available resources to make the best analytical choices.

Description of MLM and Its Advantages

Multilevel modeling is an extension of the General Linear Model (GLM) that estimates both fixed effects, as the GLM does, and random effects. Fixed effects refer to effects that are expected to generalize across the population and include the estimated effects of the specified independent variables (IV) on the dependent variable (DV). Fixed effects estimated with MLM, including betas, degrees of freedom, and associated p-values, are interpreted in a similar way as fixed effects estimated within a GLM. Unique to the MLM relative to the GLM are the random effects, which allow researchers to specify natural grouping variables (or “random factors”) in the data that result in non-independence of observations. In the case of ERP data, random factors will likely include participants and channels; however, other random factors are possible, including items or the stimuli used to elicit the ERP signal. The intercept of the random factor can be allowed to be random, meaning that a unique intercept will be estimated for each unit of that random factor (e.g., if participants are specified as a random factor, a different intercept can be estimated for each participant). Additionally, the slope associated with a particular predictor variable can be allowed to be random for each unit of the random factor (e.g., the effect of a particular predictor is estimated separately for each participant). The MLM will provide an estimate of the variance of the random intercept or slope, thereby providing an estimate of how much variability in the intercept or slope exists within a particular random factor.¹

rANOVA is essentially a special case of MLM and, thus, a multilevel model can be specified in a way that reproduces the results of a rANOVA. However, because it is the general case, MLM is much more flexible and allows for experimental or analytic designs that rANOVA cannot accommodate. For example, whereas rANOVA handles participants as the single random factor that results in dependence of observations, MLM can include multiple random factors in a variety of structures. This is particularly useful in ERP studies because repeated measurement within other factors produces dependence of observations, namely, electrodes/channels. In rANOVA, channel is often included as a predictor, which can result in unwieldy higher-order interactions that are difficult to interpret, especially as the number of channels increases (Luck, 2014). Instead, in MLM, in addition to specifying participants as a random factor, we can include channel as a random factor and estimate fixed effects of interest at the “average” channel.

Additionally, by specifying multiple random factors, MLM can test questions about the relative amounts of variance explained by different random factors using a special case of MLM called covariance component models, alternatively called cross-classified models (Dempster et al., 1981; Goldstein, 1987; Rasbash & Goldstein, 1994). For example, consider a stimulus set of emotional faces in which each target person makes a series of expressions that vary by arousal and valence. Covariance component models could be used to determine whether variance in P300 amplitude elicited by these faces is determined more by the targets or the participants (i.e., do P300s vary more as a function of which perceiver they are recorded from or which target they are elicited by?). Using unique ERP waveforms for each perceiver and target combination, we can specify perceivers and targets as crossed random factors and compare the variance in the random intercept for each group. More variance in one random factor or another suggests that either the perceiver or the target accounts for more variance in P300 amplitude. Thus, MLM expands the types of theoretical questions that can be answered using ERPs.

A second advantage of MLM is the flexibility it allows in the assumptions made about the variance and covariance between the observations in the dataset. In the early years of the field (e.g., Games, 1976; Keselman & Rogan, 1980; Wilson, 1967), researchers were particularly concerned about the possibility that the use of rANOVA for the kinds of successive measurements commonly obtained in psychophysiological studies often violate core assumptions underlying the use of rANOVA, especially the assumption of sphericity (i.e., that the variance of all pairwise differences between repeated measurements is constant). As noted by numerous researchers tackling this issue (e.g., Blair & Karniski, 1993; J. Richard Jennings & Wood, 1976; Keselman & Rogan, 1980; Vasey & Thayer, 1987), the assumption of sphericity is unrealistic when applied to psychophysiological data. Other solutions have been proposed within the context of rANOVA, including well-known adjustments to the degrees of freedom of a test—and therefore the observed p-value—based on the degree of non-sphericity it introduces (e.g., Greenhouse & Geisser, 1959; Huynh & Feldt, 1970, 1980) and multivariate tests such as Hoteling’s T² test (Mardia, 1975). MLM handles this issue by allowing models to be specified in a way that does not assume sphericity, thus making an adjustment unnecessary. Specifically, because MLMs are estimated with maximum likelihood methods, the assumed covariance structure of the data can be specified in a number of ways, including as an autoregressive covariance matrix, a compound symmetry covariance matrix (satisfies conditions of sphericity but more restrictive), or an unstructured covariance matrix, which makes no assumptions of equivalence among elements of the variance-covariance matrix (for a more in-depth discussion of variance-covariance structures, see Arnau et al., 2010; Page-Gould, 2017; Singer & Willett, 2003). In the case of violations of sphericity, MLMs with unstructured covariance matrices (and thus no assumption of sphericity) outperform rANOVA in containing the Type 1 error rate (Haverkamp & Beauducel, 2017), and thus may be particularly appropriate for analyzing ERP data. Additionally, another positive benefit of using maximum likelihood estimation is robustness to missing data (e.g., Enders & Tofighi, 2007; Graham, 2009; Krueger & Tian, 2004).

Lastly, in contrast with rANOVA, MLM allows for both continuous and categorical IVs. Continuous IVs can include observation-level variables, such as the hue of a particular stimulus if stimuli vary along a continuum of color, or a person-level variable, such as self-reported depression symptoms. Depending on how the random and fixed effects are specified, researchers can investigate questions such as how continuous individual differences influence the effect of a particular experimental manipulation within a single model (i.e., a cross-level interaction), rather than using difference or residual scores to produce a single ERP observation per participant and examine how it correlates with the individual difference variable of interest. Lastly, this feature of MLMs also allows researchers to investigate single-trial ERPs with time or trial included as a continuous variable in the model to look at change in ERPs over time, which we will address in later sections.

To introduce readers to the application of MLMs to ERP data, we will first use an example dataset with the error-related negativity and correct-response negativity (ERN/CRN) quantified from signal averaged waveforms to illustrate the steps of the analytic process. Then, we will discuss several extensions that are possible with MLM that rANOVA cannot accommodate.

Using MLMs with Signal Averaged ERP Waveforms: An Example

In the example data set, seventy-three college student participants (all African American; 22 male, 49 female, 2 trans/non-binary) completed a flanker task while EEG was recorded using 33 tin electrodes.² All scalp electrodes were referenced online to the right mastoid; an average mastoid reference was derived offline. Signals were amplified with a Neuroscan Synamps amplifier (Compumedics, Inc.), filtered on-line with a bandpass of .05–40 Hz at a sampling rate of 500 Hz. Electrode impedances were kept below 10 KΩ. Ocular artifacts (i.e., blinks) were corrected from the EEG signal using a regression-based procedure (Semlitsch et al., 1986). On each trial of the flanker task participants saw a horizontal string of five arrowheads facing to the left or right, in which the central arrowhead matched (congruent condition) or did not match (incongruent condition) the direction of the four flanker arrowheads. Participants completed 200 trials total and were allowed to rest every 50 trials. On each trial, participants first saw a fixation cross (jittered: 1400 ms, 1500 ms, 1600 ms), followed by the string of arrows (100 ms). Participants had 800 ms from the onset of the stimulus to identify the direction of the target (central) arrowhead with their right or left index fingers using a game controller. If they did not respond within the 800 ms response deadline, a ‘TOO SLOW’ message was presented on the screen before the next trial.

Following baseline correction (baseline window: −300 to −100 ms prior to the response), error trials and correct trials were averaged separately to create two averaged waveforms per participant (see Figure 1). Trials where no response was made and trials containing deflections +/− 75 μV were not included. Only participants with more than 6 artifact-free error trials were included in the analysis (Olvet & Hajcak, 2009), resulting in a sample of 60 participants (19 male, 39 female, 2 trans/non-binary) for the analysis. We quantified the ERN/CRN as the mean amplitude from 0–100 ms following incorrect/correct button presses, respectively, at channels Fz, F1, F2, F3, F4, FCz, FC3, FC4, Cz, C1, C2, C3, and C4. In addition to the flanker task, participants completed several self-report measures, including symptoms of anxiety and depression using the GAD-7 (Spitzer et al., 2006) and PHQ-9 (Kroenke & Spitzer, 2002), respectively.

Figure 1.

Averaged Waveforms from Example Dataset (ERN-CRN)

As with any other analytic strategy, the first thing to do is determine the theoretical hypothesis to test, including the predictor and outcome variables in the model. A large body of literature describes the ERN as a negative-going deflection that is larger following incorrect relative to correct responses (Holroyd & Coles, 2002; Olvet & Hajcak, 2009; Yeung, Botvinik & Cohen, 2004). To confirm this pattern in the example dataset, we first need to set up our data in long format, which means each observation is in a unique row with columns identifying each variable associated with each observation (e.g., participant number, response type, channel). This is in contrast to wide format, which is typically used in rANOVA, where each participant is in a unique row and columns represent both different variables and (usually means of) repeated observations (see Figure 2 for an illustration). We can also include individual differences variables as unique columns (e.g., anxiety and depression scores). Since each participant has only one value for each individual difference variable, that value is repeated for every row associated with a particular participant when the data is in long format.

Figure 2.

Illustration of wide and long data formats

Setting Up the Model

To fit the model, we will use the lme4 (Bates et al., 2015) and lmerTest (Kuznetsova et al., 2017) packages in R. All R code is downloadable at [https://github.com/hivolpertes/MLMbestpractices]. First, we need to determine the hypothesis we want to test, and thus the outcome variable and the fixed effects or predictor variables to include in the model. In this example, we want to test differences in the mean amplitude of the ERN/CRN following incorrect/correct responses. Thus, mean amplitude of the ERN/CRN is the outcome variable and response (e.g., incorrect, correct) is the predictor (or fixed effect).

Then, we must specify which random factors and structure to use, which reflects the hierarchical nature of the data. This includes which grouping variables (alternatively called random factors) to include and which slopes and intercepts you allow to vary for each random factor. ERP studies using averaged waveforms often have multiple observations for each channel and for each participant and thus, the most common random factors are participants and channels. Participants and channels can either be specified as independent factors (i.e., cross-classified model) or channels can be nested within participants (i.e., hierarchical model). A hierarchical model assumes that lower-level units (in this case, channels) belong to one and only one higher-level unit (in this case, participants). This might be the case if you expect the placement of the cap on each participant to vary, such that Fz measured for one participant is substantially different from Fz measured for another participant. In contrast, a cross-classified model assumes that lower-level units do not belong to one and only one higher level unit.

Additional random factors can be selected depending on the data set and theoretical hypothesis being tested, such as stimulus items. Importantly, any random factor should contain enough units or clusters that observations are clustered within, although the threshold of what is enough is debated and depends on what estimated parameter you are most interested in (Gelman & Hill, 2007; Huang, 2018; McNeish & Stapleton, 2016a, 2016b; Snijders & Bosker, 1999). In general, the fewer the units or number of clusters within a random factor, the poorer the estimation of the variance associated with the random factor (Maas & Hox, 2005). A common rule of thumb is the 30/30 rule (30 units or clusters with 30 observations within each cluster). However, when examining fixed effects, others recommend a minimum threshold of 10 (Snijders & Bosker, 1993), some suggest a minimal threshold of 5 when examining fixed effects but 10–100 when examining random effects (McNeish & Stapleton, 2016a), and others suggest that having fewer than 5 units within a random factor does no harm but also does not differentiate the multilevel model from a classical regression model when examining the fixed effects (e.g., Gelman & Hill, 2007). Since most ERP studies using MLM are primarily interested in the fixed effects, we recommend including measurements from at least 5 units or clusters within a random factor (e.g., at least 5 channels in order to use channel as a random factor). In the current example, we have repeated measurements within channels (13 channels total) and participants (60 participants total), so both are used as random factors. Since the same channels are being used for all participants, and we expect channels measured for one participant to be the same as for another participant, we will use a cross-classified model for this example.

Now that we have our random factors, we can think about which variables correspond to each level of the model. Level 1 variables correspond to individual observations, such as response type, other experimental manipulations, or aspects of the stimuli or trials that are included as predictors. Level 2 variables correspond to one level above that. In a cross-classified model where participants and channels are crossed random factors (and on the same “level,” so to speak), variables corresponding to either participants or channels are Level 2 variables. In a hierarchical model where channels are nested within participants, variables corresponding to channels are Level 2 variables and variables corresponding to participants are Level 3 variables.

Once you have chosen your random factors and decided to use a hierarchical or cross-classified model, you must decide which slopes and intercepts will vary by random factor. In general, allowing the effect of a variable to vary by a random factor (i.e., including it as a random slope) will not affect the estimate of the fixed effect for that variable, because the fixed effect is essentially the average of the random slopes. However, including a random slope will generally expand the standard error of the fixed estimate, thus increasing the associated p-value (Barr et al., 2013; Gelman & Hill, 2007). In other words, including a random slope (especially when there is a lot of group-related variance) controls the Type 1 error rate of the test of the fixed effect more tightly and provides a more conservative (and, some argue, more appropriate) test (Heisig & Schaeffer, 2019). When choosing which effects to include as random slopes, you can use either a theory-driven approach or an empirical or data-driven approach.

Theory-driven approach.

A linear model is a formal representation of a hypothesis, which extends to how you believe units within a random factor differ from one another. If you think that people differ in terms of the outcome variable (e.g., average amplitude of a given ERP component), then you will want to estimate random intercepts for each participant. If you think that the effect of a particular predictor on the outcome will differ across people in either size or direction, then you will want to estimate a random slope for that particular predictor by participant. Similar justifications can be made for including random intercepts and slopes for other random factors, such as channels. However, given that adjacent electrodes are theorized to measure similar brain activity, the effect of a predictor is not often expected to differ across channels and random slopes are not often used in this case. Thus, one way to make decisions about random effects specification is based on past empirical data or theory.

Empirical approach.

Of course, your theory may be wrong (or limited). Within the last decade, researchers in psycholinguistics began calling for researchers to follow a data-driven procedure where the maximal random effects are specified for every model (“maximal model;” Barr et al., 2013). In a maximal model, all Level-1 predictors³ are specified as random slopes. However, others have noted that using maximal models can result in significant loss of power (Matuschek et al., 2017). Additionally, as noted by Barr et al. (2013), the maximal model is frequently too complex to properly converge. When the maximal model is too complex to converge, parameter estimates are incorrect and models must be simplified. Thus, the maximal model may not always be appropriate and parsimonious models may be preferable. To determine the most appropriate parsimonious models, a number of strategies are used, including comparing nested models using likelihood ratio tests (for a comparison of strategies for model selection, see Seedorff et al., 2019). Regardless of whether you use a theory-driven or empirical approach to specify random effects, we believe that best practices involve at minimum including random slopes for the main fixed effects of interest to properly control for Type 1 error, as intercept-only models are frequently too liberal and may result in spurious findings (Bell et al., 2019). Once you have accounted for the main fixed effects of interest, you can make decisions whether to include more complex interactions as random slopes using a data-driven approach.

Last, after having determined the fixed effects and random effects, you should choose the type of variance-covariance matrix to use, which specifies assumptions about how observations within and across units in a random factor (e.g., within and across participants) vary and covary. Some variance-covariance matrices involve more stringent assumptions, such as a compound symmetry variance-covariance matrix, which more closely approximates a rANOVA. We suggest using an unstructured variance-covariance matrix, which removes the sphericity assumption of rANOVA, as the assumption of sphericity is unrealistic when applied to psychophysiological data (e.g., Blair & Karniski, 1993; J. Richard Jennings & Wood, 1976; Keselman & Rogan, 1980; Vasey & Thayer, 1987). By default, the lme4 package in R uses an unstructured variance-covariance matrix, although SAS by default uses a VC variance-covariance matrix (for more information on variance-covariance matrixes, see Haverkamp & Beauducel, 2017; Page-Gould, 2017).

As mentioned before, in our example, we are testing the effect of Response Type (RespType) on the mean amplitude of the ERN/CRN. The model includes two crossed random factors (Participant, Channel), which we are estimating using an unstructured variance-covariance matrix. Using a theory-driven approach to determine the random effects, we included 1) a random intercept by participant, 2) response type as a random slope by participant, and 3) a random intercept by channel. The full model is described using Wilkinson notation as:

$$\text{MeanAmp}~\text{RespType}+(\text{Resptype}\mid \text{Participant})+(1\mid \text{Channel})$$

Our interpretation of the fixed effects depends on how Response Type is coded, similarly to interpreting fixed effects from a single-level regression model. When Response Type is dummy-coded (correct = 0, incorrect = 1), the estimate of the intercept is b = 5.926, 95% CIs [4.51, 7.34] and the estimate of the effect of Response Type is b = −4.294, 95% CIs [−5.20, −3.39]. From these estimates, we can calculate the estimated marginal means for each group: the estimated marginal mean in the correct condition is 5.926 μV (the estimate of the intercept, since correct is coded as 0) and the estimated marginal mean in the incorrect condition is 1.632 μV (the estimate of the intercept minus the estimate of Condition). When Condition is effect-coded (correct = −1, incorrect = 1), the estimate of the intercept is b = 3.779, 95% CIs [2.42, 5.13] and the estimate of the effect of Response Type is b = −2.147, 95% CIs [−2.60, −1.69], which means that across all trials, the estimated marginal mean is 3.779 μV. Then, we can calculate the estimated marginal means for each condition by adding or subtracting the estimate of Response Type to the intercept, which gives us the equivalent marginal means for correct and incorrect trials as the dummy-coded model. Using unstandardized estimates in this way gives us a sense of the magnitude of the difference between conditions in a meaningful unit (μV). When examining latency as the outcome variable, estimates similarly can be interpreted in whichever meaningful unit the outcome variable was measured on (such as milliseconds).

Researchers using the null hypothesis significance testing (NHST) approach will additionally want to know if the effect of Response Type is statistically different from zero. Compared to single-level regression, determination of degrees of freedom (and thus, the p-value associated with a test of a fixed effect) is much more complicated in MLM. A number of possibilities exist for testing the significance of a fixed effect, including likelihood ratio tests of nested models, applying the z distribution to the Wald t values, Markov-chain Monte Carlo (MCMC) sampling, parametric bootstrapping, and different approximations for denominator degrees of freedom. We recommend the Satterthwaite approximation for denominator degrees of freedom, partly because it more appropriately controls Type 1 error and is less dependent on sample size than other methods, especially for REML-fitted models (Luke, 2017) and because of the ease of implementation—Satterthwaite approximation is the default for SAS and can be applied in R using the lmerTest package (Kuznetsova et al., 2017) in conjunction with the lme4 package (Bates et al., 2015). All examples presented in this paper use the Satterthwaite approximation when reporting p values.

Critics of NHST suggest that whether the effect of a particular predictor is different from zero is not always informative—instead, it may be more useful to understand the proportion of variance explained by the fixed effects (and therefore make judgements of the meaningfulness of the effect). In a single level regression or GLM, readers are familiar with R² as the variance explained by all of the fixed effects included in the model. However, in multilevel models, the variance explained is a little more complex, since there are now multiple residual terms. Thus, several methods of calculating a pseudo-R² have been proposed (e.g., Edwards et al., 2008; Johnson, 2014; Nakagawa et al., 2017; Nakagawa & Schielzeth, 2013; Snijders & Bosker, 1999). Importantly, there is a distinction between the marginal R², which is the proportion of the total variance explained by the fixed effects, and the conditional R², which is the proportion of the variance explained by both fixed and random effects. Either the marginal or conditional R² can then be converted to other effect sizes that may be more common in your particular research literature. For example, the model R² can be used to compute Cohen’s f² (Cohen, 1992) using:

$$f^2=\frac{R^2}{1-R^2}$$

To estimate the variance explained by a particular predictor (i.e., to obtain an estimate of the local effect size), several methods exist. One method is to estimate Cohen’s f² for each local effect by estimating R² for two nested models:

$$f^2=\frac{R_2^2-R_1^2}{1-R_2^2}$$

where R₂² represents the variance explained by a model with the effect of interest (the full model) and R₁² represents the variance explained by a model without the effect of interest (the restricted model). Cohen’s f² for a local effect can easily be directly calculated using this method in SAS (Selya et al., 2012) and in R by fitting each model separately and estimating the pseudo-R² as mentioned previously using the r.squaredGLMM() function in the MuMIn packag (Bartoń, 2020) using the or the r2beta() function in the r2glmm package (Jaeger, 2017).

An alternative method is to calculate a partial R² statistic for each predictor, $R_{\beta }^2$(Edwards et al., 2008). One $R_{\beta }^2$ statistic can be calculated for each predictor using the ANOVA output of an MLM model to get the F-statistic, the numerator (effect) degrees of freedom, and the denominator (residual or error) degrees of freedom that correspond to each predictor.

$$R_{\beta }^2=\frac{\left(\frac{d{f}_{numerator}}{d{f}_{denominator}}\right)F}{1+\left(\frac{d{f}_{numerator}}{d{f}_{denominator}}\right)F}$$

In the realm of ERPs, it remains unclear how large of an effect is meaningful, as meaningful differences in amplitude may vary depending on the ERP component of interest and the variance in the outcome is related to a number of factors, including the noisiness of the data and how many trials are included in each averaged waveform. Thus, descriptions of effect size in future ERP studies are essential to triangulate what may be a meaningful effect size in the study of ERPs.

Visualizing Data

In addition to statistical tests, visualizing data is an important component of understanding statistical results. As most ERP studies are interested in the effect of categorical predictors, a common approach using rANOVA is to use bar or line graphs to depict mean amplitude averaged across participants and channels in each condition. However, depicting averages from the data does not account for the multilevel structure of the data, nor does it depict how much variability in the effect exists across people. When using multilevel modeling, we can plot 1) the fixed effects estimates to summarize patterns across the whole sample, 2) the variance within each grouping variable (e.g., how participants vary from each other), or both. To plot mean differences across experimental conditions and still account for the multilevel structure, we can plot the model-estimated means from the fixed effects. For both bar and line charts, this should include the values of the outcome variable that are predicted from your model for each condition (i.e., estimated means) and the standard errors of these model-estimated means as error bars. Estimated means can be calculated using a user-friendly, online tool available at http://www.quantpsy.org/interact/ (Preacher et al., 2006) or the emmeans package in R (Length, 2020) and then plotted as in Figure 3.

Figure 3.

Model-estimated means plot illustrating fixed effect of Response Type

However, one benefit of MLM is being able to estimate unique effects for each unit in a random factor (e.g., participant) by including random slopes in the model. To visually represent the variance in a particular effect, plot the best linear unbiased predictions (BLUPs) estimated for each participant using a “spaghetti plot”. Spaghetti plots illustrate the variance in the effect, which we can see in differences in the slopes of the lines. If all lines are relatively parallel, there is little variance in the effect of Condition across participants (which will be reflected in a small estimate of variance of the random slope of Condition by participant), whereas lots of intersecting lines that are not parallel suggest a large amount of variance in the random slope. We suggest plotting each line with an opacity level below 100% to make each line easier to see and consider making each line its own color, if color visualizations are an option for your publishing outlet of choice (see Figure 4 for an example). As we can see in this example, most participants show the same pattern as the fixed effect (more negative ERN amplitude in the incorrect condition compared to the correct condition), but some slopes are flatter than others, and some participants even show an effect in the opposite direction.

Figure 4.

Spaghetti plot illustrating variance in effect of Condition across participants (i.e., random slopes)

Of course, one can also plot both the estimated means for each condition and variance across individuals by overlaying the two plots. We suggest plotting the “average” effect (i.e., the fixed effect) in a slightly thicker width or different color to make it stand out (e.g., see Figure 5).

Figure 5. Spaghetti plot with model estimated means overlaid.

Note. The thick black line represents the average relationship estimated by the fixed effect and the thinner, multicolored lines represent the specific relationships estimated for each person (the random slopes).

Extended Applications of MLM

One of the major benefits of MLM that rANOVA cannot accommodate is including continuous variables in the model. One example of this is testing how individual difference variables moderate the effect of the manipulated predictor. Past work has shown a link between trait anxiety and the size of the ERN/CRN, such that those who are more anxious show a more pronounced negativity following errors (Hajcak, McDonald & Simons, 2003; Weinberg, Olvet & Hacjak, 2010; Meyer, 2017). To test the effect of trait anxiety on the size of the ERN/CRN using MLM, we can simply include trait anxiety as a predictor in the model (Response Type is effect coded; Correct = −1, Incorrect = 1):

$$\text{MeanAmp}~\text{RespType}+\text{Anx}+(\text{RespType|Participant})\text{+}(\text{1|Channel})$$

As mentioned earlier, we would not include anxiety as a random slope by participant because there is only one observation per participant (and is thus invariant). In this model, the effect of Response Type remains significant, b = −1.87, 95% CIs [−2.61, −1.12], t(58.0) = −4.92, p < .001, such that mean amplitude is more negative following incorrect responses than correct responses. The effect of trait anxiety is marginally significant, b = 0.19, 95% CIs [−0.01, 0.39], t(58.0) = 1.91, p = .061. Most importantly, to examine whether trait anxiety moderates reactivity to errors, we would look at the Response Type x Anxiety interaction. In this sample, the interaction is not significant, b = −0.05, 95% CIs [−0.15, 0.05], t(58.0) = −0.93, p = .356. This is a similar test as correlating trait anxiety scores with a difference score of the ERN and CRN (e.g., ΔERN). Previous research has shown differences in ΔERN between anxious and control groups (Ladouceur et al., 2006; Pasion & Barbosa, 2019; Weinberg et al., 2010, 2012, 2015) and significant relationships between symptoms of generalized anxiety disorder and ΔERN (Bress et al., 2015; Klawohn et al., 2020), such that more anxious participants show a more negative ERN relative to the CRN, although other studies have not found consistent significant correlations between self-reported anxiety and ΔERN (e.g., Meyer et al., 2012).

Another application that MLM allows for is the investigation of ERP responses to specific stimuli or events from individual trials, allowing researchers to investigate how ERP signals meaningfully change over the course of different trials or meaningfully differ in response to specific instantiations of stimulus presentations. As mentioned previously, prior to data analysis researchers typically average all responses elicited by stimuli of the same type or experimental condition (i.e., signal averaging; Luck, 2014), which results in a data structure in which each participant has a single observation per channel for each experimental condition. This technique is effective for isolating physiological responses to events of interest (i.e., increasing signal-to-noise ratio) but makes assumptions that might not be tenable, including that the signal is constant across trials, and that any trial-to-trial variation is solely the result of noise, and therefore meaningless. A number of factors, including habituation, fatigue, sensitization, or momentary lapses in attention can result in meaningful variation (i.e., not merely noise) in ERPs across trials, thereby undermining the validity of signal averaging in some situations.

A number of approaches to analyzing single trial ERPs have been proposed (Blankertz et al., 2011; Coles et al., 1985; Debener et al., 2005; Gaspar et al., 2011; Jung et al., 2001; Pernet, Sajda, et al., 2011; Philiastides et al., 2006; Quiroga & Garcia, 2003; Ratcliff et al., 2009; Regtvoort et al., 2006; Rousselet et al., 2011; Sassenhagen et al., 2014). Multilevel modeling provides an extremely useful additional tool for researchers interested in trial-level variation in ERPs. Note, however, that because noise is not first being removed from the waveforms using the signal averaging approach, it is important that the EEG data are as clean as possible when a trial-level approach is used. Researchers should spend additional time and effort during the data collection process to ensure the highest quality data possible to reduce noise in the data and increase the ability of multilevel models to detect fixed effects of interest from individual trials.

To examine the linear effect of time on change in psychological processes, researchers can include time or trial number as an additional fixed predictor in the model (e.g., Berry et al., 2019; Brush et al., 2018; Volpert- Esmond et al., 2018). As an example, the model may be specified as:

$$\text{DV}~\text{IV}+\text{Trial}+(\text{IV|Participant})\text{+}(\text{1|Channel})$$

Note that the inclusion of Trial in this way can only capture long-range trends such as habituation and fatigue. Quadratic and other non-linear effects can be included as additional predictors, although little research has been done in this area and polynomial fitting comes with its own set of challenges (Kristjansson et al., 2007; Tremblay & Newman, 2015). By examining the fixed effect of time, or interactions between time and other fixed predictors, researchers can infer large-scale change in the amplitude or latency of ERP components over the course of an experiment, as well as different rates of change for different experimental conditions. Additionally, the variable indexing time can be included as a random slope by participant so that researchers can examine how the effect of time (including processes such as habituation or learning) differs across participants. To get estimates of individual differences in the rate of change in ERPs, researchers can extract the BLUPs, which are participant-specific estimates of the effect of time. However, including time as a random slope often results in non-convergence issues, which must be addressed before interpreting the BLUPs. Last, using MLM with single-trial ERPs opens the door to using ERP amplitude or latency as a predictor of other trial-level variables (such as reaction time or other downstream ERP components; Volpert-Esmond & Bartholow, 2020; Von Gunten et al., 2018).

Including continuous variables introduces increased complexity surrounding issues of centering variables that are unique to MLM. In typical single-level OLS regression, researchers often center and/or standardize continuous variables in order to interpret all other fixed effects as the effect observed at the mean of the centered variable. We suggest taking a similar approach to all continuous Level 2 variables (e.g., individual difference variables). However, in multilevel data, continuous Level 1 variables can either be centered across the entire data set (grand-mean centering) or centered within each level of the grouping variables (group-mean centering). The type of centering one chooses can significantly impact the interpretation of the fixed effects. There are a number of other resources discussing centering (e.g., Brauer & Curtin, 2018; Enders & Tofighi, 2007; Kreft et al., 1995; Paccagnella, 2006; Page-Gould, 2017) and contrast coding (Schad et al., 2020) within multilevel data.

One particular case of centering that may be of interest may be disaggregating between- and within-participant effects of a continuous predictor (e.g., Curran & Bauer, 2011). This is particularly relevant when using single-trial ERPs as a continuous predictor of some other outcome, such as a behavioral response within the same trial or an ERP response on a subsequent trial. In the absence of disaggregation, the relationship between trial-level ERPs and reaction time (for example) conflates the between-person effect (i.e., Do people with particularly large ERP responses also respond faster to stimuli in a particular task?) and the within-person effect (i.e., Does a larger ERP response on a particular trial, relative to a person’s average ERP response, facilitate a faster reaction time?). Depending on the theoretical question, researchers may be more interested in one relationship than the other. To disaggregate within- and between-person effects, the researcher can effectively separate the predictor variable of interest into two separate predictors. The first predictor—each participants’ mean—is entered as a Level-2 (person level) predictor and represents the between-person effect. The second predictor—the participant-centered variable—is entered as a Level-1 predictor and represents the within-person effect.

Reporting Practices

Because of the complexity surrounding MLMs, researchers have a number of degrees of freedom with respect to how MLMs are estimated and reported, including what covariance structure to use, which variables to include as fixed and random effects, how to test for interactions, how to center or effect-code variables, etc. Because of the flexibility of these models, it is imperative to provide enough information for an independent party to replicate the analysis and evaluate its suitability for the dataset at hand. Of course, providing code in an online repository such as Open Science Framework or GitHub is preferable. But we encourage researchers to include all essential information in the manuscript as well. At minimum, the entire model, including variance-covariance structure and random effects should be described (Meteyard & Davies, 2020). To most effectively communicate the structure of each model used, we suggest using Wilkinson notation, which specifies the DV, IVs, and random effects. For example,

$$\text{DV}~\text{IV}1+\text{IV}2+(1+\text{IV}1\text{|Participant})+(1\text{|Participant:}\text{Channel})$$

specifies that two predictors were included, but not their interaction; that the intercept and the effect of the first predictor was allowed to vary by participant (i.e., IV1 was included as a random slope by participant); and that the intercept was allowed to vary by channel nested within subject. Alternatively, the following model specifies participants and channels as crossed random factors:

$$\text{DV}~\text{IV}1+\text{IV}2+(1+\text{IV1|Participant})+(1\text{|Channel})$$

R users will recognize that Wilkinson notation is used in the lme4 package to specify models (and is also used in Matlab), thus providing less of a barrier than formal mathematical notation. The statistical software used to fit the models should additionally be reported, along with any changes to the default specifications (e.g., which covariance structure is specified). More extensive recommendations about reporting practices regarding model selection, model output, etc., can be found in Meteyard and Davies (2020).

In addition to reporting the structure of the models using Wilkinson notation, we suggest reporting the intraclass correlation coefficient (ICC) for each random factor, which can be calculated from the variances estimated in the random effects:

$$\text{ICC}=\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}$$

where τ² is the between-cluster variance (the variance associated with the random factor) and σ² is the residual variance (Lorah, 2018). This gives you the proportion of total variation in the data that is accounted for by a particular random factor where higher ICCs represent more variance between units within that random factors (Gelman & Hill, 2018). Since the complexity of the model affects the calculation of ICC, you should use variance estimates from an intercept-only model (i.e., a model with no fixed predictors):

$$\text{DV}~1+(1\text{|Participant})+(1\text{|Channel})$$

When including more than one random factor (e.g., including participants and channels in a cross-classified model), one would include the variance of all groups in the denominator. As an example, let’s look at sources of variance in the mean amplitude of the P2 ERP component elicited by Black and White male faces during a race categorization task⁴. To calculate the ICC associated with subject, we would look at the output for the random effects from the following intercept-only model⁵, first using the signal averaged data:

$$\text{Model:}\text{P2amp}~1+(1\text{|Participant})+(1\text{|Channel})$$

Random effects output:

Random	Name	Variance	Std. Dev.
Factors
Participant	(Intercept)	8.314	2.883
Channel	(Intercept)	0.141	0.376
Residual		2.306	1.5184

$$\begin{gathered} {\text{ICC}}_{par}=\frac{8.314}{8.314+0.141+2.306} \\ \text{=.77} \end{gathered}$$

$$\begin{gathered} {\text{ICC}}_{elec}=\frac{0.141}{8.314+0.141+2.306} \\ \text{=.01} \end{gathered}$$

In other words, variance between people accounts for 77% of the total variance, suggesting there is a lot of between-person variability in ERPs, whereas variance between channels accounts for 1% of the total variance, suggesting is there not a lot of variability between channels, which is expected given similarities in waveforms at adjacent channels. In contrast, when using trial level data, the ICC associated with subject is .09, suggesting between-person variability only accounts for 9% of the total variance. Because of the amount of within-person variance from trial to trial, between-person variance accounts for much less of the total variance when using single-trial ERPs instead of signal-averaged ERPs.

Estimating Power

Another barrier in transitioning to using MLM is the daunting prospect of having to do a power analysis. Evaluating the power of a hypothesis test, which is defined as the probability that the test will correctly reject the null hypothesis when the null hypothesis is false, is important in assessing how likely a particular result is true and able to be replicated. Additionally, ERP studies are often underpowered to find small effects (Clayson et al., 2019). Given that estimating an effect of zero—or estimating effects completely at random—is more accurate at determining the true population mean than using sample means derived from poorly powered studies (Davis-Stober et al., 2018), and that EEG studies are time-intensive and costly to run, an a priori power analysis can inform a researcher whether they have the resources to conduct a study that is well-powered enough to be informative. Additionally, according to recent guidelines for best practices in reporting of ERP studies (Keil et al., 2014), researchers should always report the achieved power of a particular design. Many tools are available to estimate power for typical single-level designs (e.g., Faul et al., 2009; Murphy et al., 2014), although discussion is still ongoing about the most appropriate ways to conduct and use a power analysis (Anderson, Kelley, & Maxwell, 2017; Cribbie, Beribisky, & Alter, 2019; Albers & Lakens, 2018; Lakens & Evers, 2012).

In multilevel designs, how power relates to sample size is more complicated, as both the number of groups (e.g., the number of individuals who participate during the study) and the number of observations per group (e.g., the number of trials or observations per individual) can vary. In multilevel models, power is affected by group sample size, observation sample size within each group, the ICC associated with group, whether you are testing a Level 1 (observation-level) or Level 2 (participant-level) effect, and numerous other parameters of the model (Arend & Schäfer, 2019). As a general rule of thumb, increasing the number of Level 2 units (e.g., the number of people participating in the study) has a larger effect on power to detect fixed effects than increasing the number of Level 1 units (e.g., the number of experimental conditions or trials within each participant; Maas & Hox, 2005; Snijders, 2005). For a more specific approximation of the sample size (at both Level 1 and Level 2) needed to achieve the desired level of power for a particular test, most researchers use a simulation approach to power using Monte Carlo simulations. This approach repeatedly simulates data from the hypothetical distribution that we expect our sampled data to come from and then fits the same multilevel model to each data set. Power is estimated by how often the true effect is detected.

To set up a power simulation, you need to make assumptions about the true treatment effect and also specify all the other parameters that characterize the study, including the size of the fixed effect of interest, ICCs of any random grouping variables, variances of random intercepts and slopes, correlations between random intercepts and slope, etc. Because of the large number of parameters needed to simulate an appropriate data set, it is often easier to conduct a power simulation on a set of pilot data, although parameters can be assumed and simulated without pilot data (Gelman & Hill, 2007). The simr package in R (Green & MacLeod, 2016) has emerged as a popular tool for power simulations using multilevel models. The package allows users to input a sample data set (either a pilot or simulated data set) and calculate observed power for a desired effect, as well as produce power curves in which power is plotted as a function of a particular aspect of the design, such as number of participants, number of observations within each participant, or effect size. To provide an example of a power curve generated using simr, we use previously published data⁶ looking at how the race of a face influences mean P2 amplitude:

$$\text{Model:}\text{P2amp}~\text{Race}+(\text{Race|Participant})+(1\text{|Channel})$$

When using signal averaged data, the effect of race is significant, b = −0.59, t(64.0) = −4.82, p < .001, such that Black male faces elicit larger P2s than White male faces. A post-hoc power simulation indicates that the observed power for this effect with the given sample size (65 participants, 7 channels for each participant, and 2 observations for each channel) is 99.6%, suggesting this design is very well-powered (and maybe even overpowered) to detect this effect. Thus, we achieve 80% power to detect an effect of this size with about 25 participants. However, using pilot data to estimate the true effect size may result in an underpowered follow-up study (Albers & Lakens, 2018; Anderson, Kelley & Maxwell, 2017). Thus, we suggest either adjusting the anticipating effect size to be smaller than that achieved in a pilot study when planning a follow up study or producing a sensitivity power curve to identify what sample size would be needed to detect the smallest meaningful effect size. Figure 6 demonstrates the decrease in power as the effect size decreases, indicating that with a sample of this size, we would be able to detect an effect as small as b = −0.35 with 80% power.

Figure 6. Power to detect the fixed effect of race on P2 amplitude as a function of effect size.

Note. 14 observations are included for each participant (7 channels with 2 observations at each channel). The sample size is set at 65 participants.

Limitations

MLM is not a panacea. As with any analytic approach, MLM comes with significant limitations. First, effectively using MLM involves gaining the expertise to organize data in the appropriate format, learning how to implement models in statistical software, making appropriate decisions for model specification, correctly interpreting the output, etc. Additionally, due to the continued evolution and development of knowledge about MLMs, there currently is a perceived lack of consensus and established, standardized procedures (Meteyard & Davies, 2020). Many resources are becoming available for researchers interested in learning this statistical approach, including workshops at prominent conferences (i.e., Society for Psychophysiological Research), stand-alone workshops hosted by societies, private organizations, and universities (e.g., APA Advanced Training Institutes, Statistical Horizons, University of Michigan, University of North Carolina, University of Connecticut, Arizona State University), and numerous tutorials and articles on applying MLM to both behavioral data (Arnau et al., 2010; Baayen et al., 2008; Brauer & Curtin, 2018; Gueorguieva & Krystal, 2004; Jaeger, 2008; Judd et al., 2012; Maas & Snijders, 2003; Quené & van den Bergh, 2004, 2008) and psychophysiological data (Bagiella et al., 2000; Kristjansson et al., 2007; Page-Gould, 2017; Tibon & Levy, 2015; Tremblay & Newman, 2015; Volpert- Esmond et al., 2018; Vossen et al., 2011). However, little is known about the effectiveness of this training and how it is implemented in practice (King et al., 2019). Moreover, the mere fact that these opportunities exist does not ensure that researchers will or can take advantage of them, and therefore this situation is far from ideal in terms of ensuring adequate quantitative methods training in the field—likely contributing to a significant gap in psychologists’ quantitative training. Thus, learning how to appropriately apply MLM to ERP data may be a significant barrier.

In addition to the time cost of learning the approach, MLM is often quite computing-power intensive and models can take much longer than a typical rANOVA to run. In the case of the P2 example given previously, the first author ran these models on a MacBook Air with a 1.6 GHz Dual-Core Intel Core i5 processer with 8 GB of RAM. To test the effect of face race on P2 amplitude using signal-averaged data, it took only a few seconds to fit the model. To run the same model using trial-level data, it took less than 10 seconds to fit the model. However, as the data set becomes larger and the model becomes more complex, the time required to fit a MLM increases dramatically. For example, this more complex model testing the effect of target race, target gender, fixation, task, and participant race on trial-level P2 data recorded in two face processing tasks (256 trials in each task)⁷ took over 12 hours to fit:

$$\begin{gathered} \text{P2amp}\text{~}\text{TarRace}\ast \text{TarGender}\ast \text{Fix}\ast \text{Task}\ast \text{parRace}+ \\ (\text{TarRace}\ast \text{TarGender}\ast \text{Fix}\ast \text{Task|Participant})\text{+} \\ (\text{1|Participant}:\text{Channel}) \end{gathered}$$

Similarly, because power simulations require fitting the same model to multiple simulated data sets, power simulations can take quite some time. To produce the power curve for the signal averaged data shown in Figure 1 (varying the sample size), the power simulation took roughly 90 min to run. The power curve shown in Figure 2 (varying the size of the effect) took roughly three hours to run. To produce the same power curve depicted in Figure 1 for trial-level data rather than signal-averaged data, the first author did not have enough processing power—when she attempted it, the simulation ran for one week straight (24 hours/day) and was only a quarter of the way finished before the author stopped the simulation. Thus, processing power is essential for running MLMs, especially when using hundreds of trial-level observations, which is increased 20-fold when running power simulations. Given limitations in researchers’ access to powerful computers or server clusters, this may be a significant limitation in the use of MLMs for ERP data, especially when using trial-level data.

Lastly, researchers may encounter estimation problems when running the model, the most common of which are convergence problems, very long estimation times, and singularity issues. Most estimation problems can be addressed with two guiding principles: Simpler models and better fitting models will have fewer estimation problems. Generally speaking, the more complex your random effects are, the more difficult the model parameters are to estimate. As mentioned earlier, one approach is to trim the most complex random effect (e.g., random slope for an interaction term), run the model again, and then progressively trim random effects in order of decreasing complexity until the model converges.

Additionally, the time that it will take to run an MLM is directly related to how well your model specification reflects reality. If it takes a very long time to run an MLM but you received a warning (e.g., the Hessian matrix was not positive-definite), look at the variance estimates in the random effects to see if any variances are zero or very close to zero. If so, then it means that the random effect does not vary much from group to group, and thus is an over-specification. Trim any random effects that have zero variance, and run the model again.

The process of troubleshooting estimation problems is done iteratively (i.e., remove one term, rerun the MLM), so that the model does not get oversimplified in the process. Given how many choices that can be made when handling these problems, it is a good idea to establish a common approach to troubleshooting estimation problems that you apply across all your research studies that use MLM and reporting your approach in your method sections.

Conclusion

Despite its limitations, MLM has great potential to advance the study of neurocognitive processing through advanced modeling of psychophysiological data. We have focused here on the use of MLM for ERP data, but of course MLM also can be used with other psychophysiological data that are structured similarly (e.g., Bourassa et al., 2016; Briollais et al., 2003; de Looff et al., 2019). Not only can MLM address limitations inherent to the use of rANOVA with such data, such as violations of assumptions leading to inflated Type 1 error rate and the need for listwise deletion when observations are missing, MLM can greatly expand the types of research questions that can be posed and tested with psychophysiology. For example, here we highlighted that MLM permits examination of change over time in physiological and behavioral responses using trial-level data, and also how trial-level data can be used to test for within- and between-trial associations among dependent measures (e.g., ERP amplitudes predicting reaction time; Volpert-Esmond & Bartholow, 2020). Even if researchers stick with a traditional signal averaging approach to signal processing, MLM affords more precise modeling of effects and more appropriate parsing of error variance (e.g., by estimating random slopes differing across individuals and independent variable conditions) than does rANOVA. Thus, we recommend that researchers invest the time—and, if necessary, the resources to bolster their computing power—to learn MLM, and discover the flexibility the technique affords for their psychophysiological research programs. Finally, we strongly recommend that researchers who adopt MLM pay close attention to the latest developments in the rapidly evolving literature on best practices in the use of the technique and its limitations.

Highlights.

• Paper describes benefits of multilevel modeling for analyzing ERP data

• Uses example data set to show decision-making points throughout process

• Effect size calculation, visualization, and power addressed