Uncovering internal states with a robust shared-state multi-neuron GLM-HMM framework

Abstract

Neural systems exhibit multiple firing states that reflect an organism’s internal state and modulate the relationship between external environmental stimuli and behavior. Several studies have inferred these latent states by supplementing the traditional hidden Markov Model (HMM) with generalized linear models (GLMs) with non-Poisson behavioral observations. However, understanding the relationship between internal brain states and behavior also requires modeling the neural activity. Nonetheless, fitting multi-neuron GLM-HMMs is non-trivial due to high sparsity, collinearity, and low trial counts in neuronal datasets. Therefore, we built a robust multi-neuron GLM-HMM framework that uncovers latent states from population activity while incorporating the influence of time-stamped task variables and spike histories. To obtain reliable model parameters, we employ a modified expectation-maximization procedure. Specifically, we show that incorporating neuron-adaptive penalization in the maximization step overcomes the covariate co-linearity issues typical of time-stamped events and sparse spiking, yielding stable estimates of Poisson GLM coefficients. Furthermore, we incorporate a trust-region algorithm to ensure stable M-step convergence in the presence of ill-conditioned Hessians that can lead to unstable Newton-Raphson updates. We further demonstrate the utility of leave-one-out cross-validation analysis for evaluating model performance on datasets with low trial counts and without breaking their temporal structure. We evaluate our framework on three electrophysiological datasets from primates and rodents as they perform a decision-making task, demonstrate stable model convergence, and discuss the behavioral relevance of the inferred states.

Author Summary:

Neural systems evolve over time: not only do the individual neurons influence each other across the network, but the network and interconnections themselves change as an animal enters different behaviors (e.g., attentive vs. disengaged) or states (e.g., hungry or tired). Analyzing the neural activity that guides behavior thus must incorporate the time-varying nature of the brain. Recent modeling work has extended the popular Generalized Linear Model, a model that can connect task and behavior to recorded neural action potentials, to incorporate a latent Hidden Markov Model. This extension allows the resulting GLM-HMM to exhibit several different relationships (different GLMs) that are switched between over time to account for the animal’s changing patterns. While GLM-HMMs have been applied extensively on behavioral data (e.g., task choice in a decision making paradigm), neural data is much more difficult due to the smaller sample sizes, sparser activity, and larger parameter space. Our work presents a new fitting approach and best practices to robustly fit GLM-HMMs to neural data. We demonstrate through numerous applications to a variety of neural datasets that by robustly fitting GLM-HMMs to data, we can identify important features of neural activity that let us better understand its relationship to behavior.

1. Introduction

Nervous systems are highly dynamic networks with rich underlying processes, often called internal states. These unobserved states modulate how environmental and physiological inputs are integrated in the brain, enabling the organism to generate appropriate behavioral and physiological responses (Karigo and Charles, 2025). With advances in high-throughput behavior-tracking techniques, internal states can be inferred from behavior by clustering similar user-identified animal pose and action features (Mathis et al., 2018; Wu et al., 2020; Pereira et al., 2022). Additionally, progress in data-driven approaches has enabled the use of unsupervised probabilistic methods, such as hidden Markov Models (HMMs), to infer underlying states from behavior. Calhoun et al. (2019) and others (Ashwood et al., 2022; Bolkan et al., 2022; Hulsey et al., 2024; Cuturela et al., 2025; Mohammadi et al., 2025; Li et al., 2023; Chen et al., 2026) have successfully combined HMMs with regression methods, such as generalized linear models (GLMs), to capture varying trial-by-trial behavioral strategies as fluctuating latent states that exhibit distinct relationships with stimuli and other covariates, including internal goals and needs. Such studies have challenged the classical computational models of perceptual decision-making that assume the same trial-based strategy (Green and Swets, 1966; Ratcliff and Rouder, 1998; Ratcliff and McKoon, 2008; Gold and Shadlen, 2007). However, inferred states from behavioral datasets cannot capture the motivational differences between poses or actions that look similar; for instance, mounting behavior could suggest aggression or mating (Karigo et al., 2021). Thus, internal state information from poses and movements must be supplemented with neuronal computations to achieve a comprehensive understanding of complex behaviors.

Prior work suggests that behavioral states are encoded and can thus be predicted from the neuronal population dynamics. For instance, Gründemann et al. (2019) showed that two distinct and antagonistic subpopulations of basal amygdala neurons drive exploratory and defensive behaviors in mice. Similar results have been reported in fish (Marques et al., 2020) and worms (Ji et al., 2021). Furthermore, using artificial stimulation techniques of subpopulations of neurons projecting to multiple brain regions, scientists have successfully induced brain states that influence a range of behavioral features like hunger, thirst, aversion, aggression, courtship, and attention (Aponte et al., 2011; Berrios et al., 2021; Livneh et al., 2017, 2020; Chen et al., 2019; Churgin et al., 2017; Flavell et al., 2013; Ji et al., 2021; Asahina et al., 2014; Anderson, 2016; Clowney et al., 2015; Hindmarsh Sten et al., 2021). Thus, uncovering ensemble activity patterns through unsupervised machine-learning algorithms like GLM-HMMs can help quantify the relationship between underlying neuronal dynamics and behavior.

Escola et al. (2011) first introduced GLM-HMMs for neural activity. However, while behavioral GLM-HMMs have found widespread use in explaining strategy switches across trials, as discussed above, current approaches for leveraging neuronal GLM-HMMs to do the same can be unstable, prohibiting their adoption. Specifically, current fitting methods for GLM-HMMs applied to neuronal data are sensitive to data statistics, parameter configurations, and the amount of data available for model training. These challenges arise as neural data is often degenerate due to sparse or highly correlated activity, has limited trials, and is higher-dimensional.

To remedy the above limitations, we modify the GLM-HMM fitting framework to be more stable under degenerate data and small sample sizes. First, we introduce a model penalization term to handle overfitting and collinear features. Next, we stabilize M-step convergence for ill-conditioned Hessians, and lastly, we assess model fit beyond a conventional train-test split for datasets with low trial counts. We test our model improvements on three electrophysiology datasets in which animals engage in decision-making, demonstrating not only stable fitting but also meaningful latent-state interpretations of behavior.

In this work, we first give an overview of the use of GLMs and HMMs on neuronal datasets in Section 2. Section 3 then provides the mathematical overview of the shared-state population GLM-HMM framework. In Section 4 we present our proposed improvements for a robust GLM-HMM framework. In Section 5, we then investigate the application of our improved shared-state GLM-HMM fitting on three electrophysiological datasets from 1) rodent medial prefrontal cortex (Peyrache et al., 2009, 2018), 2) primate premotor cortex (Lawlor et al., 2018; Perich et al., 2018a), and 3) rodent anterior insula. We assess our model’s fit and compare it with the previous GLM-HMM framework (Escola et al., 2011) across all datasets to highlight the benefits of each design choice. We also discuss meaningful state interpretations in relation to behavior. Lastly, in Section 6, we conclude with a brief summary of the methodological advances introduced by our framework and its applications in neuroscience.

2. Background

The GLM-HMM model, as per its name, comprises two main components: a Hidden Markov Model and a Generalized Linear Model. Here we review these components and define notations for the rest of the paper.

Generalized Linear Models

Generalized linear models allow for linear predictors to be defined under non-Gaussian models. In general, these models define a likelihood distribution of the data (indexed by $t$, which will later correspond to time) ${\mathit{y}}_t$ as $p\left({\mathit{y}}_t\mid {\mathit{x}}_t;\theta \right)$, where ${\mathit{x}}_t$ is a vector of regressors and $\mathit{\theta }$ is the set of model parameters, and the expectation of ${\mathit{y}}_t$ is a linear function of ${\mathit{z}}_t$:

$$g\left(\mathbb{E}\left[{\mathit{y}}_t\mid {\mathit{z}}_t\right]\right)={\mathit{\theta }}^T{\mathit{x}}_t$$ (1)

where here $g(\cdot )$ is referred to as the link function. Thus the mean of the likelihood is a linear-nonlinear function of the regressors ${\mathit{x}}_t$.

Here we are specifically interested in fitting models to spike data, describing spike counts as a function of task variables and spike history. In this case, the likelihood function over the emission matrices follows a Poisson distribution, such that the probability of observing $y_t$ spikes from a given neuron at time $t$ is given by

$$f_y\left(y_t\right)=\frac{{\left({\lambda }_t\right)}^{y_t}\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\lambda }_t\right)}{y_t!},y_t\in \left\{0,1,2,\dots \right\},$$ (2)

where ${\lambda }_t$ is the firing rate of the neuron in time-bin $t$. By the GLM assumption we have, taking the typical $\mathrm{l}\mathrm{o}\mathrm{g}(\cdot )$ link function

$${\lambda }_t=\exp \left({\mathit{\theta }}^{\mathrm{T}}{\mathit{x}}_t\right).$$ (3)

Here, $\mathit{\theta }\in {\mathbb{R}}^D$ represents the weight vector and ${\mathit{x}}_t\in {\mathbb{R}}^D$ is the covariate vector at time $t$. ${\mathit{x}}_t$ includes variables such as the task variables and spike history and potentially spiking activity of other neurons (Paninski, 2004). The spike history component consists of spike counts $\tau$ steps prior to $t$: ${\mathbf{y}}_{t-\tau :t-1}=\left(y_{t-1},\dots .,y_{t-\tau }\right)$.

$$\mathrm{l}\mathrm{o}\mathrm{g}\left({\lambda }_t\right)=\sum _{k=1}^K{h}_k^{\mathit{hist}}{y}_{t-k}+\sum _{k=0}^{K-1}{h}_k^{\mathit{task}}{x}_{t-k}={\mathit{h}}^{\mathit{hist}}\mathrm{*}\mathit{y}+{\mathit{h}}^{\mathit{task}}\mathrm{*}\mathit{x},$$ (4)

where ${\mathit{h}}^{\mathit{hist}}$ and ${\mathit{h}}^{\mathit{task}}$ are length-$K$ filters that dictate how the past history and task variables, respectively, influence the current spike rate. Fitting the filters as a function of time can be sensitive to noise and available data, so often a prior is used wherein the filters are expanded into a set of $L$ basis elements $\mathbf{\Phi }=\left[{\mathit{\varphi }}_1,{\mathit{\varphi }}_2,\dots ,{\mathit{\varphi }}_L\right]\in {\mathbb{R}}^{K\times L}$ as

$${\mathit{h}}^{hist}=\mathbf{\Phi }{\mathit{a}}^{hist}=\sum _{l=1}^L{\mathit{\varphi }}_l{a}_l^{hist},$$ (5)

for ${\mathit{h}}^{\mathit{hist}}$ and similarly

$${\mathit{h}}^{\mathit{task}}=\mathbf{\Phi }{\mathit{a}}^{\mathit{task}}=\sum _{l=1}^L{\mathit{\varphi }}_l{a}_l^{\mathit{task}}$$ (6)

for ${\mathit{h}}^{\mathit{task}}$. Here the coefficients ${\mathit{a}}^{\mathit{hist}}$ and ${\mathit{a}}^{\mathit{task}}$ represent how much of each basis element is used to construct each filter, and are the free parameters in the GLM that are to be fit. With estimates of ${\mathit{a}}^{\mathit{hist}}$ and ${\mathit{a}}^{\mathit{task}}$, we can reconstruct the actual filters ${\mathit{h}}^{\mathit{hist}}$ and ${\mathit{h}}^{\mathit{task}}$, respectively as ${\mathit{h}}^{\mathit{hist}}=\mathbf{\Phi }{\mathit{a}}^{\mathit{hist}}$ and ${\mathit{h}}^{\mathit{task}}=\mathbf{\Phi }{\mathit{a}}^{\mathit{task}}$. With these definitions we can rewrite the GLM expression for the log-expectation in terms of ${\mathit{a}}^{\mathit{hist}}$ and ${\mathit{a}}^{\mathit{task}}$ as

$$\begin{array}{ll}\mathrm{l}\mathrm{o}\mathrm{g}\left({\lambda }_t\right)& =\left(\sum _{l=1}^L{\varphi }_l{a}_l^{\mathit{hist}}\right)\mathrm{*}\mathit{y}+\left(\sum _{l=1}^L{\varphi }_l{a}_l^{\mathit{task}}\right)\mathrm{*}\mathit{x}\\ & =\left(\sum _{l=1}^L{\varphi }_l\mathrm{*}\mathit{y}\right)a_l^{\mathit{hist}}+\left(\sum _{l=1}^L{\varphi }_l\mathrm{*}\mathit{x}\right)a_l^{\mathit{task}},\end{array}$$ (7)

where we can reorganize the terms due to the linearity of convolution. This expression means that the regressor for each basis of the basis expansion of the kernels $\mathit{h}$ are the convolution of the corresponding time-series (i.e., the spike history or the task inputs/outputs) with the corresponding basis elements. Following earlier (Park et al., 2014) in GLMs for neural activity, we use as the basis vectors ${\varphi }_l$ for $l=1,2,\dots ,L$, a set of raised cosine basis functions given by

$${\varphi }_l(\tau )=\left\{\begin{array}{ll}\frac{1}{2}\left[\mathrm{c}\mathrm{o}\mathrm{s}\left(\pi \frac{\mathrm{l}\mathrm{o}\mathrm{g}(\tau +\delta )-\mathrm{l}\mathrm{o}\mathrm{g}\left(c_l+\delta \right)}{\mathrm{\Delta }}\right)+1\right],& \text{if}\left|\mathrm{l}\mathrm{o}\mathrm{g}(\tau +\delta )-\mathrm{l}\mathrm{o}\mathrm{g}\left(c_l+\delta \right)\right|\le \mathrm{\Delta }\\ 0,& \text{otherwise}\end{array}\right.$$ (8)

where $\delta$ is the small positive offset to prevent divergence near zero, $c_l$ is the center of the $l^{\mathit{th}}$ basis function and $\mathrm{\Delta }$ is the non-linear spacing between the bases. Optimized versions of Poisson GLMs have also been proposed (Medvedeva et al., 2025) that overcome binning of spike data to construct efficient design matrices and enable precise functional connectivity inference using Laguerre polynomials as basis functions.

While the GLM described above is a good local model for any particular time-point, we note that the specific interactions learned in $\mathit{\theta }$ might change over time. These changes can be a consequence of learning, adaptation, neuromodulation, or context changes in the brain (among other possibilities). Thus neural time-series might be better represented by a set of $S$ GLMs where each $y_t$ is well represented by one of the $S$ GLMs for each $t$. We can denote these different GLMs by indexing the model parameters, i.e., ${\mathit{\theta }}_s$ for $s=1,2,\dots ,S$. However, we do not know a priori which GLM best describes each data point (i.e., which state $s$ is active at a given time $t$). Thus, the GLM-HMM uses a hierarchical model to overlay a Hidden Markov Model over the state $z_t$ at each time point $t$, where $z_t$ can take one of the $S$ discrete states $\{1,2,\dots .S\}$.

Hidden Markov Models

HMMs (Rabiner, 1989) build on the assumption that the state at any time point $t$, denoted $z_t$, is drawn from $\{1,2,\dots .S\}$, and extends a state transition assumption that connects the state at any time point to the preceding state. This forms a consistency of states: a state is more consistent if the probability that $z_t=z_{t-1}$ is high. First, we note that the Markovian assumption of the HMM means that we can write the conditional probability of the state at time $t$ as only a function of the previous state; i.e.,

$$p\left(z_t\mid \left\{z_0,z_1,\dots ,z_{t-1}\right\},\left\{{\mathit{y}}_0,{\mathit{y}}_1,\dots ,{\mathit{y}}_{t-1}\right\}\right)=p\left(z_t\mid z_{t-1}\right).$$ (9)

Thus the current state only depends on the state at time $t-1$ without any impact of the emissions. Next, we define the transition probabilities via the transition matrix $\mathit{A}\in {\mathbb{R}}^{S\times S}$, where

$$A_{rs}=p\left(z_t=s\mid z_{t-1}=r\right).$$ (10)

Each entry $0\le A_{rs}\le 1$ captures the probability of transitioning from state $s$ (the column) to state $r$ (the row), and therefore the diagonal $A_{rr}$ entries are the “stickiness” of the model, or the probability of staying within a given state $r$. $\mathit{A}$ is homogeneous in that it is assumed constant for all time points. Traditionally, the probability of observing $y_t$ only depends on the current state $z_t$ and not on past states or emission sequences:

$$p\left(y_t\mid \left\{z_0,z_1,\dots ,z_{t-1}\right\},\left\{{\mathit{y}}_0,{\mathit{y}}_1,\dots ,{\mathit{y}}_{t-1}\right\}\right)=p\left(y_t\mid z_t\right).$$ (11)

Thus, the full log-likelihood of the model over all the sequence of states is given by

$$\mathrm{l}\mathrm{o}\mathrm{g}p(\mathit{y}\mid \mathit{A},\mathit{\pi })=\mathrm{l}\mathrm{o}\mathrm{g}\sum _{z_0,..,z_T}\left({\pi }_{z_0}\prod _{t=1}^T{A}_{z_{t-1},z_t}\prod _{t=0}^{T}p\left(y_t=k\mid z_t=s\right)\right),$$ (12)

where the system has $K$ discrete emission classes $\{0,1,2,\dots .K\}$, and $\mathit{\pi }$ is the initial state probability distribution vector. These model parameters are learned from the data by maximizing Equation (12).

GLM-HMMs

In combining the GLM and HMM models (introduced first by Bengio and Frasconi (1995)), the main modeling specification is that each distribution $p\left(y_t=k\mid z_t=s\right)$ is defined as a GLM with parameters indexed by the state: $\mathit{\theta }\to {\mathit{\theta }}_s$. In the case of behavioral models, this is, e.g., the logistic distribution (e.g., Calhoun et al. (2019)). Here we follow the above Poisson GLM with the log link function (Simoncelli et al., 2004; Paninski, 2004; Truccolo et al., 2005). This means that each state is characterized by its own set of filters: a per-state spike-history filter, a task-variable filter, etc. as described above. Thus the model can capture different modes of the network, such as if a given task variable sometimes excites, and sometimes inhibits, the same neuron. Such a multi-model response (potentially dependent on contextual or neural cues not observed in the experiment) cannot be captured with a single GLM.

3. Shared-state multi-neuron GLM-HMM model

The HMM allows us to find hidden patterns in simultaneously recorded neuronal data, while the GLM provides a way of integrating the task variables and spike history information to understand how the states influence their processing. For ease of notation in the subsequent exposition we define: ${𝒴}^{\left(n\right)}=\{y_0^{\left(n\right)},\dots ,y_T^{\left(n\right)}\}$ as the full set of activity for neuron $n,𝒴=\bigcup _{n=1}^N{𝒴}^{\left(n\right)}$ as the full set of neural activity across all neurons, $𝒳=\left\{{\mathit{x}}_0,\dots ,{\mathit{x}}_T\right\}$ as the full set of inputs across all time, $𝒵=\left\{z_0,\dots ,z_T\right\}$ as the set of all latent states across time, and $\mathbf{\Theta }=\left\{{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_S\right\}$ be the full set of GLM parameter sets across all states.

One additional assumption that can be made is that all simultaneously recorded neurons share a state; i.e., if we assume $N$ simultaneously recorded neurons whose spikes are recorded at $T$ time points, the probability of observing the ensemble at a given state $z_t$ and the predictors $𝒳$ across all time and neurons is,

$$p\left({𝒴}_t\mid z_t,{\mathit{x}}_t\right)=\prod _{n=1}^{N}p(y_t^{\left(n\right)}\mid z_t,{\mathit{x}}_t),$$ (13)

such that each cell’s emission follows a Poisson Distribution given by Equation (2). Thus, the total log-likelihood of the model across all neurons, states and time bins becomes,

$$\mathrm{l}\mathrm{o}\mathrm{g}p(𝒴\mid \mathit{A},\mathit{\pi },𝒳)=\mathrm{l}\mathrm{o}\mathrm{g}\sum _{𝒵}\left({\pi }_{z_0}\prod _{t=1}^T{A}_{z_{t-1},z_t}\prod _{n=1}^N\prod _{t=0}^{T}p(y_t^{\left(n\right)}\mid z_t,{\mathit{x}}_t)\right).$$ (14)

Baum-Welch Expectation-Maximization algorithm

To fit the model parameters, we would like to maximize the marginal likelihood function in Equation (14). However, this marginal likelihood is intractable to optimize, preventing direct inference. Instead, as with prior work, we turn to the standard Expectation-Maximization (EM) algorithm. Specifically, we use the iterative Baum-Welch EM algorithm (Dempster et al., 1977) to fit the model parameters and optimize the data likelihood. EM proceeds in two steps: During the E-step (Expectation), the posterior probabilities of the latent states are computed given the data and parameters, while in the M-step (Maximization), the next set of parameters is computed by minimizing the negative log-likelihood with respect to the current parameters, using the posterior probability distributions from the E step. These steps are iterated over until a convergence criterion is met.

For ease of notation in the subsequent exposition, for each $𝒳$, $𝒵$, $𝒴$, and ${𝒴}^{\left(n\right)}$ we use the subscripts $<t$, $\le t$, $>t$, $\ge t$, and $\ne t$ to indicate the subset of that includes members of the set with index satisfying the subscripťs inequality. Mathematically we can write

$$\begin{array}{c}\text{E-step:}\left\{{\gamma }_{ts}\right\}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\gamma }{\mathrm{m}\mathrm{i}\mathrm{n}}{D}_{KL}\left[\gamma \Vert p\left[z_{tr}=s\mid 𝒳,𝒴,\mathbf{\Theta }\right]\right]\\ \text{M-step:}{\mathit{\theta }}_s=\arg \underset{{\mathit{\theta }}_s}{\mathrm{m}\mathrm{a}\mathrm{x}}{\mathbb{E}}_{\gamma }\left[\log p\left(𝒳,𝒴,z_t=s\mid {\mathit{\theta }}_s\right)\right].\end{array}$$ (15)

In these equations ${\gamma }_{ts}$ represents the posterior of the active state at time $t$ being $s$ (i.e., conditioned on the data including the task time series ${\mathit{x}}_t$ and neural activity $y_t$, and $D_{KL}(p\Vert q)$ is the KL-divergence between two distributions.

E-step

The E-step in the EM algorithm infers the probability of state membership at each time-point with old model parameters using the forward-backward algorithm (Baum et al., 1970). Effectively the approach approximates the posterior ${\gamma }_{tr}=p\left(z_t=r\mid 𝒴,𝒳,\mathbf{\Theta }\right)$.

To compute ${\gamma }_{tr}$, we note that the marginal posterior probability of being in state $r$ at time $t$ can be expressed in a form that separates the conditional information from the past and future time-stems. Specifically,

$$\begin{array}{ll}{\gamma }_{tr}& =p\left(z_t=r\mid 𝒴,\mathbf{\Theta },\mathit{A}\right)\\ & =\frac{p\left(𝒴\mid \mathbf{\Theta },\mathit{A}{z}_t=r\right)p\left(z_t=r\right)}{p(𝒴\mid \mathbf{\Theta },\mathit{A})}\\ & =\frac{p\left({𝒴}_{\le t},z_t=r\mid \mathbf{\Theta },\mathit{A}{z}_t=r\right)p\left({𝒴}_{>t}\mid \mathbf{\Theta },\mathit{A},z_t=r\right)}{\sum _{s=1}^{S}p\left(𝒴|\mathbf{\Theta },\mathit{A}|z_t=s\right)p\left(z_t=s\right)}\\ & =\frac{p\left({𝒴}_{\le t}\mid \mathbf{\Theta },\mathit{A}{z}_t=r\right)p\left({𝒴}_{>t}\mid \mathbf{\Theta },\mathit{A},z_t=r\right)}{\sum _{s=1}^{S}p\left({𝒴}_{\le t},z_t=s\mid \mathbf{\Theta },\mathit{A}\right)p\left({𝒴}_{>t}\mid \mathbf{\Theta },\mathit{A},z_t=s\right)}\\ & =\frac{{\alpha }_{tr}{\beta }_{tr}}{\sum _{s=1}^S{\alpha }_{ts}{\beta }_{ts}},\end{array}$$ (16)

where ${\alpha }_{tr}$ and ${\beta }_{tr}$ are the forward and backwards probabilities, respectively, defined as

$${\alpha }_{tr}=p\left({𝒴}_{\le t},z_t=r\right)$$ (17)

$${\beta }_{tr}=p\left({𝒴}_{>t}\mid z_t=r\right).$$ (18)

These probabilities are convenient as they can be recursively computed in an efficient manner. For the forward probability, ${\alpha }_t\left(s\right)$, we can write the distribution as a marginal over all past states,

$${\alpha }_{ts}=\sum _{{𝒵}_{<t}}p({𝒴}^{\left(n\right)},{𝒵}_{<t},z_t=s\mid \mathbf{\Theta }).$$ (19)

At $t=0$, this probability is simply

$$({\alpha }_{0s}=p\left(y_0,z_0=s\right)=p\left(z_0=s\right)p\left(y_0\mid z_0=s\right).$$ (20)

For $t>0$, we can use the Markov property of the state transition in $z_t$ to write

$${\alpha }_{ts}=p\left({𝒴}_{\le t},z_t=s\right)=\sum _{r=1}^{S}p\left({𝒴}_{\le t},z_{t-1}=r,z_t=s\right).$$ (21)

Using the chain rule and the Markovian property from Equations. (9) and (11) in Equation. (21) yields

$$\begin{array}{ll}{\alpha }_{ts}& =p\left(y_t\mid z_t=s\right)p\left(z_t=s\mid z_{t-1}=r\right)p\left(z_{t-1}=r,{𝒴}_{<t}\right)\\ & =p\left(y_t\mid z_t=s\right)\sum _{r=1}^{S}p\left(z_t=s\mid z_{t-1}=r\right){\alpha }_{(t-1)r}\\ & =p\left(y_t\mid z_t=s\right)\sum _{r=1}^S{\alpha }_{(t-1)r}{A}_{rs},\end{array}$$ (22)

meaning that the current probability of the emission and state depends only on the past distribution of the emissions passed through the HMM state transition matrix.

For the backward probabilities we can similarly express the distribution as the marginal over the next state

$${\beta }_{tr}=\sum _{s=1}^{S}p\left({𝒴}_{>t},z_{t+1}=s\mid z_t=r\right).$$ (23)

After applying chain rule and using the Markovian property from Equation (11),

$${\beta }_{tr}=\sum _{s=1}^{S}p\left({𝒴}_{>t}\mid z_{t+1}=s\right)p\left(z_{t+1}=s\mid z_t=r\right),$$ (24)

such that,

$${\beta }_{tr}=\sum _{s=1}^{S}p\left(y_{t+1}\mid z_{t+1}=s\right){\beta }_{(t+1)s}{A}_{rs}.$$ (25)

Together, ${\alpha }_{tr}$ and ${\beta }_{tr}$ can then be used to compute the esitmates ${\gamma }_{tr}$ as in Equation (16).

M-step

In the M-step we use the estimated state probabilities from the E-step in order to update the model parameters: the transition matrix $\mathit{A}$ and the GLM weights ${\mathit{\theta }}_r^{\left(n\right)}$ for each state $r=1,\dots ,S$ and each neuron $n=1,\dots ,N$. The pairwise posterior probability for all neurons at time $t$ is given as,

$${\xi }_{t,rs}=p\left(z_t=r,z_{t+1}=s\mid 𝒴\right)=\frac{p\left(z_t=r,z_{t+1}=s,𝒴\right)}{p\left(𝒴\right)}.$$ (26)

By expanding the numerator and applying the chain rule,

$$\left.{\xi }_{t,rs}=p\left({𝒴}_{t+1}\mid z_{t+1}=s\right){\beta }_{(t+1)s}\right){\alpha }_{tr}{A}_{rs},$$ (27)

or,

$${\xi }_{t,rs}={\beta }_{(t+1)s}{\alpha }_{tr}{A}_{rs}\prod _{n=1}^{N}p(y_{t+1}^{\left(n\right)}\mid z_{t+1}=s,{\mathit{x}}_{t+1},{\mathit{\theta }}_s^{\left(n\right)}).$$ (28)

The new transition matrix for the next EM iteration is

$$A_{rs}=\frac{\sum _t{\xi }_{t,rs}}{\sum _t{\gamma }_{tr}}.$$ (29)

For each state $r$ and neuron $n$, the GLM weights ${\mathit{\theta }}_r^{\left(n\right)}$ are updated by maximizing the weighted Poisson log-likelihood:

$${\mathit{\theta }}_r^{\left(n\right)}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathit{\theta }}_r^{\left(n\right)}}{\mathrm{m}\mathrm{a}\mathrm{x}}\sum _{t=0}^{T}p\left(z_t=r\right)\mathrm{l}\mathrm{o}\mathrm{g}\left[p({𝒴}^{\left(n\right)}\mid z_t=r,{\mathit{\theta }}_r^{\left(n\right)})\right]$$ (30)

$$=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathit{\theta }}_r^{\left(n\right)}}{\mathrm{m}\mathrm{a}\mathrm{x}}\left[\sum _{t=0}^T{\gamma }_{tr}\left(y_t^{\left(n\right)}{\mathit{\theta }}_r^{\left(n\right)\mathrm{T}}{\mathit{x}}_t-\mathrm{e}\mathrm{x}\mathrm{p}({\mathit{\theta }}_r^{\left(n\right)\mathrm{T}}{\mathit{x}}_t)\right)\right].$$ (31)

To prevent underflow due to the multiplication of probabilities, Equations (22), (25), (16), and (28) are transformed to the log scale and rescaled for numerical stability (see Supplement).

4. Robust GLM-HMM framework

Our robust GMM-HMM fitting framework aims to 1) improve robustness to variability in neuronal activity levels, 2) decrease the sensitivity to the expected ill-conditioned data Hessians, and 3) enable proper model fitting for data-limited experiments.

Model initialization with neuron-adaptive regularization

Most electrophysiology studies investigate neuronal responses aligned to discrete time-stamped events. To fit a GLM to neuron datasets, the event predictors are binned at millisecond resolution and, to generate the per-basis coefficient regressors as per Equation (7), often convolved with a set of basis functions. The resulting covariate matrices, unfortunately, are typically highly correlated and sparse. Additionally, sparsely firing neurons may be active around certain isolated task events, which can lead to overfitting and the assignment of biologically implausible stimulus coefficients. In contrast, behavioral GLMs often have more well-behaved, continuously varying covariates, which generally produce better-conditioned design matrices and are less susceptible to coefficient inflation.

In the GLM-HMM framework, the issue of poor covariate conditioning is particularly relevant in the M-step, where each neuron has its own state-specific GLM. As a non-convex optimization, parameter initialization can have a strong impact on the estimation stability, especially under the neuron-by-neuron firing rate variability. We thus sought to improve estimation stability while accounting for neuron-specific firing statistics by initializing the GLM weights via a per-neuron, single-state initial parameter fitting. This procedure follows prior initialization approaches in the GLM-HMM literature (Ashwood et al., 2022) with the exception that we introduce a ridge-regularizion term via a Gaussian prior over the weights to induce stable initialization. Mathematically, our modification changes Equation (31) to minimize the log-posterior (including the log-prior as the regularization term) probability as

$$\underset{{\mathit{\theta }}_r^{\left(n\right)}}{\mathrm{m}\mathrm{i}\mathrm{n}}L(\mathit{\theta })=\underset{{\mathit{\theta }}_r^{\left(n\right)}}{\mathrm{m}\mathrm{i}\mathrm{n}}\left[\sum _{t=0}^T{\gamma }_{tr}\left(\mathrm{e}\mathrm{x}\mathrm{p}({\mathit{\theta }}_r^{\left(n\right)\mathrm{T}}{\mathit{x}}_t)-y_t^{\left(n\right)}{\mathit{\theta }}_r^{\left(n\right)\mathrm{T}}{\mathit{x}}_t\right)+{\rho }^{\left(n\right)}{\Vert {\mathit{\theta }}_r^{\left(n\right)}\Vert }_2^2\right],$$ (32)

where ${\rho }^{\left(n\right)}$ is the regularization strength for the $n^{\mathit{th}}$ neuron, which was solved using the trust-region method (see below). Ashwood et al. (2022) noted that the parameter fits across states can be highly similar, which we also found when including the regularization. We thus also used (as in Ashwood et al. (2022)) an independent Gaussian perturbation scaled to 25% of the norm of the baseline solution to break symmetry between states.

During the initialization we also set the regularization parameter for each neuron ${\rho }^{\left(n\right)}$ by fitting single-state, ridge-regularized GLM with five-fold cross-validation. The neuron-specific ridge penalty selected during initialization is retained and used in M-step optimization, which improves numerical stability and keeps the coefficients well-constrained.

Robust M-step convergence using the trust-region method

The weights for each neuron $n$ in a particular state are updated for each EM iteration by the above optimizing Equation (32) that minimizes the negative log-posterior $L\left(\mathit{\theta }\right)$. Optimizing this code function depends on computing the gradient ${\nabla }_{\theta }L\left(\mathit{\theta }\right)$ and Hessian $\left.\mathit{H}={\nabla }_{\theta }^{2}L\left(\mathit{\theta }\right)\right)$, which are expressed as:

$$\nabla L(\mathit{\theta })=\sum _t{\gamma }_{tr}\left({\mathit{x}}_t\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathit{x}}_t^{\mathrm{T}}\mathit{\theta }\right)-y_t^{\left(n\right)}\left({\mathit{x}}_t^{\mathrm{T}}\right)\right)+2\mathit{\theta }\rho ,$$ (33)

$${\nabla }^{2}L(\mathit{\theta })=\mathit{H}=\sum _t{\gamma }_{tr}\left({\mathit{x}}_t{\mathit{x}}_t^{\mathrm{T}}\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathit{x}}_t^{\mathrm{T}}\mathit{\theta }\right)\right)+2\rho I.$$ (34)

The Newton-Raphson method approximates the optimization function $L\left(\mathit{\theta }\right)$ locally about the current estimate of $\mathit{\theta }$ with a quadratic surrogate function. This surrogate allows for an efficient update that accounts for both the gradient and curvature via a Newton iteration as:

$${\mathit{\theta }}_{new}=\mathit{\theta }-{\mathit{H}}^{-1}\nabla L\left(\mathit{\theta }\right).$$ (35)

However, if the local curvature is anisotropic, $\mathit{H}$ can become ill-conditioned, resulting in a large Newton step and unstable weight estimates.

To prevent such large weight updates during optimization, we instead use a trust-region method. At each iteration, a second-order Taylor approximation of the objective function $L\left(\mathit{\theta }\right)$ is computed, given as

$$L\left({\mathit{\theta }}_{new}\right)=L(\mathit{\theta }+\mathit{s})\approx L\left(\mathit{\theta }\right)+\nabla L\left({\mathit{\theta }}^{\mathrm{T}}\right)\mathit{s}+\frac{1}{2}{\mathit{s}}^{\mathrm{T}}\mathit{H}\mathit{s}.$$ (36)

By ensuring that the step, $\Vert s{\Vert }_2<=\mathrm{\Delta }$, where $\mathrm{\Delta }$ is the radius within which the step is taken, the step is computed as

$$\mathit{s}=-{(\mathit{H}+uI)}^{-1}\nabla L(\mathit{\theta }),$$ (37)

for some $u\ge 0$ such that $\Vert \mathit{s}\Vert \le \mathrm{\Delta }$. If $L(\mathit{\theta }+\mathit{s})>L\left(\mathit{\theta }\right)$, the region of trust is shrunk and vice-versa. The trust-region optimization was implemented using MATLAB’s fminunc solver with the trust-region algorithm in the M-step.

Model fit cross-validation with minimal data

Conventional k-fold cross-validation may not be optimal for trial-based electrophysiology datasets due to two reasons: 1) randomly partitioning temporally-structured observations can disrupt within-trial dependencies that are necessary for GLM-HMM inference, and 2) low-trial-count datasets may not provide enough data for appropriately training the model.

We therefore use a leave-one-trial-out cross-validation procedure for assessing our model. Specifically, if a dataset has $N$ trials, we train our model on $N-1$ trials, test it on the $N^{\mathit{th}}$ trial, and repeat until all trials are tested. For computing the log-likelihood of the held-out test trial, we run only the forward pass and use the parameters estimated after training the model on $N-1$ trials. For each held-out test trial, predictive performance is quantified in bits per second, described by Calhoun et al. (2019) as:

$$\frac{L_{test}-L_{null}}{\mathrm{l}\mathrm{o}\mathrm{g}\left(2\right)T},$$ (38)

where

$$L_{\mathit{test}}=\sum _n\sum _t\log p(y_{t,\mathit{test}}^{\left(n\right)}\mid \mathit{A},{\mathit{\theta }}_s),$$ (39)

$$L_{\mathit{null}}=\sum _n\sum _t(y_{t,\mathit{test}}^{\left(n\right)}\mathrm{l}\mathrm{o}\mathrm{g}{\lambda }_{0,\mathit{train}}^{\left(n\right)}-{\lambda }_{0,\mathit{train}}^{\left(n\right)}),$$ (40)

where ${\lambda }_{0,\mathit{train}}^{\left(n\right)}$ is the mean firing rate of the $n^{\mathit{th}}$ neuron in the training trials, and $T$ is the total trial duration of the held-out test trial. To assess if the predictive performance is statistically significant, Equation (38) is recomputed on the held-out test trials for the model trained on shuffled neuronal spikes per trial.

5. Fitting to electrophysiology data

In this section, we apply our framework to three electrophysiology datasets, including neurons with a mean firing rate of ≥ 0.2 Hz across the session, to assess its ability to learn the model parameters appropriately and converge stably.

Medial prefrontal cortex during a contingency task

We first applied our methods to a medial prefrontal cortex (mPFC) dataset in which freely moving rats performed an attentional shift task (Peyrache et al., 2009, 2018). Each trial started with the rat in the root of a Y-maze. According to the spatial contingency rule, one of the choice arms (the right arm in the dataset) was rewarded. We trained our model to a session where the animal performed 22 trials (13 correct and 9 incorrect) and had 31 neurons recorded simultaneously. To prepare the predictor matrix, trials were concatenated across the session and binned into 50-millisecond time bins. Timestamps for trial start and end for choice and outcome encoding, respectively, along with the spike history were used predictors and convolved with 5 raised-cosine basis functions with a time support of 2.5 seconds. No position information was given to the model. Since the true number of hidden states was unknown, we considered $S\in \{3,4,5\}$. While the negative log-likelihood of our model decreased as $S$ increased, the state identities became less clearly defined (Supp. Fig 1). Therefore, we decided to select the 4-state model.

Our model revealed that the recruited states sequences differed between the correct and incorrect trial types (Fig 1). Importantly, at trial start and end, when the animal attended to the contingency, went to the correct arm, and received the reward, state 3 was on, whereas when the animal went to the incorrect arm, state 2 was on. To gain further insight into population dynamics, we investigated the population-averaged kernels at trial start for states 2 and 3. The kernel modulation was higher and positive in state 3 than in state 2 (see the insets in Fig 1), highlighting active engagement of the mPFC at trial start. These results corroborate prior studies implicating the mPFC in the representation of reward-associated memories during task performance (Birrell and Brown, 2000; Floresco et al., 1999; Jones and Wilson, 2005).

Figure 1. Attention-contingent state recruitment in the mPFC.

Three example correct (turn right) and incorrect (turn left) animal position trajectories color-coded by the highest probability state in an attentional shift task. Insets show population-averaged kernel deviations for states 3 and 2, associated with correct and incorrect trial starts, respectively (Mean±SEM).

Additionally, our model showed an appropriate decrease in the negative log-likelihood and squared error in the model parameters across the EM iterations, highlighting stable convergence (Fig 2A,B). Furthermore, the average per-neuron correlation between the GLM weights obtained from the M-step were minimally correlated (Fig 2C), proving that the states were non-redundant. To establish that our model could extract meaningful information when trained on such low trial counts, we performed a held-out trial analysis where we trained the model on 21 trials and used the model parameters to estimate the log-likelihood on the 22^nd held-out trial (Fig 2D). Our model showed a significantly higher held-out likelihood on the test trial than when it was trained on shuffled spike data (Wilcoxon signed-rank test).

Figure 2. Shared 4-state GLM-HMM validation and convergence evaluation for rodent mPFC activity.

(A) Negative log-likelihood across EM iterations. (B) Change in model parameters ($\mathit{A}$, $\mathbf{\Theta }$) across EM iterations. (C) Mean GLM weight correlation between states per neuron. (D) Held-out log-likelihood for model trained real and shuffled spikes from 21 trials and tested on 22^nd trial (Wilcoxon signed-rank test). Each dot represents the held out trial.

The initial model weights for each neuron were calculated by fitting a Poisson GLM with a ridge penalty with five-fold cross-validation. Subsequently, this neuron-adaptive regularization was used in the M-step to ensure that the weights remained within biological limits. We compared our model initialization with two alternatives: (1) initialization using an unregularized GLM, and (2) random initialization without GLM-based fitting. Fig 3A shows the distribution of GLM weights for each method for the final EM iteration. The unregularized GLM initialization produced heavy-tailed weight distributions, indicating that a subset of coefficients reached extreme magnitudes due to overfitting. The random initialization coefficients, while not as extreme as those of the unregularized GLM, still contained a subset of high, biologically implausible coefficients. In contrast, our ridge-regularized initialization resulted in well-bound weight distributions.

Figure 3. Comparison of initialization and optimization strategies for GLM-HMM fitted on mPFC activity.

(A) Probability density distribution of final absolute GLM weights in log10 scale across all neurons using penalized GLM, unpenalized GLM, and random initializations. (B) Negative log-likelihood comparison across iterations when using trust-region and Newton-Raphson methods for M-step convergence. (C) Fraction of non-finite GLM weights across EM iterations when using trust-region and Newton-Raphson methods in the M-step.

To evaluate the convergence of our shared 4-state model, we compared optimization using the trust-region method in the M-step to a pure Newton-Raphson approach. We tracked the negative log-likelihoods and GLM weights for every EM iteration. The negative log-likelihood exponentially decreased when the model used trust-region method unlike the Newton-Rapson method, showing that Newton-Raphson is not the best way to optimize the GLM weights in the M-step for sparse electrophysiology datasets (Fig 3B). Additionally, the Newton-Raphson updates led to numerical instability: 99% of the weights become non-finite after the second EM iteration. The trust-region method, by comparison, maintained fully finite parameter estimates for all iterations (Fig 3C).

Premotor cortex during reach task

We next applied our methods to a premotor cortex dataset in which monkeys were trained to perform a sequential reach task to move the computer cursor to a visual cue specifying the target location. Four sequential correct reaches resulted in liquid reward delivery (Lawlor et al., 2018; Perich et al., 2018a). The analyzed dataset had 94 simultaneously recorded neurons and a total of 496 reaches. To prepare the predictor matrix, target reach epochs were concatenated across the session and binned into 10-millisecond time bins. Timestamps for each target onset event (approximated) and spike history were used as predictors. These events were convolved with 5 raised cosine basis functions with a time support of 0.4 seconds. We chose a 2-state model since a 3-state model exhibited strong correlations among GLM weights per neuron (Supp. Fig 2).

Our two-state model revealed that for the boundary targets (targets 1 and 4, corresponding to the movement sequence initialization and termination), state 1 was consistently recruited, whereas for intermediate targets (targets 2 and 3, corresponding to ongoing sequence execution), state 2 was recruited (Fig 4A,B). Furthermore, the population-averaged spike history kernel was significantly different for the two states (Fig 4C), which suggests that the premotor cortex dynamics differ within a trial. The consistent state recruitment across rewarded trials also shows that our model captured trial-by-trial variability in population neuronal activity without explicitly implementing dynamic time warping as previously done by Lawlor et al. (2018).

Figure 4. Consistent recruitment of distinct states for intermediate and peripheral targets across trials in a sequential target reach task.

(A) Posterior probabilities for an example trial. (B) Fraction of trials assigned to each state ±1s relative target onset (Gaussian-smoothened over 5 time bins).(C) Population-averaged spike history kernel deviations for states 1 and 2 (Mean±SEM; Wilcoxon signed-rank test).

Similar to the 4-state model trained on the mPFC dataset, our 2-state model showed an appropriate decrease in the negative log-likelihood and squared error in the model parameters across the EM iterations, highlighting stable convergence (Fig 5A, B). Furthermore, the average per-neuron correlation between the GLM weights obtained from the M-step were minimally correlated, showing state non-redundancy (Fig 5C). We also did a held-out trial analysis by training our model on a subset of reaches (133 reaches to target 2). Even with a quarter of the total reaches in the entire dataset, the model trained on real data outperformed the model trained on the shuffled dataset, suggesting that our model could capture the population activity patterns in the premotor cortex with minimal data (Fig 5D; Wilcoxon signed-rank test).

Figure 5. Convergence and validation of the 2-state GLM-HMM trained on premotor cortex data.

(A) Negative log-likelihood across EM iterations. (B) Change in model parameters ($\mathit{A}$, $\mathbf{\Theta }$) across EM iterations. (C) Mean GLM weight correlation between states per neuron. (D) Held-out log-likelihood for model trained on real and shuffled spikes from 132 target 2 reaches and tested on the 133^rd reach (Wilcoxon signed-rank test). Each dot represents the held-out trial.

We also checked how the differences in different GLM weight initializations with and without penalization impacted the final per-neuron GLM weights. Consistent with previous results from the mPFC dataset, ridge-regularized per-neuron initializations produced the most stable and biologically plausible final weights compared with unpenalized GLM or random weight initializations (Fig 6A). Lastly, we compared the convergence of the model using the trust-region optimization versus a standard Newton-Raphson method in the M-step (Fig 6B,C). Under Newton-Raphson, the negative log-likelihood increased across EM iterations, accompanied by over 50% non-finite GLM weights. In contrast, the trust-region method yielded more stable convergence with well-behaved finite parameter estimates. These results highlight the improved numerical stability of the trust-region approach.

Figure 6. Initialization and optimization testing of the shared 2-state GLM-HMM fitted on primate premotor cortex activity.

(A) Probability density distribution of final absolute GLM weights in log10 scale across all neurons using penalized GLM, unpenalized GLM, and random initializations. (B) Negative log-likelihood comparison across iterations when using trust-region and Newton-Raphson methods for M-step convergence. (C) Fraction of non-finite GLM weights across EM iterations when using trust-region and Newton-Raphson methods in the M-step.

Anterior insula during alcohol self-administration

Lastly, we applied our methods to recordings made in the anterior insula while rats self-administered alcohol in a discrete-trial task (Supp. Fig 3). Specifically, each trial started with lever insertion, which served as a cue signaling alcohol availability, and the animal made a choice to seek or abstain from engaging with the lever to receive an alcohol reward (Fig 7A). We trained our model on a dataset comprising 45 seek and 27 abstain trials, with 96 simultaneously-recorded single units. Time-stamped events, including lever insertion, lever press, lever retraction, port entry, and spike history, were convolved with five raised-cosine basis functions over a 2.5-second time window. In addition to the temporal predictors, scalar predictors, including lever press latency and trial number, which were used to track the internal state, were also included in the predictor matrix. The data was binned in 50-millisecond time bins.

Figure 7. Distinct latent-state dynamics in the anterior insula during alcohol seeking and abstinence behavior.

(A) Alcohol self-administration task schematic. (B) Posterior state probabilities during an example seek trial. (C) State probability heatmaps aligned to seek lever insertion (sorted by lever press latency). (D) Population-averaged kernel at seek lever insertion (Mean±SEM). (E) and (F) Same as (C) and (D) but aligned to abstain lever insertion.

Since we did not have prior knowledge about the true number of states, we tried different numbers of states, i.e., $S\in \{3,4,5\}$. The negative log-likelihood of the model fits showed a similar trend to the mPFC and premotor cortex dataset: as the number of states increased, the negative log-likelihoods decreased across EM iterations, and the distinction between states became subtle (Supp. Fig 4). Therefore, we decided to use a 4-state model.

The posterior probabilities for one seek trial are shown in Fig 7B, with state transitions around alcohol-associated cues (lever insertion and retraction) and alcohol intake (post-port entry). We compared the state probability heat maps for seek and abstain trials and observed that the state transitions in seek trials appeared more structured, when sorted by latency to lever press, than those in abstain trials (Fig 8C,E). Furthermore, the states observed at lever insertion for the two trial types are distinct: states 2 and 4 for seek and abstain trials, respectively. To understand the differences in ensemble activity across these states, we visualized the population-averaged kernels for states 2 and 4 (Fig 8D,F). The anterior insula population kernels showed a positive deviation for state 2 and a negative deviation for state 4. This corroborates previous findings of increased anterior insula engagement in the presence of alcohol cues to drive seeking behavior (Cofresí et al., 2019; Campbell et al., 2019).

Figure 8. Convergence and validation of the 4-state GLM-HMM trained on anterior insula data.

(A) Negative log-likelihood across EM iterations. (B) Change in model parameters ($\mathit{A}$, $\mathbf{\Theta }$) across EM iterations. (C) Mean GLM weight correlation between states per neuron. (D) Held-out log-likelihood for model trained on real and shuffled spikes from 39 trials and tested on 40^th trial (Wilcoxon signed-rank test). Each dot represents the held-out trial.

To test our model fit, we tracked the convergence of our 4-state model’s negative log-likelihood and change in parameters across iterations (Fig 8A,B) and confirmed that the states were non-redundant (Fig 8C). We also tested the performance of the 4-state model trained on the first 40 trials and compared it with the model trained on shuffled spikes by computing the log-likelihood of observing the ensemble spiking in a held-out test trial. The log-likelihoods of the held-out test trials were significantly higher when the model was trained on the real dataset than when it was trained on a shuffled dataset (Fig 8D; Wilcoxon signed-rank test), highlighting that our model was able to select useful information when trained on nearly half the trials of the complete dataset.

When the 4-state model was initialized with a set of weights obtained by running a per-neuron ridge-penalized GLM and using that penalty in the M-step, as done for the above-mentioned datasets, the final GLM weights were well constrained within biological limits, whereas using a set of initial weights either by fitting a GLM with no penalization or using a random initialization yielded biologically implausible weights (Fig 9A). This underscores the importance of neuron-adaptive penalization for overcoming overfitting on sparse datasets. We also compared the M-step convergence of the dataset using the trust-region and Newton-Raphson methods. While the negative log-likelihood for the model decreased exponentially with the trust-region method, it increased exponentially with the Newton-Raphson method (Fig 9B). Furthermore, the Newton-Raphson method became increasingly unstable across iterations, as evidenced by the high fraction of non-finite GLM weights (Fig. 9C).

Figure 9. Comparison of initialization and optimization approaches for GLM-HMM fitted on rodent anterior insula activity.

(A) Probability density distribution of final absolute GLM weights in log10 scale across all neurons using penalized GLM, unpenalized GLM, and random initializations. (B) Negative log-likelihood comparison across iterations when using trust-region and Newton-Raphson methods for M-step convergence. (C) Fraction of non-finite GLM weights across EM iterations when using trust-region and Newton-Raphson methods in the M-step.

6. Conclusion

In this work we developed and implemented a robust fitting procedure for multi-neuron GLM-HMM framework. Our work builds on prior work for neuronal GLM-HMMs (Escola et al., 2011), extending and adopting necessary changes to enable stable fitting on neuronal recordings. Our work addresses the key challenges that prevent current procedures used in, e.g., behavioral modeling with GLM-HMMs from working on neuronal data: the sparsity of neuronal activity, the limited data availability in many electrophysiology datasets, and the high correlations between stimulus features. Our framework incorporates the following features: 1) neuron-adaptive regularization in the M-step, obtained by fitting a five-fold cross-validated GLM to each neuron prior to model fitting, to yield biologically plausible GLM coefficients, 2) a trust-region optimization algorithm for stable M-step convergence when local curvature of the objective function is not quadratic, and 3) a leave-one-out trial cross-validation approach to test the model performance trained on limited data while preserving its temporal structure. Across three real electrophysiology datasets spanning multiple brain regions and species, as well as a series of benchmark analyses, our framework not only converges stably but also yields behaviorally interpretable latent states. In all, we have developed a practical approach for applying GLM-HMMs to neural population recordings, laying the groundwork for understanding how discrete latent states emerge from neuronal activity and shape behavior.

Our work complements the significant progress that has been made in identifying internal states by applying GLM-HMMs on behavioral datasets (Calhoun et al., 2019; Ashwood et al., 2022; Bolkan et al., 2022; Hulsey et al., 2024; Cuturela et al., 2025; Mohammadi et al., 2025; Li et al., 2023; Chen et al., 2026; Keeley et al., 2026; Moore et al., 2026). However, since internal states are an emergent property of neuronal population dynamics, models that directly incorporate ensemble spiking alongside task-related stimuli are needed to gain mechanistic insights underlying the latent states. In contrast to dynamical systems models that identify continuously evolving latent states (Sussillo et al., 2015; Kleinman et al., 2021; Pandarinath et al., 2018; Churchland et al., 2012; Mudrik et al., 2024; Yezerets et al., 2025; Mudrik et al., 2025), the categorical nature of HMMs in the GLM-HMM framework is often sought after to more directly model discrete behavioral modes. By removing the practical impediments of fitting GLM-HMM models to data, we aim to increase the ability of neuroscientists to study such time-varying neural activity.

Data availability

The prefrontal cortex and premotor cortex data are both publicly available datasets (Peyrache et al., 2009, 2018; Lawlor et al., 2018; Perich et al., 2018b). Data for the anterior insula example is available as part of the Github database at https://github.com/AamnaLawrence/GLM-HMM-Neuronal-Encoding/tree/main, with Zenodo DOI https://doi.org/10.5281/zenodo.20970583.

Code availability

The MATLAB implementation of our GLM-HMM fitting is available on Github at https://github.com/AamnaLawrence/GLM-HMM-Neuronal-Encoding/tree/main, with Zenodo DOI https://doi.org/10.5281/zenodo.20970583.