A reaction network scheme for hidden Markov model parameter learning

Abstract

With a view towards artificial cells, molecular communication systems, molecular multiagent systems and federated learning, we propose a novel reaction network scheme (termed the Baum–Welch (BW) reaction network) that learns parameters for hidden Markov models (HMMs). All variables including inputs and outputs are encoded by separate species. Each reaction in the scheme changes only one molecule of one species to one molecule of another. The reverse change is also accessible but via a different set of enzymes, in a design reminiscent of futile cycles in biochemical pathways. We show that every positive fixed point of the BW algorithm for HMMs is a fixed point of the reaction network scheme, and vice versa. Furthermore, we prove that the ‘expectation’ step and the ‘maximization’ step of the reaction network separately converge exponentially fast and compute the same values as the E-step and the M-step of the BW algorithm. We simulate example sequences, and show that our reaction network learns the same parameters for the HMM as the BW algorithm, and that the log-likelihood increases continuously along the trajectory of the reaction network.

1. Introduction

The implementation of abstract dynamical systems with molecular systems has gained scientific interest as a promising piece in the nanotechnology toolbox. Several automated theoretical schemes can now compile arbitrary networks of formal reactions into DNA oligonucleotide sequences [1–8]. Experimental demonstrations have synthesized these oligonucleotides, mixed them in a single test tube, and verified that they interact via base pairing reactions to implement the dynamics of the formal network [2,7,9–11]. In this way, if an algorithm can be described by reaction network dynamics, it might equally be carried out in a test tube by a DNA machine. Understanding how to describe algorithms in terms of reaction network dynamics is thus becoming a matter of interest.

Dynamical systems described by formal reaction networks are known to be computationally universal [12]. Several examples of functions computed by reaction network dynamics have been described [13–20]. However, the computational universality proofs often do not lend to the most elegant ways of implementing the algorithms with chemical reaction networks. We are particularly interested in problems arising in statistical learning theory, such as maximum-likelihood estimation and related optimization problems, Bayesian posterior sampling, and inference from incomplete data. These problems are at the core of statistical theory and may have many practical applications when implemented inside artificial cells. Our work demonstrates that these problems are particularly amenable to implementation by reaction network dynamics by exploiting the formal connection between the notion of entropy in statistics and in statistical mechanics.

Molecule-based statistical inference is receiving increasing attention. In [21], dynamic Bayesian decision-making in the form of a two-state hidden Markov model (HMM) is implemented by means of intracellular kinetics (which might be interpreted in terms of reaction networks), where the target is to infer an approximate posterior distribution. Similar ideas have been applied to decode information in biological processes [22]. On a different note, a cell’s potential for solving tasks and learning requires a thorough understanding of design principles in reaction networks and how molecular sources of stochastic information are communicated within and between cells [23,24]. Such tasks might be seen as computational and statistical problems posed to the cell.

In this article, we describe a reaction network scheme whose dynamics learns parameters of HMMs. HMMs are standard statistical models widely used in machine learning to model complex data with a linear spatial structure as in bioinformatics [25], or a temporal structure as in speech recognition [26]. HMMs also form an essential component of communication systems as well as of intelligent agents trained by reinforcement learning methods. They might be used in an exploratory sense without stipulating the interpretation of the hidden variables in advance, or in a concrete sense to learn the strength and influence of known hidden variables. Our group has previously worked on statistical learning theory from the perspective of reaction networks and log-linear models [27–31], and the current paper builds on these experiences.

Our proposed algorithm has similarities to, and important differences from, the Baum–Welch (BW) algorithm [32] which is the standard learning algorithm for HMMs. The BW algorithm is an iterative expectation-maximization (EM) algorithm where one step is performed at a time in a prescribed sequence. Our reaction network scheme is divided into four subnetworks that correspond to the forward algorithm, the backward algorithm, the expectation step (E-step), and the maximization step (M-step) of the BW algorithm. Each subnetwork describes a system of ordinary differential equations (ODEs) that might be run separately, exactly mimicking the steps of the BW algorithm; or run simultaneously in continuous time and in a distributed manner, obtaining a variant on the BW theme where all four stages of the BW algorithm are being performed at the same time. Because our algorithm permits the different stages to be run simultaneously and without coordination, it is particularly suited to software federated learning implementations where HMMs might need to be run on a network of edge devices in a distributed manner.

We obtain the following results.

• — Our scheme can be partitioned into two modules, the inference module (forward, backward, E-step) and the learning module (M-step), which provide feedback to each other. We show in theorem 4.2 that each module separately converges exponentially fast to the correct value (the value of the BW algorithm) when the other module is kept switched off.
• — We prove in theorem 4.3 that our scheme (when divided into two modules, or when considered jointly) and the BW algorithm have the same set of positive fixed points.
• — We demonstrate practical feasibility of our algorithm by simulation of example HMMs, and by showing that parameters can be learned successfully with performance comparable to that of the BW algorithm.

The proposed reaction network is a formal (abstract) reaction network without particular chemical features. It might be turned into a chemically realistic reaction network by compiling the formal reactions into reactions between DNA oligonucleotide sequences, using recently proposed techniques [1–4]. Promising areas of application of this work come from cellular biology. In many cellular processes, only partial information about the environment is available in the form of a sequence of observations. For example, this might happen when an enzyme acts processively on a substrate or a molecular walker locates its position on a grid [33–35]. In the future, a molecule-based HMM device might learn a molecular environment within an organism by sensing and interacting with the environment at the molecular level. It might take action according to the learning outcome, for example, choosing among different drug options, or a molecule-based HMM might be used as a building block in an artificial cell or population of cells, enabling cooperative behaviour among cells or facilitating various tasks.

2. The Baum–Welch algorithm

A stochastic map between two finite sets P and Q is a matrix A = (a_pq)_P×Q, such that a_pq ≥ 0 for p ∈ P, q ∈ Q, and $\sum _{q\in Q}{a}_{\hspace{0.17em}pq}=1$ for p ∈ P. One might think of a stochastic map as a collection of conditional probability distributions.

An HMM is a tuple (H, V, θ, ψ, π) of two finite sets H (for ‘hidden’) and V (for ‘visible’), an initial probability distribution π = (π_h)_h∈H on H, and two stochastic maps: the transition matrix θ : H → H, and the emission matrix ψ : H → V. See figure 1 for an example.

Figure 1.

Learning HMMs from sequences. (a) HMM: the hidden states, elements of H = {H₁, H₂}, are not directly observable. Instead, we observe elements from a set V = {V₁, V₂} of visible states. The parameters θ₁₁, θ₁₂, θ₂₁, θ₂₂ denote the probability of transitions between the hidden states. The probability of observing the states V₁, V₂ depends on the parameters ψ₁₁, ψ₁₂, ψ₂₁, ψ₂₂, as indicated in the figure. (b) The forward algorithm computes the likelihood of the first ℓ + 1 observed states (the position ℓ + 1 likelihood) ${\alpha }_{\ell +1,1}={\alpha }_{\ell 1}{\theta }_{11}{\psi }_{1v_{\ell +1}}+{\alpha }_{\ell 2}{\theta }_{21}{\psi }_{1v_{\ell +1}}$ by forward propagating the position ℓ likelihoods α_ℓ1 and α_ℓ2. Here, v_ℓ, v_ℓ+1 ∈ V are the observed symbols at position ℓ and ℓ + 1, respectively. (c) The backward algorithm computes the conditional probability of the observed states ℓ + 1, …, L, given the observed state ℓ (the position ℓ conditional probability) ${\beta }_{\ell 1}={\theta }_{11}{\psi }_{1v_{\ell +1}}{\beta }_{\ell +1,1}+{\theta }_{12}{\psi }_{2v_{\ell +1}}{\beta }_{\ell +1,2}$ by propagating the position ℓ conditional probabilities β_ℓ+1,1 and β_ℓ+1,2 backwards. (d) The BW algorithm is a fixed point expectation-maximization computation. The E-step calls the forward and backward algorithm as subroutines and, conditioned on the entire observed sequence $(v_1,v_2,\dots ,v_L)\in V^L$, computes the probabilities γ_ℓg of being in states g ∈ H at position ℓ and the probabilities ξ_ℓgh of taking the transitions (g, h) ∈ H² at position ℓ. The M-step updates the parameters θ and ψ to maximize their likelihood given the observed sequence.

The likelihood $P(v_1,\dots ,v_L|\theta ,\psi )$ of an observed sequence v₁, …, v_L ∈ V given the parameter (θ, ψ) is the probability

$$P(v_1,\dots ,v_L|\theta ,\psi )=\sum _{\eta \in H^L}\hspace{0.17em}{\pi }_{h_1}{\psi }_{h_1{v}_1}\prod _{\ell =2}^L{\theta }_{h_{\ell -1}{h}_{\ell }}{\psi }_{h_{\ell }{v}_{\ell }},$$ 2.1

where the sum is over all sequences η = (h₁, …, h_L) ∈ H^L, with L ≥ 2 (to avoid triviality). Knowing π and the sequence $(v_1,v_2,\dots ,v_L)$ of visible states, one can estimate maximum-likelihood values for (θ, ψ) by means of the BW algorithm, which is a particular instance of the EM algorithm. In §5, we show how to extend this construction to unknown π and multiple sequences while preserving the theoretical guarantees.

The standard BW algorithm is composed of four subroutines. The forward algorithm (figure 1b) outputs the quantities α_ℓh computed from initial values of (θ, ψ) and π by the recursion

$${\alpha }_{1h}={\pi }_h{\psi }_{h{v}_1}\mathrm{and}{\alpha }_{\ell +1,h}=\sum _{g\in H}{\alpha }_{\ell g}{\theta }_{gh}{\psi }_{h{v}_{\ell +1}},$$

and the backward algorithm (figure 1c) outputs the quantities β_ℓh computed by the recursion

$${\beta }_{Lh}=1\mathrm{and}{\beta }_{\ell h}=\sum _{g\in H}{\theta }_{hg}{\psi }_{g{v}_{\ell +1}}{\beta }_{\ell +1,g},$$

where h ∈ H and ℓ = 1, …, L − 1. The E-step computes

$${\xi }_{\ell gh}=\frac{{\alpha }_{\ell g}{\theta }_{gh}{\psi }_{h{v}_{\ell +1}}{\beta }_{\ell +1,h}}{\sum _{\hspace{0.17em}f\in H}{\alpha }_{\ell f}{\beta }_{\ell f}}\mathrm{and}{\gamma }_{\ell h}=\frac{{\alpha }_{\ell h}{\beta }_{\ell h}}{\sum _{\hspace{0.17em}f\in H}{\alpha }_{\ell f}{\beta }_{\ell f}},$$

for ℓ ∈ {1, 2, …, L} and g, h ∈ H. Finally, the M-step computes an update of (θ, ψ),

$${\theta }_{gh}^{\prime }=\frac{\sum _{\ell =1}^{L-1}{\xi }_{\ell gh}}{\sum _{\ell =1}^{L-1}\sum _{\hspace{0.17em}f\in H}{\xi }_{\ell gf}}\mathrm{and}{\psi }_{hw}^{\prime }=\frac{\sum _{\ell =1}^L{\gamma }_{\ell h}{\delta }_{w,v_{\ell }}}{\sum _{\ell =1}^L{\gamma }_{\ell h}},$$

for g, h ∈ H and w ∈ V, where δ is the Dirac delta function ${\delta }_{w,v_{\ell }}=1$ iff w = v_ℓ. Here, θ_gh′ and ψ_hw′ (with primes) are the updated parameter values of θ_gh and ψ_hw, respectively, after one iteration of the BW algorithm. As π is assumed known, it is not updated.

The BW algorithm (figure 1d) sequentially runs the forward algorithm, the backward algorithm, the E-step, and the M-step until the change in the likelihood (2.1) is insignificant. In practice, this means until the difference between (θ, ψ) and (θ′, ψ′) becomes smaller than a prescribed tolerance level. For the steps to be well defined, division by zero is not allowed. Denote the parameter region for which division by zero does not happen in any step of an iteration by

$$\Theta =\{(\theta ,\psi )\hspace{0.17em}:\hspace{0.17em}\sum _{\ell =1}^L{\alpha }_{\ell h}{\beta }_{\ell h}>0,\hspace{0.17em}h\in H\}$$

(see the electronic supplementary material for details and proof). Furthermore, let ${\Theta }_0=\{(\theta ,\psi )\hspace{0.17em}|\hspace{0.17em}\theta >0,\psi >0\}$ and ${\Theta }_1=\{(\theta ,\psi )|\hspace{0.17em}\theta \ge 0,\psi \ge 0\}$ (the full parameter space), where vector/matrix inequalities are taken coordinate-wise. It follows that ${\Theta }_0\subseteq \Theta \subseteq {\Theta }_1$, and that all steps in one iteration of the BW algorithm are valid, provided $(\theta ,\psi )\in \Theta$. Each row (a conditional distribution) of the update (θ′, ψ′) has unit length automatically.

The following lemma is essential. See the electronic supplementary material for a proof.

Lemma 2.1. —

Assume all letters of V are in the observed sequence v₁, …, v_L and π > 0, L ≥ 2, or all letters of V are in v₂, …, v_L (excluding v₁) and π ≥ 0, L ≥ 3. If $(\theta ,\psi )\in {\Theta }_0$, then the same holds for the updated parameter value, $({\theta }^{\prime },{\psi }^{\prime })\in {\Theta }_0$.

Let (θ_n, ψ_n) denote the value of (θ′, ψ′) after n iterations of the BW algorithm. According to lemma 2.1, if $({\theta }_0,{\psi }_0)\in {\Theta }_0$, then $({\theta }_n,{\psi }_n)\in {\Theta }_0$ for all n ≥ 0. If $({\theta }_n,{\psi }_n)\to ({\theta }^{\ast },{\psi }^{\ast })\in {\Theta }_0$ as n → ∞, then it is a local extremum or saddle point of the likelihood (2.1) [36–39]. If $({\theta }^{\ast },{\psi }^{\ast })\in \Theta$, then it is a fixed point of the BW algorithm. However, the limit might be outside $\Theta$ [36–38]. In general, the BW algorithm might have multiple fixed points depending on the initial point (θ₀, ψ₀) and the observed sequence.

Henceforth, we make the assumptions of lemma 2.1. If v ∈ V is not observed, then ψ_hv = 0 for all h ∈ H, and one might equivalently consider a HMM with the state v removed from V. Hence, the real (mild) restriction is π > 0, L ≥ 2, or π ≥ 0, L ≥ 3.

3. Baum–Welch reaction network

Let S be a finite set. A formal (chemical) reaction over S is a pair $a,b\in {\mathbb{Z}}_{\ge 0}^S$ of non-negative integer vectors. This is commonly represented in chemical equation notation as $\sum _{i\in S}{a}_i{X}_i\to \sum _{i\in S}{b}_i{X}_i$, where X_i, i ∈ S, represent formal (chemical) species. Given a rate constant k > 0, mass-action kinetics describes the change of concentrations through time by the system of ODEs $\dot{X}(t)=(b-a)k\prod _{i\in S}{X}_i{(t)}^{a_i}$, where species names are overloaded to also represent the vector of concentrations X(t) = (X_i(t))_i∈S at time t ≥ 0. A reaction network is a finite collection (a₁, b₁), (a₂, b₂), $\dots ,(a_m,b_m)$ of formal reactions, together with a choice of reaction rate constants. Combining the effect of all reactions yields the ODE system $\dot{X}(t)=\sum _{\hspace{0.17em}j=1}^m(b_j-a_j)k_j\prod _{i\in S}{X}_i{(t)}^{a_{\hspace{0.17em}ji}}$. For background on reaction networks, see [40].

We now describe a reaction network that implements HMM learning. Our scheme bears close correspondence to the BW algorithm as presented in the previous section. Let an HMM (H, V, θ, ψ, π) and an observed sequence $(v_1,v_2,\dots ,v_L)\in V^L$ be given. Choose an arbitrary hidden state h* ∈ H and an arbitrary visible state v* ∈ V. This choice is merely an artifice to break symmetry, and our results hold independently of these choices. We represent every variable appearing in the BW algorithm by a separate species. The species are θ_gh, θ_gh′, ψ_hw, ψ_hw′, π_h, α_ℓh, β_ℓh, γ_ℓh, ξ_ℓgh with indices g, h ∈ H, w ∈ V, ℓ = 1, …, L. That is, both (θ, ψ) and the update (θ′, ψ′) are represented as species.

We first work out in full detail how the forward algorithm may be translated into chemical reactions. Recall that the forward algorithm is the recursion

$${\alpha }_{1h}={\pi }_h{\psi }_{h{v}_1}\mathrm{and}{\alpha }_{\ell +1,h}=\sum _{g\in H}{\alpha }_{\ell g}{\theta }_{gh}{\psi }_{h{v}_{\ell +1}},$$

for ℓ = 1, …, L − 1 and g, h ∈ H. This implies the balance equations

$${\alpha }_{1h}{\pi }_{h^{\ast }}{\psi }_{h^{\ast }{v}_1}={\alpha }_{1h^{\ast }}{\pi }_h{\psi }_{h{v}_1}$$ 3.1

and

$${\alpha }_{\ell +1,h}\sum _{g\in H}{\alpha }_{\ell g}{\theta }_{g{h}^{\ast }}{\psi }_{h^{\ast }{v}_{\ell +1}}={\alpha }_{\ell +1,h^{\ast }}\sum _{g\in H}{\alpha }_{\ell g}{\theta }_{gh}{\psi }_{h{v}_{\ell +1}},$$ 3.2

for all h ∈ H and ℓ = 1, …, L − 1. For the initialization step, this prompts the use of the reactions

$${\alpha }_{1h}+{\pi }_{h^{\ast }}+{\psi }_{h^{\ast }{v}_1}\stackrel{1}{\to }{\alpha }_{1h^{\ast }}+{\pi }_{h^{\ast }}+{\psi }_{h^{\ast }{v}_1}$$

and

$${\alpha }_{1h^{\ast }}+{\pi }_h+{\psi }_{h{v}_1}\stackrel{1}{\to }{\alpha }_{1h}+{\pi }_h+{\psi }_{h{v}_1},$$

for all h ∈ H, where $\stackrel{1}{\to }$ indicates the reaction rate constant is put to 1. Only one species change in each reaction (the red species), the other species are catalysts that remain unchanged by the reaction. The rate by which α_1h is converted into α_1h* depends on the concentrations of the catalysts and, thus, is time-dependent. By design, at equilibrium the set of reactions fulfil the balance equation (3.1). For the recursion step, we use the reactions

$$\begin{array}{rl}& {\alpha }_{\ell +1,h}+{\alpha }_{\ell g}+{\theta }_{g{h}^{\ast }}+{\psi }_{h^{\ast }{v}_{\ell +1}}\stackrel{1}{\to }{\alpha }_{\ell +1,h^{\ast }}+{\alpha }_{\ell g}+{\theta }_{g{h}^{\ast }}+{\psi }_{h^{\ast }{v}_{\ell +1}}\end{array}$$

and

$$\begin{array}{rl}& {\alpha }_{\ell +1,h^{\ast }}+{\alpha }_{\ell g}+{\theta }_{gh}+{\psi }_{h{v}_{\ell +1}}\stackrel{1}{\to }{\alpha }_{\ell +1,h}+{\alpha }_{\ell g}+{\theta }_{gh}+{\psi }_{h{v}_{\ell +1}},\end{array}$$

for all g, h ∈ H, and ℓ = 1, …, L − 1. Again by design, at equilibrium the balance equation (3.2) is fulfilled for this set of reactions.

The reactions depend on the observed sequence $(v_1,v_2,\dots ,v_L)\in V^L$ of visible states. This is a problem because one would have to design different reaction networks for different observed sequences, defeating the whole purpose. To solve this problem, we introduce the catalyst species E_ℓw with $\ell =1,\dots ,L$ and w ∈ V. The E_ℓw species are initialized such that at time zero, E_ℓw(0) = 1 for w = v_ℓ, and E_ℓw(0) = 0 for $w\ne v_{\ell }$. Thus, their concentrations remain fixed throughout time.

The other parts of the BW algorithm may be translated into chemical reactions using a similar logic. The full set of reactions is shown in table 1 and is termed the BW reaction network. It consists of four parts that are further divided into smaller subnetworks. Each of the four parts corresponds to one of the four parts of the BW algorithm, as shown in table 1. The corresponding equations of the ODE system are shown in appendix A (equations (A 1)–(A 8)). Each of the smaller subnetworks in table 1 when run independently can be separately analysed as a mono-molecular reaction network for which the dynamical behaviour can be fully described as shown in appendix A.

Table 1.

The BW reaction network, divided into four parts corresponding to the forward (${\mathcal{R}}_1^{\alpha },{\mathcal{R}}_{\ell }^{\alpha }$) and the backward (${\mathcal{R}}_{\ell }^{\beta }$) algorithm, the E-step (${\mathcal{R}}_{\ell }^{\gamma },{\mathcal{R}}_{\ell }^{\xi }$) and the M-step (${\mathcal{R}}^{\theta },{\mathcal{R}}^{\psi }$). All parts but the M-step are further divided into small subnetworks, one for each ℓ = 1, …, L − 1 (or L). Catalytic species are in black, non-catalytic in red. The indices vary over g, h ∈ H, w ∈ V, and $\ell =1,\dots ,L-1$. For γ species, ℓ = L is also allowed.

BW algorithm	BW reaction network	subnetwork
$\begin{array}{c}{\alpha }_{1h}={\pi }_h{\psi }_{h{v}_1}\hspace{0.17em}\\ {\alpha }_{\ell +1,h}={\displaystyle \sum _{g\in H}{\alpha }_{\ell g}{\theta }_{gh}{\psi }_{h{v}_{\ell +1}}}\end{array}$	$\begin{array}{rl}& {\alpha }_{1h}+{\pi }_{h^{\ast }}+{\psi }_{h^{\ast }w}+E_{1w}\to {\alpha }_{1h^{\ast }}+{\pi }_{h^{\ast }}+{\psi }_{h^{\ast }w}+E_{1w}\\ & {\alpha }_{1h^{\ast }}+{\pi }_h+{\psi }_{hw}+E_{1w}\to {\alpha }_{1h}+{\pi }_h+{\psi }_{hw}+E_{1w}\\ & {\alpha }_{\ell +1,h}+{\alpha }_{\ell g}+{\theta }_{g{h}^{\ast }}+{\psi }_{h^{\ast }w}+E_{\ell +1,w}\to {\alpha }_{\ell +1,h^{\ast }}+{\alpha }_{\ell g}+{\theta }_{g{h}^{\ast }}+{\psi }_{h^{\ast }w}+E_{\ell +1,w}\\ & {\alpha }_{\ell +1,h^{\ast }}+{\alpha }_{\ell g}+{\theta }_{gh}+{\psi }_{hw}+E_{\ell +1,w}\to {\alpha }_{\ell +1,h}+{\alpha }_{\ell g}+{\theta }_{gh}+{\psi }_{hw}+E_{\ell +1,w}\end{array}$	$\begin{array}{c}{\mathcal{R}}_1^{\alpha }\\ \\ \\ {\mathcal{R}}_{\ell }^{\alpha }\end{array}$
$\begin{array}{c}{\beta }_{Lh}=1\hspace{0.17em}\\ \\ {\beta }_{\ell h}={\displaystyle \sum _{g\in H}{\theta }_{hg}{\psi }_{g{v}_{\ell +1}}{\beta }_{\ell +1,g}}\end{array}$	$\begin{array}{r}{\beta }_{\ell h}+{\beta }_{\ell +1,g}+{\theta }_{h^{\ast }g}+{\psi }_{gw}+E_{\ell +1,w}\to {\beta }_{\ell h^{\ast }}+{\beta }_{\ell +1,g}+{\theta }_{h^{\ast }g}+{\psi }_{gw}+E_{\ell +1,w}\\ {\beta }_{\ell h^{\ast }}+{\beta }_{\ell +1,g}+{\theta }_{hg}+{\psi }_{gw}+E_{\ell +1,w}\to {\beta }_{\ell h}+{\beta }_{\ell +1,g}+{\theta }_{hg}+{\psi }_{gw}+E_{\ell +1,w}\end{array}$	${\mathcal{R}}_{\ell }^{\beta }$
$\begin{array}{rl}& {\gamma }_{\ell h}=\frac{{\alpha }_{\ell h}{\beta }_{\ell h}}{{\displaystyle \sum _{g\in H}{\alpha }_{\ell g}{\beta }_{\ell g}\hspace{0.17em}}}\\ & {\xi }_{\ell gh}=\frac{{\alpha }_{\ell g}{\theta }_{gh}{\psi }_{h{v}_{\ell +1}}{\beta }_{\ell +1,h}}{{\displaystyle \sum _{\hspace{0.17em}f\in H}{\alpha }_{\ell f}{\beta }_{\ell f}}}\end{array}$	$\begin{array}{rl}& {\gamma }_{\ell h}+{\alpha }_{\ell h^{\ast }}+{\beta }_{\ell h^{\ast }}\to {\gamma }_{\ell h^{\ast }}+{\alpha }_{\ell h^{\ast }}+{\beta }_{\ell h^{\ast }}\\ & {\gamma }_{\ell h^{\ast }}+{\alpha }_{\ell h}+{\beta }_{\ell h}\to {\gamma }_{\ell h}+{\alpha }_{\ell h}+{\beta }_{\ell h}\\ \\ & {\xi }_{\ell gh}+{\alpha }_{\ell h^{\ast }}+{\theta }_{h^{\ast }{h}^{\ast }}+{\beta }_{\ell +1,h^{\ast }}+{\psi }_{h^{\ast }w}+E_{\ell +1,w}\to \\ & {\xi }_{\ell h^{\ast }{h}^{\ast }}+{\alpha }_{\ell h^{\ast }}+{\theta }_{h^{\ast }{h}^{\ast }}+{\beta }_{\ell +1,h^{\ast }}+{\psi }_{h^{\ast }w}+E_{\ell +1,w}\\ & {\xi }_{\ell h^{\ast }{h}^{\ast }}+{\alpha }_{\ell g}+{\theta }_{gh}+{\beta }_{\ell +1,g}+{\psi }_{hw}+E_{\ell +1,w}\to \\ & {\xi }_{\ell gh}+{\alpha }_{\ell g}+{\theta }_{gh}+{\beta }_{\ell +1,g}+{\psi }_{hw}+E_{\ell +1,w}\end{array}$	$\begin{array}{c}{\mathcal{R}}_{\ell }^{\gamma }\\ \\ \\ \\ {\mathcal{R}}_{\ell }^{\xi }\end{array}$
$\begin{array}{c}{\theta }_{gh}^{\prime }=\frac{{\displaystyle \sum _{\ell =1}^{L-1}{\xi }_{\ell gh}}}{{\displaystyle \sum _{\ell =1}^{L-1}{\displaystyle \sum _{\hspace{0.17em}f\in H}{\xi }_{\ell gf}}}}\\ {\psi }_{hw}^{\prime }=\frac{{\displaystyle \sum _{\ell =1}^L{\gamma }_{\ell h}{\delta }_{w,v_{\ell }}}}{{\displaystyle \sum _{\ell =1}^L{\gamma }_{\ell h}}}\end{array}$	$\begin{array}{c}{\theta }_{gh}^{\prime }+{\xi }_{\ell g{h}^{\ast }}\to {\theta }_{g{h}^{\ast }}^{\prime }+{\xi }_{\ell g{h}^{\ast }}\\ {\theta }_{g{h}^{\ast }}^{\prime }+{\xi }_{\ell gh}\to {\theta }_{gh}^{\prime }+{\xi }_{\ell gh}\\ \\ \\ \\ \\ {\psi }_{hw}^{\prime }+{\gamma }_{\ell h}+E_{\ell v^{\ast }}\to {\psi }_{h{v}^{\ast }}^{\prime }+{\gamma }_{\ell h}+E_{\ell v^{\ast }}\\ {\psi }_{h{v}^{\ast }}^{\prime }+{\gamma }_{\ell h}+E_{\ell w}\to {\psi }_{hw}^{\prime }+{\gamma }_{\ell h}+E_{\ell w}\end{array}$	$\begin{array}{c}{\mathcal{R}}^{\theta }\\ \\ \\ \\ \\ \\ {\mathcal{R}}^{\psi }\end{array}$

4. The dynamics of the Baum–Welch reaction network

In the following, we expose the relationship between the dynamics of the BW algorithm and the BW reaction network. For this, we define three different, alternative ways of running the BW reaction network in table 1.

Let α_ℓ(t) = (α_ℓh(t))_h∈H denote the vector of concentrations at time t ≥ 0, with similar notation for other quantities. For convenience, we leave out the species E_ℓw.

4.1. BW1

Initialization. Fix an initial value, $(\theta ,\psi )=({\theta }_0,{\psi }_0)\in {\Theta }_0$. Initialize the concentrations at time t = 0: α_ℓ(0), β_ℓ(0), γ_ℓ(0) $\in {\mathbb{R}}_{\ge 0}^H$, ξ_ℓ(0), ${\theta }^{\prime }(0)\in {\mathbb{R}}_{\ge 0}^{H\times H}$ and ${\psi }^{\prime }(0)\in {\mathbb{R}}^{H\times V}$, such that $\sum _h{\alpha }_{\ell h}(0)=A_{\ell }$, $\sum _h{\beta }_{\ell h}(0)=B_{\ell }$, $\sum _h{\gamma }_{\ell h}(0)=1$, $\sum _h{\eta }_{\ell gh}(0)=1$, $\sum _h{\theta }_{gh}^{\prime }(0)=1$ and $\sum _v{\psi }_{hv}^{\prime }(0)=1$, for arbitrary positive constants A_ℓ, B_ℓ.

Execution. The ODE systems of the subnetworks are executed sequentially in the order ${\mathcal{R}}_1^{\alpha },\dots ,{\mathcal{R}}_L^{\alpha }$, ${\mathcal{R}}_{L-1}^{\beta },\dots ,{\mathcal{R}}_1^{\beta }$, ${\mathcal{R}}_1^{\gamma },$ $\dots ,{\mathcal{R}}_L^{\gamma }$, ${\mathcal{R}}_1^{\xi },\dots ,{\mathcal{R}}_{L-1}^{\xi }$, ${\mathcal{R}}^{\theta }$, ${\mathcal{R}}^{\psi }$ (table 1), such that the ODE system of the kth subnetwork (k = 1, …, 4L − 1) is run until an equilibrium is obtained with the chosen initial values, before the ODE system of the (k + 1)th subnetwork is executed. The catalyst species concentrations of the kth subnetwork are fixed to their equilibrium values obtained in the k′th subnetworks, k′ < k. In particular, the concentrations of the species θ, ψ are fixed to their initial values $({\theta }_0,{\psi }_0)\in {\Theta }_0$.

Iteration. The above procedure is then iterated. After completion of the nth iteration, n ≥ 0, the concentrations of the species θ, ψ in the (n + 1)th iteration are initialized to the equilibrium values $({\theta }_n,{\psi }_n)\in {\Theta }_0$ of the species (θ′, ψ′) in the nth iteration. All other species are initialized at their current values.

Completion. This process is continued until convergence of (θ_n, ψ_n) has been achieved. The limit of (θ_n, ψ_n) is denoted the equilibrium of BW1 with initial point (θ₀, ψ₀).

4.2. BW2

Initialization. As in BW1.

Execution. The ODE systems corresponding to the inference module (forward, backward and E-step) and the learning module (M-step) of the BW algorithm are executed sequentially and run until an equilibrium is obtained, with the concentrations of the species θ, ψ fixed to the initial values $({\theta }_0,{\psi }_0)\in {\Theta }_0$.

Iteration and completion. BW2 is iterated similarly to BW1. The limit of (θ_n, ψ_n) is denoted the equilibrium of BW2 with initial point (θ₀, ψ₀). See figure 2.

Figure 2.

(a) Iterations of BW2. In the nth iteration, n ≥ 0, the concentrations of (θ, ψ) are fixed at (θ_n, ψ_n). The ODE system is initialized at (θ′(0), ψ′(0)) = (θ_n, ψ_n). Over time (θ′(t), ψ′(t)) converges to (θ_n+1, ψ_n+1). As n → ∞, (θ_n, ψ_n) converges to the blue dot. (b) A trajectory of BW3 converging towards the blue dot, initialized at a (random) point in the parameter space.

4.3. BW3

Initialization. As in BW1 with (θ′(0), ψ′(0)) = (θ₀, ψ₀).

Execution. The species θ, ψ are identified with the species θ′, ψ′. That is, unprimed species are substituted for primed ones in table 1. The full ODE system as described in §3 is executed at the same time without fixing any species concentrations so that all species concentrations are dynamic.

Completion. The limit of (θ′(t), ψ′(t)) (if it exists) is denoted the equilibrium of BW3 with initial point (θ₀, ψ₀).

BW1 replicates the BW algorithm with one equilibrium update for each step in one iteration of the BW algorithm. In total, there are 4L steps in one iteration of the BW algorithm, equalling the number of subnetworks (table 1). BW2 combines all 4L steps in one iteration of the BW algorithm, corresponding to the simultaneous calculation of the forward and backward algorithm, the E-step and the M-step in one iteration. BW3 makes all updates simultaneously. BW3 is a feedback system where the concentrations of all species are adjusted continuously.

Proofs of the following statements are in appendix A.

Theorem 4.1. —

If the BW algorithm and BW1 are initiated at the same point $({\theta }_0,{\psi }_0)\in {\Theta }_0$, then their equilibria always exist and agree.

Theorem 4.2. —

If BW1 and BW2 are initiated at the same point $({\theta }_0,{\psi }_0)\in {\Theta }_0$, then their equilibria always exist and agree. Furthermore, this equilibrium is globally asymptotically stable for the BW2 dynamics and convergence is exponentially fast, subject to the invariant subspace defined by the initialization of θ′, ψ′: the sum of the entries of each row of θ′(t), respectively, ψ′(t), is one.

The two theorems imply that the initial conditions of the species other than (θ′, ψ′) are irrelevant, hence we need not be concerned with these. Furthermore, BW1 and BW2 compute the same parameter value as the BW algorithm provided $({\theta }_0,{\psi }_0)\in {\Theta }_0$. Theorems 4.1 and 4.2 might not be true if (θ₀, ψ₀) is at the boundary $\Theta \setminus {\Theta }_0$; see below for discussion. In particular, an equilibrium of BW3 at the boundary might not be an equilibrium of the BW algorithm.

Theorem 4.3. —

The sets of positive equilibria of BW1, BW2 and BW3 agree.

Any of the three algorithms as well as the BW algorithm might have several positive equilibria depending on the observed data and the initial point. In general, one should expect coexistence of boundary and positive equilibria [37,39].

Lemma 4.4. —

The solution to the ODE system of BW3 exists for all times and any initial condition. Furthermore, assume $({\theta }^{\prime }(t),{\psi }^{\prime }(t))\to ({\theta }^{\ast },{\psi }^{\ast })\in {\Theta }_0$ as t → ∞ for some initial condition, then x(t) → x* for some positive vector x*, where x(t) denotes the vector of concentrations of all species except the species θ′, ψ′.

If the initial point (θ₀, ψ₀) belongs to $\Theta \setminus {\Theta }_0$, then the dynamics of BW1/BW2 and BW3 might differ. Imagine that one or more of the (time-dependent) reaction rates are set to zero in figure 3. This could happen in different ways. In figure 3b, the graph is broken into two, effectively replacing one conserved quantity with two conserved quantities, one for each subgraph (one graph has only one node, the other six). The equilibrium depends on how much ‘mass’ is allocated to each subgraph initially. The same is the case if two dead-end nodes are created, nodes from which no mass can flow out; see figure 3c. Also here, an additional conserved quantity is created. See lemma A.3 for further details.

Figure 3.

Each subnetwork in table 1 is a catalytic mono-molecular reaction network with time-dependent reaction rates. The designated species with indices h*, (h*, h*) or v*, depending on the subnetwork, correspond to the nodes in the middle of the reaction graphs. (a) If the reaction rates are all positive, then the reaction graph is strongly connected. (b) Three reaction rates are zero, breaking the graph into two connected components and creating one node (blue) from which all mass eventually disappears. (c) Two reaction rates are zero, creating two dead-end nodes (blue) from which mass cannot flow out. All mass eventually accumulates in the blue nodes.

To complete the story, discrepancies between BW1 and BW2 (and similarly BW3) might also be found if some reaction rates eventually become zero. We illustrate this with a simple example. Imagine a two-species reaction network X₁ → X₂ with rate κ(t) = e^−t, t ≥ 0, and conserved amount T = x₁(t) + x₂(t). It has solution x₁(t) = x₁(0)exp (e^−t − 1) → x₁(0)/e as t → ∞, whereas the limiting reaction network with zero reaction rate has constant solution. Thus, a trajectory of the limiting reaction network only approximates a trajectory of the reaction network with time-dependent reaction rate, if the initial condition of the latter is e ≈ 2.71 times that of the former.

5. Multiple sequences and unknown π

If there are multiple observed sequences, v₁, …, v_R, $v_i=(v_{i1},\dots ,v_{i{L}_i})$, i = 1, …, R, potentially of different length, then the forward and backward algorithms in the BW algorithm are replaced with R forward and backward algorithms, one for each sequence and initialized with the same values of (θ, ψ) and π [41]. Similarly, the E-step is replaced by R E-steps, one for each sequence. The main difference lies in the M-step, which is replaced by [41]

$${\pi }_g^{\prime }=\frac{1}{R}\sum _{i=1}^R{\gamma }_{ig},{\theta }_{gh}^{\prime }=\frac{\sum _{i=1}^R\sum _{\ell =1}^{L_i-1}{\xi }_{i\ell gh}}{\sum _{i=1}^R\sum _{\ell =1}^{L_i-1}\sum _{\hspace{0.17em}f\in H}{\xi }_{i\ell gf}}$$

and

$${\psi }_{hw}^{\prime }=\frac{\sum _{i=1}^R\sum _{\ell =1}^{L_i}{\gamma }_{i\ell h}{\delta }_{w,v_{i\ell }}}{\sum _{i=1}^R\sum _{\ell =1}^{L_i}{\gamma }_{i\ell h}},$$

where π now is updated and the additional index i in γ_iℓh and ξ_iℓgh refers to the ith sequence, and otherwise similar to the one sequence algorithm (if π is considered known, the update step for π is just ignored). The forward and backward algorithms and the E-step are implemented similarly to the one sequence reaction network with a set of species for each observed sequence and common species for θ, ψ and π. The M-step might be implemented by the reactions

$${\pi }_g^{\prime }+{\gamma }_{i1g^{\ast }}\to {\pi }_{g^{\ast }}^{\prime }+{\gamma }_{i1g^{\ast }}$$ 5.1

$${\pi }_{g^{\ast }}^{\prime }+{\gamma }_{i1g}\to {\pi }_g^{\prime }+{\gamma }_{i1g}$$ 5.2

$$\begin{array}{rl}{\theta }_{gh}^{\prime }+{\xi }_{i\ell g{h}^{\ast }}& \to {\theta }_{g{h}^{\ast }}^{\prime }+{\xi }_{i\ell g{h}^{\ast }}\\ {\theta }_{g{h}^{\ast }}^{\prime }+{\xi }_{i\ell gh}& \to {\theta }_{gh}^{\prime }+{\xi }_{i\ell gh}\\ {\psi }_{hw}^{\prime }+{\gamma }_{i\ell h}+E_{\ell v^{\ast }}& \to {\psi }_{h{v}^{\ast }}^{\prime }+{\gamma }_{i\ell h}+E_{\ell v^{\ast }}^i\\ {\psi }_{h{v}^{\ast }}^{\prime }+{\gamma }_{i\ell h}+E_{\ell w}& \to {\psi }_{hw}^{\prime }+{\gamma }_{i\ell h}+E_{\ell w}^i,\end{array}$$

where $E_{\ell w}^i$ is defined similarly to E_ℓw, but for the ith sequence alone. All these reactions take the same mono-molecular form as in the one sequence algorithm, resulting in statements analogous to theorems 4.1–4.3 (with analogous proofs). The reaction networks for the R sequences are executed simultaneously for the R sequences in the three different ways BW1, BW2 and BW3.

6. Examples

To illustrate the performance of BW3 versus the BW algorithm, we simulated an observed sequence of length L = 100 from an HMM with two hidden states and two visible states v₁, v₂; see appendix A for details and figure 4. Out of the 100 symbols, 49 were v₁, while 51 were v₂. Then, we ran the two algorithms for different choices of initial values with π = (0.5, 0.5) fixed. In one case, the two algorithms returned the same estimated values (figure 4a), namely

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.5& 0.5\\ 0.5& 0.5\end{array}\right],\left[\begin{array}{cc}0.51& 0.49\\ 0.51& 0.49\end{array}\right]),$$

far from the true values (see appendix A), while in the second case (figure 4b), the two algorithms returned markedly different values. For the BW algorithm, we obtained

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.150& 0.850\\ 0.998& 0.002\end{array}\right],\left[\begin{array}{cc}0.642& 0.358\\ 0.356& 0.644\end{array}\right]),$$

with log-likelihood −68.5617, while for BW3, we obtained values on the boundary,

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.000& 1.000\\ 0.229& 0.771\end{array}\right],\left[\begin{array}{cc}0.000& 1.000\\ 0.631& 0.369\end{array}\right]),$$

with log-likelihood −68.6750.

Figure 4.

Dynamics of the BW3 and the BW algorithm for an HMM with two hidden states and two visible states for an observed sequence of length L = 100, and different initialization. (a) Log-likelihood of the sequence for BW3 (left) compared to BW algorithm (right) starting with equal initial parameter values. (b) Log-likelihood of the sequence for the BW3 (left) compared to BW algorithm (right) starting with random initial parameter values. In both cases, the likelihood is non-decreasing over time/iterations. See the main text and appendix A for details.

As a second example, we study a small observed sequence v₂, v₁, v₂, v₁, v₂ of length 5, generated from an HMM with two hidden states and two visible states. With initial parameter values given in appendix A, the BW algorithm returns the boundary equilibrium

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right],\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]),$$

while the BW3 returns a different boundary equilibrium

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.366& 0.634\\ 1& 0\end{array}\right],\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]).$$

Both examples illustrate that at the boundary different things might happen. In the latter case, the BW3 equilibrium belongs to $\Theta$. However, when initiated at that point, the BW algorithm returns the first equilibrium point. Hence, the BW3 equilibrium is not an equilibrium of the BW algorithm (as the equilibrium in not positive, then theorem 4.3 is not contradicted).

We further study an example where the initial probabilities can be tuned by adding new reactions according to equations (5.1)–(5.2). Consider a casino game which depends on the output of a three-sided die. A fair die would have equal probabilities of each outcome [42]. However, the casino can switch to an unfair die with unequal outcome probabilities. The player only observes the die outcome sequence. An HMM with two hidden states of the casino (Honest and Dishonest) with three visible states as the outcome of the die can be used to predict if and when the casino is dishonest [42]. We use the following HMM:

$$({\pi }^{\prime },{\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{c}1.0\\ 0.0\end{array}\right],\left[\begin{array}{cc}0.95& 0.05\\ 0.25& 0.75\end{array}\right],\left[\begin{array}{ccc}0.34& 0.33& 0.33\\ 0.01& 0.01& 0.98\end{array}\right])\hspace{0.17em}$$

to generate sample data with 300 rolls, where the Dishonest state has unequal probabilities of the die roll. The statistics of the generated sample is visualized in figure 5a. We then start with a random initial distribution of the HMM parameters and train it using both BW and BW3 on the training data (first 150 rolls) and infer the hidden states based on the learned parameters on the test data (next 150 rolls). BW converges to

$$({\pi }^{\prime },{\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{c}0.0\\ 1.0\end{array}\right],\left[\begin{array}{cc}0.9& 0.01\\ 0.03& 0.96\end{array}\right],\left[\begin{array}{ccc}0.03& 0.07& 0.90\\ 0.37& 0.35& 0.28\end{array}\right]),$$

with a log-likelihood of −147.9. BW3 converges to

$$({\pi }^{\prime },{\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{c}0.81\\ 0.19\end{array}\right],\left[\begin{array}{cc}0.48& 0.52\\ 0.43& 0.57\end{array}\right],\left[\begin{array}{ccc}0.28& 0.23& 0.49\\ 0.27& 0.35& 0.38\end{array}\right]),$$

with a log-likelihood of −157.4. The inference result of the training is visualized in figure 5b. Spikes indicate the switch to the hidden state Dishonest. We find that both BW and BW3 predict the hidden state sequence with high accuracy when compared with the HMM used for generating the sequences. More details can be found in appendix A.

Figure 5.

Detection of a dishonest casino. Inferred statistics in the dishonest casino example. (a) Outcome probabilities of the loaded and fair die estimated from the generator HMM for 300 die rolls. (b) Prediction of the hidden states on the test data (last 150 rolls) using the generator HMM and after training BW and BW3 on first 150 rolls with a random initial distribution of parameters.

7. Discussion

Some machine learning algorithms like gradient descent are based on continuous-time dynamical systems while others like message passing appear essentially discrete. Before our work, the BW algorithm has fallen in the second category. Performing the different steps in order was seen as an important part of the algorithm. Here, we have shown a continuous-time dynamical system based on the BW algorithm which implements HMM learning. Our work has exposed that there is a greater design space for BW algorithm implementations than was previously known. Specifically, the steps need not be run to completion, and need not be run in order. This reduces the synchronization burden for distributed implementations of such extended BW algorithms. As a result, our algorithm can be implemented on a distributed network of edge devices, and in this manner serves as a federated learning scheme. Our work exposes that reaction networks are a natural design language to think about the design of machine learning algorithms when the data and computation need to be handled in a distributed manner with minimum overhead of synchronization.

Our implementation of this algorithm is explicitly given in the form of a chemical reaction network. This opens the possibility of molecular implementations of this algorithm. If implemented in an artificial cell, it might provide the cell with the ability to sample possible realities, and act according to these imaginings. This is important because the world of molecules is a noisy world. To obtain exquisite control over a large number of molecules—which is one of the main goals of nanotechnology—requires algorithms that will be robust to such noise. Denoising or error correction algorithms are essentially statistical algorithms of the kind that we have implemented here. Thus, not only does our work point to future technological directions in nanotechnology, we believe reaction network implementations of algorithms of this kind are inevitable when attempting to exquisitely control large ensembles of molecules.

One of the big challenges in biology has been the immense complexity of living cells. While on one hand, cells are capable of behaviours of incredible sophistication, on the other hand our imagination about their workings has been limited to a very low level of abstraction described by systems biology. To understand the behaviour of cells will require the invention of layers of abstraction above that of systems biology. These layers will have to explain the algorithmic power of biochemical reaction networks. Biochemical reaction networks in living systems are known to be performing inference. There may be opportunity to understand them from the vantage point of our work which is giving concrete schemes by which reaction networks can lead to inference. Our design has reproduced certain characteristics found in biochemical reaction networks like the ubiquitous presence of enzymes and futile cycles, for example in phosphorylation and dephosphorylation of the same site on a protein. This reproduction has led to a novel idea to explain futile cycles: their purpose is to achieve probabilistic inference. For example, reactions of the form $A\to A^{\ast }$ catalysed in the forward direction by a kinase and in the backward direction by a phosphatase are shown here to be the paradigmatic form of reactions needed to carry out probabilistic inference. A research programme to pursue this idea to its logical conclusions is suggested by the work here, but is beyond the scope of the current paper.

Appendix A

A.1. The ODE system of the Baum–Welch reaction network

With mass-action kinetics the ODE system of the reaction network in table 1 becomes

$${\dot{\alpha }}_{1h}={\alpha }_{1h^{\ast }}{\pi }_h{\psi }_{h{v}_1}-{\alpha }_{1h}{\pi }_{h^{\ast }}{\psi }_{h^{\ast }{v}_1},$$ A 1

$${\dot{\alpha }}_{\ell h}={\alpha }_{\ell h^{\ast }}\sum _{g\in H}{\alpha }_{\ell -1,g}{\theta }_{gh}{\psi }_{h{v}_{\ell }}-{\alpha }_{\ell h}\sum _{g\in H}{\alpha }_{\ell -1,g}{\theta }_{g{h}^{\ast }}{\psi }_{h^{\ast }{v}_{\ell }},$$ A 2

$${\dot{\beta }}_L=0,$$ A 3

$${\dot{\beta }}_{\ell h}={\beta }_{\ell h^{\ast }}\sum _{g\in H}{\beta }_{\ell +1,g}{\theta }_{gh}{\psi }_{h{v}_{\ell +1}}-{\beta }_{\ell h}\sum _{g\in H}{\beta }_{\ell +1,g}{\theta }_{g{h}^{\ast }}{\psi }_{h^{\ast }{v}_{\ell +1}},$$ A 4

$${\dot{\gamma }}_{\ell h}={\gamma }_{\ell h^{\ast }}{\alpha }_{\ell h}{\beta }_{\ell h}-{\gamma }_{\ell h}{\alpha }_{\ell h^{\ast }}{\beta }_{\ell h^{\ast }},$$ A 5

$$\begin{array}{rl}& {\dot{\xi }}_{\ell gh}={\xi }_{\ell h^{\ast }{h}^{\ast }}{\alpha }_{\ell g}{\theta }_{gh}{\beta }_{\ell +1,g}{\psi }_{h{v}_{\ell +1}}\\ & -{\xi }_{\ell gh}{\alpha }_{\ell h^{\ast }}{\theta }_{h^{\ast }{h}^{\ast }}{\beta }_{\ell +1,h^{\ast }}{\psi }_{h^{\ast }{v}_{\ell +1}},\end{array}$$ A 6

$${\dot{\theta }}_{gh}^{\prime }={\theta }_{g{h}^{\ast }}^{\prime }\sum _{\ell =1}^{L-1}{\xi }_{\ell gh}-{\theta }_{gh}^{\prime }\sum _{\ell =1}^{L-1}{\xi }_{\ell g{h}^{\ast }}$$ A 7

$$\mathrm{and}{\dot{\psi }}_{hw}^{\prime }={\psi }_{h{v}^{\ast }}^{\prime }\sum _{\ell =1}^L{\gamma }_{\ell h}{\delta }_{w,v_{\ell }}-{\psi }_{hw}^{\prime }\sum _{\ell =1}^L{\gamma }_{\ell h}{\delta }_{v^{\ast },v_{\ell }},$$ A 8

where h* ∈ H and v* ∈ V are fixed states, $h\ne h^{\ast }$ in (A 1)–(A 5), $(g,h)\ne (h^{\ast },h^{\ast })$ in (A 6), g ∈ H, $h\ne h^{\ast }$ in (A 7), and h ∈ H, $w\ne v^{\ast }$ in (A 8). In (A 2), ℓ = 2, …, L, in (A 4) and (A 6), ℓ = 1, …, L − 1, and in (A 5), ℓ = 1, …, L. The equations for the remaining species are obtained as ${\dot{\alpha }}_{1h^{\ast }}=-{\sum }_{h\ne h^{\ast }}{\dot{\alpha }}_{1h}$, and similarly for the other species, reflecting that the sum $A_{\ell }={\sum }_h{\alpha }_{1h}(t)$ is conserved through time. The explicit dependence on time is suppressed in (A 1)–(A 8). The species E_ℓw are left out for convenience.

A.2. Reformulation of the Baum–Welch algorithm

The BW algorithm can be formulated in an equivalent way that exposes the relationship to the dynamics of the BW reaction network; see the electronic supplementary material for proof and details. Define α_ℓh, β_ℓh, h ∈ H, ℓ = 1, …, L, recursively by

$${\alpha }_{1h}=\frac{A_1{\pi }_h{\psi }_{h{v}_1}}{\sum _{g\in H}{\pi }_g{\psi }_{g{v}_1}},{\alpha }_{\ell h}=\frac{A_{\ell }\sum _{g\in H}{\alpha }_{\ell -1,g}{\theta }_{gh}{\psi }_{h{v}_{\ell }}}{\sum _{\hspace{0.17em}f\in H}\sum _{g\in H}{\alpha }_{\ell -1,f}{\theta }_{\hspace{0.17em}fg}{\psi }_{g{v}_{\ell }}}$$

and

$${\beta }_{Lh}=\frac{B_L}{\mathrm{\#}H},{\beta }_{\ell h}=\frac{B_{\ell }\sum _{g\in H}{\theta }_{hg}{\psi }_{g{v}_{\ell +1}}{\beta }_{\ell +1,g}}{\sum _{\hspace{0.17em}f\in H}\sum _{g\in H}{\theta }_{\hspace{0.17em}fg}{\psi }_{g{v}_{\ell +1}}{\beta }_{\ell +1,g}},$$

where A_ℓ, B_ℓ are arbitrary positive constants, ℓ = 1, …, L. These two recursions correspond to the forward and backward algorithms of the standard BW algorithm, respectively, with the essential difference being that α_ℓh, β_ℓh scale to A_ℓ, B_ℓ, respectively. The E-step and the M-step are identical to the corresponding steps of the standard BW algorithm.

In this formulation, α_ℓh, β_ℓh, h ∈ H, in the BW algorithm can be seen as the equilibrium values of the corresponding reaction networks, ${\mathcal{R}}_{\ell }^{\alpha },{\mathcal{R}}_{\ell }^{\beta }$; see lemma A.3. As the sum of γ_ℓh, ξ_{ℓg h}, θ_ℓh, respectively, over h is one, a similar interpretation applies in this case. Similarly for ψ_ℓv, where the sum over v is one.

A.3. Mono-molecular reaction networks

Properties of mono-molecular reaction networks are of particular importance for the proofs of the statements in the main text.

Consider a mono-molecular reaction network with n species, X₁, …, X_n, and reactions

$$X_{r_i}\stackrel{{\kappa }_i(t)}{\to }{X}_{\hspace{0.17em}{p}_j},\text{for}\hspace{0.17em}i=1,\dots ,m,$$

where $r_i\ne p_i$, r_i, p_i ∈ {1, …, n} and κ_i(t), i = 1, …, m, are mass-action time-dependent reaction rate constants, t ≥ 0. We assume κ_i(t) > 0 and let κ(t) = (κ₁(t), …, κ_m(t)) be the vector of reaction rate constants. Furthermore, we assume there is at most one reaction i such that (r_i, p_i) = (r, p) ∈ {1, …, n}². (If there is more than one reaction, the reaction rate constants are summed.)

The (non-autonomous) ODE system of the reaction network is

$${\dot{x}}_i=-x_i\hspace{0.17em}\sum _{\hspace{0.17em}j\hspace{0.17em}:\hspace{0.17em}{r}_j=i}{\kappa }_j(t)+\sum _{\hspace{0.17em}j\hspace{0.17em}:\hspace{0.17em}{p}_j=i}{\kappa }_j(t)x_{\hspace{0.17em}{p}_j},i=1,\dots ,n.$$

Define the n × n matrix A(t) = (a_ij(t))_{i,j=1, …,n} by

$$a_{ij}(t)={\kappa }_k(t),\text{if there is }k\in \{1,\dots ,m\}\hspace{0.17em}\text{such that}\hspace{0.17em}(r_k,p_k)=(j,i)$$

and

$$a_{ii}(t)=-\sum _{\hspace{0.17em}j\hspace{0.17em}:\hspace{0.17em}{r}_j=i}{\kappa }_j(t).$$

Then, the ODE system can be written as

$$\dot{x}=A(t)x.$$ A 9

The matrix A(t) is a Laplacian matrix with column sums zero, the off-diagonal elements are non-negative and the diagonal elements are non-positive. In particular, the non-zero eigenvalues of A(t) have negative real part [44,45].

The Laplacian graph G corresponding to A(t) is independent of time as κ(t) is positive for t ≥ 0 by assumption. Let λ be the number of connected components of G and τ the number of maximal strongly connected subgraphs. Always τ ≥ λ. In figure 3a: τ = λ = 1, figure 3b: τ = λ = 2, and figure 3c: τ = 2, λ = 1. Each connected component of the Laplacian graph gives rise to a conserved quantity, namely the sum of the variables in the connected component:

$$T_j=\sum _{i\in G_j}{x}_i,T_j\ge 0,$$

where G_j is the jth connected component, j = 1, …, λ. The non-negative vector $T=(T_1,\dots ,T_{\lambda })$ defines an invariant subspace of (A 9). Let $z_j\in {\mathbb{R}}^n$ be the vector with ith component 1 if i ∈ G_j and otherwise zero. Then, z_j is a left kernel vector of A(t), z_jA(t) = 0, and a linear first integral of (A 9).

Lemma A.1. —

The following hold:

• i The rank of A(t) equals n − λ if and only if τ = λ.
• ii If τ = λ, then $z_1,\dots ,z_{\lambda }$ is a basis of the left kernel of A(t). Any linear first integral is a linear combination of $z_1,\dots ,z_{\lambda }$.
• iii Assume A(t) = A is independent of time. If τ = λ, then there is a unique non-negative exponentially and globally stable equilibrium within each invariant subspace given by T ≥ 0. The equilibrium is positive if and only if the connected components are strongly connected and T > 0.

Lemma A.2. —

Assume τ = λ and that κ(t), t ≥ 0, converges exponentially fast towards $\kappa =({\kappa }_1,\dots ,{\kappa }_m)\in {\mathbb{R}}_{>0}^m$ as t → ∞, i.e. there exists γ₁ > 0 and K₁ > 0 such that

$$\Vert \kappa (t)-\kappa \Vert \le K_1{e}^{-{\gamma }_{1}t}\text{for}\hspace{0.17em}t\ge 0.$$

Let A be the matrix A(t) with κ inserted for κ(t). Then, any solution to $\dot{x}=A(t)x$ converges exponentially fast towards the unique globally stable equilibrium $x^{\ast }\in {\mathbb{R}}_{>0}^n$ of the ODE system $\dot{x}=Ax$ within the relevant invariant subspace, provided T > 0. That is, there exists γ₂ > 0 and K₂ > 0 such that

$$\Vert x(t)-x^{\ast }\Vert \le K_2{e}^{-{\gamma }_{2}t}\text{for}\hspace{0.17em}t\ge 0.$$

Lemma A.3. —

Consider a mono-molecular reaction network with reactions

$$X_i\underset{{\kappa }_i}{\overset{{\kappa }_n}{\rightleftharpoons }}{X}_n,\text{for}\hspace{0.17em}i=1,\dots ,n-1,$$

such that κ_i > 0 for i = 1, …, n − 1 (then the Laplacian graph is as in figure 3a). In particular, λ = t = 1. Then, the unique positive equilibrium for T > 0 is

$$x^{\ast }=(x_1^{\ast },\dots ,x_n^{\ast })\mathrm{and}{x}_i^{\ast }=\frac{T{\kappa }_i}{{\sum }_{\hspace{0.17em}j=1}^n{\kappa }_j},\text{for}\hspace{0.17em}i=1,\dots ,n.$$

A.4. Proofs

We provide proofs of theorems 4.1–4.3 and lemma 4.4. The remaining statements are proven in the electronic supplementary material, except lemma A.3 which is straightforward. We assume π > 0, L ≥ 2, and note that similar arguments apply if π ≥ 0 and L ≥ 3.

A.4.1. Proof of theorem 4.1

It is sufficient to demonstrate that (θ_n, ψ_n), n ≥ 0, are the same and positive after each iteration of the BW algorithm and BW1. For this, assume θ, ψ (and π) are positive. Then the subnetwork ${\mathcal{R}}_1^{\alpha }$ is mono-molecular with species α_1h, h ∈ H, and fixed positive reaction rates. The Laplacian graph is strongly connected and τ = λ = 1. Lemma A.1(iii) gives global stability of the unique positive equilibrium for the concentrations of the species α_1h, h ∈ H, identical to that of the BW algorithm (compare lemma A.3). This argument is repeated inductively for each subnetwork (in the order listed for BW1). Specifically, for the kth subnetwork (k = 2, …, 4L), the equilibrium of the k′th subnetwork, k′ < k, is positive by induction hypothesis. Hence the reaction rates for the kth subnetwork are positive and fixed, given by the equilibrium values of the subnetworks k′ < k and θ, ψ. Hence, the Laplacian graph is strongly connected with τ = λ = 1. Using lemma A.1(iii) completes the argument. In particular, (θ′, ψ′) is positive and the same for the BW algorithm and BW1.

A.4.2. Proof of theorem 4.2

The proof is similar to that of theorem 4.1. The layered structure of BW2 implies that the equilibrium of the species in the kth subnetwork is unaffected by the presence of the k′th subnetwork, k′ > k. Assume θ, ψ (and π) are positive. Then the subnetwork ${\mathcal{R}}_1^{\alpha }$ is mono-molecular with species α_1h, h ∈ H, and fixed positive reaction rates. The Laplacian graph is strongly connected and τ = λ = 1. Lemma A.1(iii) gives exponential stability of the unique positive equilibrium for the concentrations of the species α_1h, h ∈ H. By lemma A.1(iii), it is the same as that of BW1. Consider next the subnetwork ${\mathcal{R}}_2^{\alpha }$. It is mono-molecular with positive time-dependent reaction rates, given by the concentrations of the species α_1h, h ∈ H, at time t ≥ 0 and θ, ψ. Hence, the Laplacian graph is strongly connected and τ = λ = 1. Applying lemma A.2 yields exponential convergence towards the unique positive equilibrium for the concentrations of the species α_2h, h ∈ H. This equilibrium is the same as that of BW1 for ${\mathcal{R}}_2^{\alpha }$ (lemma A.2). The theorem now follows by repeated application of the above procedure for all subnetworks.

A.4.3. Proof of theorem 4.3

If the catalyst species concentrations of a subnetwork at equilibrium are positive, then the unique equilibrium concentrations of the non-catalyst species are likewise positive (lemma A.3). Consequently, if θ, ψ (and π) are positive, then the unique equilibrium concentrations of the species α_{1 h}, h ∈ H, in ${\mathcal{R}}_1^{\alpha }$ are positive. Inductively, the equilibrium concentrations in any subnetwork are positive, if θ, ψ (and π) are positive.

For BW1 and BW2, if $(\theta ,\psi )\in {\Theta }_0$ is an equilibrium, then (θ′, ψ′) is well-defined and (θ′, ψ′) = (θ, ψ). Consequently, the species concentrations at equilibrium solve the same equations in the three cases, thus the sets of equilibria are the same. The same reasoning gives the uniqueness statement. Theorem 4.1 gives uniqueness for the BW algorithm.

A.4.4. Proof of lemma 4.4

The first part follows by standard arguments as the trajectories are confined to a compact space. The second part follows from [46, theorem 1.2] and lemma A.1(iii).

A.5. Simulated examples

We consider an HMM with two hidden states and two visible states v₁, v₂, and generate a random sequence of length L = 100 from π = (0.5, 0.5) (fixed throughout) and

$$(\theta ,\psi )=(\left[\begin{array}{cc}0.2& 0.8\\ 0.7& 0.3\end{array}\right],\left[\begin{array}{cc}0.75& 0.25\\ 0.3& 0.7\end{array}\right]).$$

The generated sequence is

$$\begin{array}{rl}& (v_2,v_1,v_2,v_2,v_1,v_2,v_2,v_1,v_2,v_1,v_1,v_1,v_2,v_1,v_1,v_1,v_2,\\ & v_1,v_1,v_2,v_2,v_2,v_2,v_1,v_1,v_2,v_2,v_1,v_1,v_1,v_2,v_1,v_2,v_2,\\ & v_2,v_2,v_1,v_1,v_1,v_2,v_2,v_2,v_1,v_2,v_1,v_2,v_1,v_1,v_2,v_2,v_2,\\ & v_2,v_1,v_1,v_1,v_1,v_2,v_1,v_2,v_1,v_2,v_1,v_1,v_1,v_1,v_1,v_1,v_2,\\ & v_1,v_2,v_2,v_1,v_2,v_1,v_2,v_1,v_2,v_1,v_2,v_1,v_2,v_1,v_1,v_2,v_2,\\ & v_2,v_2,v_1,v_2,v_2,v_1,v_1,v_1,v_1,v_1,v_1,v_2,v_2,v_2,v_2).\end{array}$$

The log-likelihood of the sequence is −68.7955. We trained the HMM using BW3 and the BW algorithm, starting from two different initials values: (1) equal transition and emission probabilities

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.5& 0.5\\ 0.5& 0.5\end{array}\right],\left[\begin{array}{cc}0.5& 0.5\\ 0.5& 0.5\end{array}\right])\hspace{0.17em}$$

and (2) randomly chosen values

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.774& 0.226\\ 0.427& 0.573\end{array}\right],\left[\begin{array}{cc}0.916& 0.084\\ 0.510& 0.490\end{array}\right]).\hspace{0.17em}$$

The other species in the reactions of the BW3 were initialized with (1) equal concentrations and (2) concentrations obtained from one step of the BW algorithm and appropriate normalization.

In the second example, the initial values are π = (0.6, 0.4) (fixed), and initial transition and emission probabilities

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.7& 0.3\\ 0.4& 0.6\end{array}\right],\left[\begin{array}{cc}0.6& 0.4\\ 0.6& 0.4\end{array}\right]).$$

In the final example of a dishonest casino, the randomly chosen initial values are π = (0.83, 0.17), and initial transition and emission probabilities

$$({\theta }^{\prime },{\psi }^{\prime })=(\left[\begin{array}{cc}0.1& 0.9\\ 0.02& 0.98\end{array}\right],\left[\begin{array}{ccc}0.12& 0.08& 0.80\\ 0.07& 0.89& 0.04\end{array}\right]).$$

For all of the examples, BW3 was simulated using the ODEINT package from the Scipy library for a resolution of 100 points starting with time 0 to the final time. For the BW algorithm, the hmmlearn package was used from the Scikit library.

Data accessibility

Code used for the simulation of examples in this paper may be found at: https://github.com/GeekyPeas/Chemical-Baum-Welch-Algorithm.

The data are provided in electronic supplementary material [43].

Authors' contributions

C.W.: conceptualization, formal analysis, funding acquisition, methodology, writing—original draft; A.B.: conceptualization, methodology, software, validation, writing—review and editing; A.S.: conceptualization, funding acquisition, methodology, software, validation, writing—review and editing; M.G.: conceptualization, writing—original draft.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

The work presented in this article is supported by Novo Nordisk Foundation, grant no. NNF19OC0058354. A.B. thanks Bharti Center for Communication, IIT Bombay. A.S. thanks the INSPIRE DST Scholarship for Higher Education.