Fig. 6 |. Graphical model for the decoding process.

Circles represent random variables; doubled circles are deterministic functions of their inputs. a, The true generative process (above) and the encoder–decoder model (below). The true relationship between neural activity ($\mathbf{N}$), the speech-audio signal ($\mathbf{A}$) and word sequences ($\mathbf{W}$), denoted $P(\mathbf{a},\mathbf{w}\mid \mathbf{n})$, is unknown (although we have drawn the graph to suggest that $\mathbf{W}$ and $\mathbf{A}$ are independent given $\mathbf{N}$). However, we can observe samples from all three variables, which we use to fit the conditional model, $Q\left(\mathbf{a},\mathbf{w},{\mathbf{s}}^d\left({\mathbf{s}}^{\mathbf{e}}\right),{\mathbf{s}}^{\text{e}}(\mathbf{n})\mid \mathbf{n};\Theta \right)$, which is implemented as a neural network. The model separates the encoder states, ${\mathbf{S}}^e$, which directly generate the audio sequences, from the decoder states, ${\mathbf{S}}^d$, which generate the word sequences. During training, model parameters $\Theta$ are changed so as to make the model distribution, $Q$, over $\mathbf{A}$ and $\mathbf{W}$ look more similar to the true distribution, $P$. b, Detail of the graphical model for the decoder, unrolled vertically in sequence steps. Each decoder state is computed deterministically from its predecessor and the previously generated word or (in the case of the zeroth state) the final encoder state and an initialization token, $\langle \text{EOS}\rangle$.