Fig. 1.

A schematic illustration of RNN cell models. (a) The nonlinear basic recurrent cell unit; (b) Long short-term memory unit; (c) Gated recurrent unit; (d) The interconnections in a common recurrent hidden layer. Neurons in recurrent layer are fully interconnected and new hidden states can be influenced by all former states. Squares indicate linear combination and nonlinearity. Circles indicate element wise operations. Gates in the units control the information flow between adjacent time points.