Figure 1. Actor-critic architecture.

The environment (E) informs the critic and the actor about the current state (s). In addition, it transmits the current reward information (r) to the critic. The critic calculates based on the value function of the current and previous state and the reward information the TD error signal, which is used to update the policy and the value function of the previous state. The actor selects based on the policy of the current state an action (a), which is read out by the environment. (Figure adapted from [1]).