Fig 3. Example state representations.

a) Agent position (rodent image) in a maze whose hallways are indicated by grey. b) Punctate representation of the agent’s current state. Model-free behavior results from TD computation applied to this representation c,d) Possible successor representations of agent’s state. Model-based behavior may result from TD applied to this type of representation. The successor representation depends on the action selection policy the agent is expected to follow in future states. The figures show the representation of the current state under a random policy (c) versus a policy favoring rightward moves (d).