10. Deep Reinforcement Learning

Chapter 10
Deep Reinforcement Learning

This lecture provides a very brief introduction to the field of reinforcement learning (RL) and its appearance in deep learning. The canonical reference for reinforcement learning is Sutton and Barto [97]. Murphy [75] is also an excellent new reference for deep RL. We start with a presentation of the Markov decision process and classic reinforcement learning algorithms. We discuss some problems such as exploration versus exploitation and issues with large state or action spaces. We then show how deep learning can address the latter problem by approximating lookup tables with learnable functions, and present various flavours of deep reinforcement learning algorithms.

A Markov decision process (MDP). The agent observes the current state of the world and decides on an action to take . In response the state of the world is chan
Figure 127: A Markov decision process (MDP). The agent observes the current state of the world \(s_t\) and decides on an action to take \(a_t\). In response the state of the world is changed \(s_{t+1}\) and the agent is rewarded (or punished) for its action with \(r_{t+1}\).

A large area of artificial intelligence research is concerned with planning and decision making. When a series of decisions need to be made by an agent to execute a plan or achieve a goal this is known as sequential decision making [89]. A standard model for sequential decision making is the Markov decision process (MDP) where a single agent is interactive with its environment, often depicted as in Figure 127. Under this model and agent observes the state of the world \(s_t\) at some time \(t\). You can think of the state as being encoded as an \(n\)-dimensional vector, and may include things that the agent has remembered from previous time steps.1 With the current state of the world at hand, the agent decides on an action to take \(a_t\). This results in a new world state \(s_{t+1}\), which is either a deterministic or a probabilistic function of the previous world state and the action taken. The agent also receives a reward (or punishment) \(r_{t+1}\), which is a signal that tells the agent how well it is performing.

  1. 1. A variant of the Markov decision process (MDP) known as a partially observable Markov decision process (POMDP) has the agent only observe part of the world state, with other parts kept hidden from the agent.