10. Deep Reinforcement Learning › 10.2 So What Exactly is Reinforcement Learning

Section 10.2
So What Exactly is Reinforcement Learning

Reinforcement learning is about learning a function from states to actions (or behaviour), i.e., a policy,

\begin{align} \pi_\theta : \cS \to \cA \tag{288}\end{align}

We can contrast this to standard supervised learning seen in earlier lectures of this course where the task is to learn a function from an input space \(\cX\) to an output space \(\cY\),

\begin{align} f_\theta : \cX \to \cY \tag{289}\end{align}

However, unlike supervised machine learning where predictions can be evaluated immediately, the predicted action in reinforcement learning cannot be evaluated directly. Rather actions receive an interim positive or negative reward, and it is only after a long sequence of actions can we really determine whether a policy is good or not. Consequently the goal of reinforcement learning is to find a policy \(\pi_\theta\) that maximizes the total expected reward and not a function \(f_\theta\) that minimizes a loss over outputs predicted on, what is usually assumed to be, independently and identically distributed (IID) training data. Indeed, in reinforcement learning training data is not IID since current actions \(a_t\) affect future states \(s_\tau\) (\(\tau > t\)).

The reason why we formally maximize expected reward (and not simply reward) is that the world is typically stochastic and hence exhibits non-deterministic dynamics, i.e., the next state is sampled from a distribution given the current state and action,

\begin{align} s_{t+1} \sim P(s' \mid s_t, a_t) \tag{290}\end{align}

Moreover, policies may be probabilistic,

\begin{align} a_t \sim \pi_\theta(a \mid s_t) \tag{291}\end{align}

and, in general, the reward can depend on not just the current state but also the transition to the next state,

\begin{align} r_t &= R(s_t, a_t, s_{t+1}) \tag{292}\end{align}

However, for simplicity in this lecture we will consider only \(R(s_t)\).

If we define the return (also known as the reward-to-go) as the discounted sum of rewards from time \(t\),

\begin{align} G_t &= r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \ldots \tag{293}\end{align}

the value function can be written as

\begin{align} V_{\pi}(s) &= E\left[ G_t \mid s_t = s \right] \tag{294}\\ &= R(s) + \gamma \sum_{s' \in \cS} P(s' \mid s) V(s') \tag{295}\end{align}

and this is the thing we want to maximize.1

The discount factor \(\gamma\) is a technicality that avoids infinite rewards (unless episodes are finite). If \(\gamma = 0\) then we only care about the immediate reward and not any potential rewards in the future. If \(\gamma = 1\) then future rewards are just as important as the immediate reward. It has been observed that humans often act with \(\gamma\) somewhere between zero and one, but not at either extreme.

  1. 1. We obtain \(P(s' \mid s)\) by summing \(P(s' \mid s, a)\pi(a \mid s)\) over actions.