2.12 Machine learning lecture 12 course notes  (Page 3/8)

Note that ${\pi }^{*}\left(s\right)$ gives the action $a$ that attains the maximum in the “max” in Equation  [link] .

It is a fact that for every state $s$ and every policy $\pi$ , we have

The first equality says that the $V^{{\pi }^{*}}$ , the value function for ${\pi }^{*}$ , is equal to the optimal value function $V^{*}$ for every state $s$ . Further, the inequality above says that ${\pi }^{*}$ 's value is at least a large as thevalue of any other other policy. In other words, ${\pi }^{*}$ as defined in Equation  [link] is the optimal policy.

Note that ${\pi }^{*}$ has the interesting property that it is the optimal policy for all states $s$ . Specifically, it is not the case that if we were starting in some state $s$ then there'd be some optimal policy for that state, and if we were starting in some other state $s^{\text{'}}$ then there'd be some other policy that's optimal policy for $s^{\text{'}}$ . Specifically, the same policy ${\pi }^{*}$ attains the maximum in Equation  [link] for all states $s$ . This means that we can use the same policy ${\pi }^{*}$ no matter what the initial state of our MDP is.

Value iteration and policy iteration

We now describe two efficient algorithms for solving finite-state MDPs. For now, we will consider only MDPs with finite state and action spaces ( $\left|S\right|<\infty ,\left|A\right|<\infty$ ).

The first algorithm, value iteration , is as follows:

1. For each state $s$ , initialize $V\left(s\right):=0$ .
2. Repeat until convergence $\{$
   • For every state, update $V\left(s\right):=R\left(s\right)+{max}_{a\in A}\gamma {\sum }_{s^{\text{'}}}{P}_{sa}\left(s^{\text{'}}\right)V\left(s^{\text{'}}\right)$ .
3. $\}$

This algorithm can be thought of as repeatedly trying to update the estimated value function using Bellman Equations  [link] .

There are two possible ways of performing the updates in the inner loop of the algorithm. In the first, we can first compute the new values for $V\left(s\right)$ for every state $s$ , and then overwrite all the old values with the new values. This is called a synchronous update. In this case, the algorithm can be viewed as implementing a “Bellman backup operator” that takes a current estimateof the value function, and maps it to a new estimate. (See homework problem for details.) Alternatively, we can also perform asynchronous updates. Here, we would loop over the states (in some order), updating the values one ata time.

Under either synchronous or asynchronous updates, it can be shown that value iteration will cause $V$ to converge to $V^{*}$ . Having found $V^{*}$ , we can then use Equation  [link] to find the optimal policy.

Apart from value iteration, there is a second standard algorithm for finding an optimal policy for an MDP. The policy iteration algorithm proceeds as follows:

1. Initialize $\pi$ randomly.
2. Repeat until convergence $\{$
   1. Let $V:=V^{\pi }$ .
   2. For each state $s$ , let $\pi \left(s\right):=arg{max}_{a\in A}{\sum }_{s^{\text{'}}}{P}_{sa}\left(s^{\text{'}}\right)V\left(s^{\text{'}}\right)$ .
3. $\}$

Thus, the inner-loop repeatedly computes the value function for the current policy, and then updates the policy using the current value function. (The policy $\pi$ found in step (b) is also called the policy that is greedy with respect to $V$ .) Note that step (a) can be done via solving Bellman's equations as described earlier, which in the case of a fixed policy, is just a setof $\left|S\right|$ linear equations in $\left|S\right|$ variables.

After at most a finite number of iterations of this algorithm, $V$ will converge to $V^{*}$ , and $\pi$ will converge to ${\pi }^{*}$ .