Lecture 4: Model Free Control

Lecture 4 主要介绍无模型的 control，包含 MC control 和 TD control。

• On-policy learning
  • Direct experience
  • Learn to estimate and evaluate a policy from experience obtained from following that policy
• Off-policy learning
  • Learn to estimate and evaluate a policy using experience gathered from following a different policy

Monte Carlo Control

Monte Carlo with Exploring Starts

A Blackjack game is presented to elucidate MCES.

On-policy MC Control

Maintain an $ϵ$-greedy policy

Only achieve the best policy among the $ϵ$-greedy policies.

Greedy in the Limit of Infinite Exploration (GLIE)

• All state-action pairs are visited an infinite number of times $$\underset{i\to \mathrm{\infty }}{lim}{N}_i(s,a)\to \mathrm{\infty }$$
• Behavior policy (policy used to act in the world) converges to greedy policy $\underset{i\to \mathrm{\infty }}{lim}\pi (a\mid s)\to \arg \underset{a}{max}Q(s,a)$ with probability 1

A simple GLIE strategy is $ϵ$-greedy where $ϵ$ is reduced to 0 with the following rate: ${ϵ}_i=1/i$

GLIE Monte-Carlo control converges to the optimal state-action value function $Q(s,a)\to Q^{\ast }(s,a)$.

Off-policy MC Control

Require that the behavior policy be soft, to ensure each pair of state and action be visited.

TD Control

On-policy SARSA

Quintuple of events $(S_t,A_t,R_t,S_{t+1},A_{t+1})\to \text{SARSA}$

$$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha [R_{t+1}+\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t)]$$

SARSA for finite-state and finite-action MDPs converges to the optimal action-value, $Q(s,a)\to Q^{\ast }(s,a)$, under the following conditions (Robbins-Munro sequence)

1. The policy sequence ${\pi }_t(a\mid s)$ satisfies the condition of GLIE
2. The step-sizes ${\alpha }_t$ satisfy the Robbins-Munro sequence such that $$\begin{array}{rl}& \sum _{t=1}^{\mathrm{\infty }}{\alpha }_t=\mathrm{\infty }\\ & \sum _{t=1}^{\mathrm{\infty }}{\alpha }_t^2<\mathrm{\infty }\end{array}$$

A typical selection is ${\alpha }_t=o(1/t)$ .

Off-policy Q-learning

$$Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha [R_{t+1}+\gamma \underset{a}{max}Q(S_{t+1},a)-Q(S_t,A_t)]$$

Directly approximates $q^{\ast }$, the optimal state-action value function.

Converges to optimal $q^{\ast }$ if visit all $(s,a)$ pairs infinitely often and ${\alpha }_t$ satisfy Robbins-Munro sequence.

Expected Sarsa

$$\begin{array}{rl}Q(S_t,A_t)& \leftarrow Q(S_t,A_t)+\alpha [R_{t+1}+\gamma {\mathbb{E}}_{\pi }[Q(S_{t+1},A_{t+1})\mid S_{t+1}]-Q(S_t,A_t)]\\ & \leftarrow Q(S_t,A_t)+\alpha [R_{t+1}+\gamma \sum _a\pi (a\mid S_{t+1})Q(S_{t+1},a)-Q(S_t,A_t)]\end{array}$$

Expected Sarsa eliminates the variance due to the random selection of $A_{t+1}$, thus outperforms Sarsa by and large.

It can safely set $\alpha =1$ without suffering any degradation of asymptotic performance.

If $\pi$ is a greedy policy, expected sarsa is exactly Q-learning.

In a nutshell, expected Sarsa subsumes and generalizes Q-learning while reliably improving over Sarsa.

Maximization Bias

Consider single-state MDP $(|S|=1)$ with 2 actions, and both actions have 0-mean random rewards, ie. $\mathbb{E}(r\mid a=a_1)=\mathbb{E}(r\mid a=a_2)=0$.

Then $Q(s,a_1)=Q(s,a_2)=0=V(s)$ for any policy.

However, the esimate can be biased

$${\hat{V}}^{\hat{\pi }}(s)=\mathbb{E}[max\{\hat{Q}(s,a_1),\hat{Q}(s,a_2)\}]>max\{\mathbb{E}\left[\hat{Q}(s,a_1)\right],\left[\hat{Q}(s,a_2)\right]\}=max[0,0]=V^{\pi }$$

The greedy policy w.r.t. estimated $Q$ values can yield a maximization bias during finite-sample learning.

Double Q-learning