Reinforcement Learning Cheatsheet

1. MDPs

What is MDPs?

MDPs are a classical formalization of sequential decision making, where actions influence not just immediate rewards, but also subsequent situations, or states, and through those future rewards.

The dynamics of MDP is define as:

\[p(s',r|s,a) = Pr\{S_t=s',R_t=r|S_{t-1}=s,A_{t-1}=a\} \]

What is Markov property?

The state must include information about all aspects of the past agent–environment interaction that make a difference for the future. If it does, then the state is said to have the Markov property

What is finite MDP?

In a finite MDP, the sets of states, actions, and rewards (S, A, and R) all have a finite number of elements. In this case, the random variables \(R_t\) and \(S_t\) have well defined discrete probability distributions dependent only on the preceding state and action.

What can one do if the state or action is continuous?

To quantize it.

What makes MDPs different from k-bandit algo?

Whereas in bandit problems we estimated the value \(q_*(a)\) of each action \(a\), in MDPs we estimate the value \(q_*(s,a)\) of each action \(a\) in each state \(s\), or we estimate the value \(v_*(s)\) of each state given optimal action selections.

Policy function

A policy (e.g. \(\pi(a|s)\)) is a mapping from states to probabilities of selecting each possible action.

Value function

State-value function

State-value function, denoted \(v_{pi}\), namely the value function of a state \(s\) under a policy \(pi\), means the expected return an agent can get following policy \(\pi\) starting from state \(s\).

\[v_{\pi}(s) = \mathrm{E_\pi} \sum_{k=0} ^{\infty} {\gamma^{k} R_{t+k+1} | S_t = s}, for \, \forall s \in S \]

Action-value function

$ q_{\pi}(s, a) $, called action-value function for policy \(\pi\).
Similar to the definition of value function, action-value function means the expected return an agent can get following policy \(\pi\) starting from state-action pair \((s, a)\) at time t.

\[q_{\pi}(s, a) = \mathrm{E_\pi} \sum_{k=0} ^{\infty} {\gamma^{k} R_{t+k+1} | S_t = s, A_t = a}, for \, \forall s \in S and \, \forall a \in A \]

2. Dynamic Programing

DP refers to a collection of algos that can be used to compute the value functions, thus to find optimal policies, given a perfect model of the model of the environment as an MDP.