贝尔曼公式

Calculating return

Direct calculate
Bootstrapping (returns rely on each other)

Bellman equation

Calculate returns in bootstrapping
Matrix-vector form is expected

State value

A_{t} (S_{t}) = R_{t + 1}, S_{t + 1}, A_{t + 1} (S_{t + 1}) = R_{t + 2}, S_{t + 2}, \dots

$A_t (S_t)=R_{t+1}, S_{t+1} \quad , \quad A_{t+1} (S_{t+1})=R_{t+2}, S_{t+2} \quad , \quad \dots$

G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + \dots

$G_t = R_{t+1}+\gamma R_{t+2} + \gamma ^2 R_{t+3} + \dots$

v_{π} (s) = E [G_{t} | S_{t} = s], can also be expressed as v (s, π)

$v_\pi (s) = \mathbb{E}[G_t | S_t = s], \text{ can also be expressed as } v(s, \pi)$

It is a function of s .
It is based on the policy \pi .

Deriving the Bellman equation

G_{t} = R_{t + 1} + γ G_{t + 1}, \begin{aligned} (1) & v_{π} (s) & = E [G_{t} | S_{t} = s] \\ (2) & = E [R_{t + 1} | S_{t} = s] + γ E [G_{t + 1} | S_{t} = s] \end{aligned}

$G_t = R_{t+1}+\gamma G_{t+1}, \\ \\ \begin{align} v_\pi (s) &= \mathbb{E}[G_t | S_t = s] \\ &=\mathbb{E}[R_{t+1} | S_t = s]+ \gamma \mathbb{E}[G_{t+1} | S_t = s] \end{align}$

\begin{aligned} (3) & E [R_{t + 1} | S_{t} = s] & = Σ_{a} π (a | s) E [R_{t + 1} | S_{t} = s, A_{t} = a] \\ (4) & = Σ_{a} π (a | s) Σ_{r} p (r | s, a) r \end{aligned}

$\begin{align} \mathbb{E}[R_{t+1} | S_t = s] &= \Sigma_a \pi (a|s) \mathbb{E}[R_{t+1} | S_t=s, A_t=a] \\ &= \Sigma_a \pi (a|s) \Sigma_r p(r|s,a)r \end{align}$

\begin{aligned} (5) & E [G_{t + 1} | S_{t} = s] & = Σ_{s^{'}} E [G_{t + 1} | S_{t + 1} = s^{'}] p (s^{'} | s) \\ (6) & = Σ_{s^{'}} v_{π} (s^{'}) p (s^{'} | s) \\ (7) & = Σ_{s^{'}} v_{π} (s^{'}) Σ_{a} p (s^{'} | s, a) π (a | s) \end{aligned}

$\begin{align} \mathbb{E}[G_{t+1} | S_t = s] &= \Sigma_{s'}\mathbb{E}[G_{t+1}|S_{t+1}=s']p(s'|s) \\ &= \Sigma_{s'}v_\pi (s')p(s'|s) \\ &= \Sigma_{s'}v_\pi (s') \Sigma_a p(s'|s,a) \pi (a|s) \end{align}$

Therefore, we have

v_{π} (s) = Σ_{a} π (a | s) [Σ_{r} p (r | s, a) r + γ Σ_{s^{'}} p (s^{'} | s, a) v_{π} (s^{'})], \forall s \in S .

$v_\pi (s) = \Sigma_a \pi (a|s) [\Sigma_r p(r|s,a)r+\gamma \Sigma_{s'}p(s'|s,a)v_\pi(s')], \quad \forall s \in S.$

贝尔曼公式使用n个式子求解，一般只写上述这一个式子
贝尔曼公式依赖于policy
后面几个章节都默认已知dynamic model（environment model），对于未知的model会在未来学习

Matrix-vector form of the Bellman equation

Rewrite the Bellman equation as

v_{π} (s) = r_{π} (s) + γ Σ_{s^{'}} p_{π} (s^{'} | s) v_{π} (s^{'})

$v_\pi (s) = r_\pi (s) + \gamma \Sigma_{s'} p_\pi (s'|s)v_\pi (s')$

Add the index

v_{π} (s_{i}) = r_{π} (s_{i}) + γ Σ_{s_{j}} p_{π} (s_{j} | s_{i}) v_{π} (s_{j})

$v_\pi (s_i) = r_\pi (s_i) + \gamma \Sigma_{s_j} p_\pi (s_j|s_i)v_\pi (s_j)$

\underset{v_{π}}{\underset{⏟}{[\begin{matrix} v_{π} (s_{1}) \\ v_{π} (s_{2}) \\ v_{π} (s_{3}) \\ v_{π} (s_{4}) \end{matrix}]}} = \underset{r_{π}}{\underset{⏟}{[\begin{matrix} r_{π} (s_{1}) \\ r_{π} (s_{2}) \\ r_{π} (s_{3}) \\ r_{π} (s_{4}) \end{matrix}]}} + γ \underset{P_{π}}{\underset{⏟}{[\begin{matrix} p_{π} (s_{1} | s_{1}) & p_{π} (s_{2} | s_{1}) & p_{π} (s_{3} | s_{1}) & p_{π} (s_{4} | s_{1}) \\ p_{π} (s_{1} | s_{2}) & p_{π} (s_{2} | s_{2}) & p_{π} (s_{3} | s_{2}) & p_{π} (s_{4} | s_{2}) \\ p_{π} (s_{1} | s_{3}) & p_{π} (s_{2} | s_{3}) & p_{π} (s_{3} | s_{3}) & p_{π} (s_{4} | s_{3}) \\ p_{π} (s_{1} | s_{4}) & p_{π} (s_{2} | s_{4}) & p_{π} (s_{3} | s_{4}) & p_{π} (s_{4} | s_{4}) \end{matrix}]}} \underset{v_{π}}{\underset{⏟}{[\begin{matrix} v_{π} (s_{1}) \\ v_{π} (s_{2}) \\ v_{π} (s_{3}) \\ v_{π} (s_{4}) \end{matrix}]}}

$\underbrace{ \left [ \begin{matrix} v_\pi (s_1) \\ v_\pi (s_2) \\ v_\pi (s_3) \\ v_\pi (s_4) \end{matrix} \right ] }_{v_\pi} = \underbrace{ \left [ \begin{matrix} r_\pi (s_1) \\ r_\pi (s_2) \\ r_\pi (s_3) \\ r_\pi (s_4) \end{matrix} \right ] }_{r_\pi} + \gamma \underbrace{ \left [ \begin{matrix} p_\pi (s_1|s_1) & p_\pi (s_2|s_1) & p_\pi (s_3|s_1) & p_\pi (s_4|s_1) \\ p_\pi (s_1|s_2) & p_\pi (s_2|s_2) & p_\pi (s_3|s_2) & p_\pi (s_4|s_2) \\ p_\pi (s_1|s_3) & p_\pi (s_2|s_3) & p_\pi (s_3|s_3) & p_\pi (s_4|s_3) \\ p_\pi (s_1|s_4) & p_\pi (s_2|s_4) & p_\pi (s_3|s_4) & p_\pi (s_4|s_4) \end{matrix} \right ] }_{P_\pi} \underbrace{ \left [ \begin{matrix} v_\pi (s_1) \\ v_\pi (s_2) \\ v_\pi (s_3) \\ v_\pi (s_4) \end{matrix} \right ] }_{v_\pi}$

Closed-form solution

v_{π} = (I - γ P_{π})^{- 1} r_{π}

$v_\pi = (I-\gamma P_\pi)^{-1} r_\pi$

iterative solution

v_{k + 1} = r_{π} + γ P_{π} v_{k}, v_{k} \to v_{π} = (I - γ P_{π})^{- 1} r_{π}, k \to \infty

$v_{k+1}=r_\pi +\gamma P_\pi v_k, \\ v_k \to v_\pi = (I-\gamma P_\pi)^{-1} r_\pi, \quad k \to \infin$

证明方法：定义 v_k 和 v_\pi 的差

Action value

q_{π} (s, a) = E [G_{t} | S_{t} = s, A_{t} = a]

$q_\pi (s,a)=\mathbb{E}[G_t|S_t=s,A_t=a]$

在s状态进行a动作

a function of the state-action pair (s, a)
depends on \pi

Relation to state value

\underset{v_{π} (s)}{\underset{⏟}{E [G_{t} | S_{t} = s]}} = Σ_{a} \underset{q_{π} (s, a)}{\underset{⏟}{E [G_{t} | S_{t} = s, A_{t} = a]}} π (a | s)

$\underbrace{\mathbb{E}[G_t|S_t=s]}_{v_\pi (s)} = \Sigma_a \underbrace{\mathbb{E}[G_t|S_t = s, A_t = a]}_{q_\pi (s,a)} \pi (a|s)$

Compare state value (v_ \pi) and action value(q_ \pi) , we have

q_{π} (s, a) = Σ_{r} p (r | s, a) r + γ Σ_{s^{'}} p (s^{'} | s, a) v_{π} (s^{'})

$q_\pi (s,a)=\Sigma_r p(r|s,a)r+\gamma \Sigma_{s'}p(s'|s,a)v_\pi(s')$

posted on 2023-06-10 11:00 POLAYOR 阅读(70) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· LinearRegression_1

· 学校Java Week6

· 强化学习的数学原理-02贝尔曼公式

· 【强化学习的数学原理】课程笔记（二）——贝尔曼公式

· 强化学习理论-第2课-贝尔曼公式

阅读排行：
· 震惊！C++程序真的从main开始吗？99%的程序员都答错了
· winform 绘制太阳，地球，月球运作规律
· 【硬核科普】Trae如何「偷看」你的代码？零基础破解AI编程运行原理
· 上周热点回顾（3.3-3.9）
· 超详细：普通电脑也行Windows部署deepseek R1训练数据并当服务器共享给他人

POLAYOR