RL 基础 | Value Iteration 的收敛性证明

（其实是专业课作业🤣 感觉算法岗面试可能会问，来存一下档）

目录

• 问题：证明 Value Iteration 收敛性
• 0 Definitions - 定义
• 1 Bellman operator is a Contraction Mapping - 贝尔曼算子是压缩映射
• 2 Contraction Mapping converges to the optimal value function - 压缩映射能收敛到最优值函数
• Reference - 参考资料

相关链接：RL 基础 | Policy Iteration 的收敛性证明

---

问题：证明 Value Iteration 收敛性

Please prove the convergence of Value Iteration algorithm in discounted reward setting, and analyze the performance between resulting policy and optimal policy. 证明 Value Iteration 的收敛性。

背景：

• Policy Iteration：先得到一个 policy，然后算出它的 value function，基于该 value function 取 max_a Q(s,a) 作为新 policy，再算新 policy 的 value function…
• Value Iteration：不停对 value function 使用贝尔曼算子，新 V = max_a [r(s,a) + γV(s')]。

0 Definitions - 定义

First, we define

• Discrete state space $S=\{s_1,\cdots ,s_n\}$, discrete action space $A=\{a_1,\cdots ,a_n\}$.
• Value function: $V(s)=E_{a\sim \pi (a|s)}[r(s,a)+\gamma E_{s^{\prime }\sim p(s^{\prime }|s,a)}V(s^{\prime })]$ .
• Bellman operator: for all $s\in S$ ,

$$\begin{array}{rl}{B}_{\pi }V(s)& \triangleq E_{a\sim \pi (a|s)}[r(s,a)+\gamma E_{s^{\prime }\sim p(s^{\prime }|s,a)}V(s^{\prime })]\\ B_{\ast }V(s)& \triangleq \underset{a}{max}[r(s,a)+\gamma E_{s^{\prime }\sim p(s^{\prime }|s,a)}V(s^{\prime })]\end{array}$$

The Value Iteration algorithm is to conduct Bellman operator $B_{\ast }$ on the value function $V(s)$. We want to prove that the sequence will converge to the optimal value function $V^{\ast }$, which should be the of the Bellman equation $B_{\ast }V=V$ .

我们定义了状态空间、动作空间、值函数（value function）和贝尔曼算子（Bellman operator）。VI 算法，就是对 value function 一直使用 Bellman operator，希望最后能收敛到最优的 value function。这样，该最优 value function 必然是 Bellman equation $B_{\ast }V=V$ 的不动点。

1 Bellman operator is a Contraction Mapping - 贝尔曼算子是压缩映射

Then, we need to prove that Bellman Operator is a Contraction Mapping. The definition of Contraction Mapping is as follows:

$$d(f(x_1),f(x_2))\le \gamma d(x_1,x_2),$$

where $d$ is a measurement over a metric space, $\gamma \in (0,1)$, and $f$ is the Contraction Mapping. In our case, the definition of Contraction Mapping is as follows:

$$|f(x_1)-f(x_2){|}_{\mathrm{\infty }}\le \gamma |x_1-x_2{|}_{\mathrm{\infty }},$$

where $|{|}_{\mathrm{\infty }}$ is the infinite norm (return the max value over all dimensions of a vector).

接下来，证明贝尔曼算子是压缩映射（Contraction Mapping）。在这里，压缩映射的定义是，两个变量的无穷范数（向量各维之差的最大绝对值）会被 $f$ 映射压缩 $\gamma$ 倍。

Let's prove that Bellman Operator $B_{\ast }$ is a Contraction Mapping. The proof is as follows.

$$\begin{array}{rl}& |B_{\ast }{V}_1(s)-B_{\ast }{V}_2(s)|\\ & =|\underset{a}{max}[r(s,a)+\gamma E_{s^{\prime }p(s^{\prime }|s,a)}{V}_1(s^{\prime })]-\underset{a}{max}[r(s,a)+\gamma E_s{V}_2(s^{\prime })]|\\ & \le |\underset{a}{max}[r(s,a)+\gamma E_{s^{\prime }}{V}_1(s^{\prime })-r(s,a)-\gamma E_{s^{\prime }}{V}_2(s^{\prime })]|\\ & =|\gamma \underset{a}{max}{E}_{s^{\prime }}(V_1(s^{\prime })-V_2(s^{\prime }))|\\ & \le \gamma \underset{a}{max}|E_{s^{\prime }}(V_1(s^{\prime })-V_2(s^{\prime }))|\\ & =\gamma \underset{a}{max}|\sum _{s^{\prime }}p(s^{\prime }|s,a)[V_1(s^{\prime })-V_2(s^{\prime })]|\\ & \le \gamma \underset{a}{max}\sum _{s^{\prime }}p(s^{\prime }|s,a)|V_1(s^{\prime })-V_2(s^{\prime })|\\ & =\gamma \sum _{s^{\prime }}p(s^{\prime }|s,a_{\ast }(s))|V_1(s^{\prime })-V_2(s^{\prime })|\\ & \le \gamma \sum _{s^{\prime }}p(s^{\prime }|s,a_{\ast }(s))\underset{s}{max}|V_1(s)-V_2(s)|\\ & =\gamma |V_1-V_2{|}_{\mathrm{\infty }}\end{array}$$

贝尔曼算子是压缩映射的证明：一连串放缩，如上。

2 Contraction Mapping converges to the optimal value function - 压缩映射能收敛到最优值函数

If Bellman Operator is a Contraction Mapping, then we can use the Banach fixed point theorem to prove that, the optimal value function $V^{\ast }$ is the unique fixed point of the Bellman Operator $B_{\ast }$, and the sequence $B_{\ast }(B_{\ast }(\cdots B_{\ast }V))$ converges to $V^{\ast }$.

如果贝尔曼算子是压缩映射，则可以用巴拿赫不动点定理（Banach fixed point theorem），证明 $V^{\ast }$ 是 Bellman Equation $B_{\ast }V=V$ 的唯一不动点，并且一定会收敛到该不动点。

The first step is to prove that the fixed point $V^{\ast }$ is unique. Use proof by contradiction. Suppose there are two converged value $V_1^{\ast },V_2^{\ast }$, then we have $|B_{\ast }{V}_1^{\ast }-B_{\ast }{V}_2^{\ast }{|}_{\mathrm{\infty }}=|V_1^{\ast }-V_2^{\ast }{|}_{\mathrm{\infty }}$. However, Bellman operator $B_{\ast }$ is a Contraction Mapping, so there is $|B_{\ast }{V}_1^{\ast }-B_{\ast }{V}_2^{\ast }{|}_{\mathrm{\infty }}\le \gamma |V_1^{\ast }-V_2^{\ast }{|}_{\mathrm{\infty }}$, which contradicts the previous one. Thus, there can be only one converged value, and it is obviously the fixed point $V^{\ast }$.

第一步，证明收敛到的不动点是唯一的。使用反证法，假设有两个不动点，那么它们俩的压缩映射都映射到原地（不动点嘛），距离没有 γ 倍压缩，矛盾，从而得证。

The second step is to prove that the fixed point $V^{\ast }$ really exists. If we can prove that $\{V,B_{\ast }V,B_{\ast }^{2}V,\cdots \}$ is a Cauchy sequence, the fixed point $V^{\ast }$ exists. Take two value $B_{\ast }^a,B_{\ast }^b$ from the sequence, where $a\gg b$. Use the triangle inequality of $|{|}_{\mathrm{\infty }}$ as a measurement, we can get

$$\begin{array}{rl}|B_{\ast }^a-B_{\ast }^b{|}_{\mathrm{\infty }}& \le |B_{\ast }^a-B_{\ast }^{a-1}{|}_{\mathrm{\infty }}+|B_{\ast }^{a-1}-B_{\ast }^b{|}_{\mathrm{\infty }}\\ & \le |B_{\ast }^a-B_{\ast }^{a-1}{|}_{\mathrm{\infty }}+|B_{\ast }^{a-1}-B_{\ast }^{a-2}{|}_{\mathrm{\infty }}+|B_{\ast }^{a-2}-B_{\ast }^b{|}_{\mathrm{\infty }}\\ & \le |B_{\ast }^a-B_{\ast }^{a-1}{|}_{\mathrm{\infty }}+\cdots +|B_{\ast }^{b+1}-B_{\ast }^b{|}_{\mathrm{\infty }}\\ & \le {\gamma }^b|B_{\ast }V-V|\sum _{i=0}^{a-b-1}{\gamma }^i\le \frac{{\gamma }^b}{1-\gamma }|B_{\ast }V-V|\end{array}.$$

So, if we can choose a large enough $b$, the value of $|B_{\ast }^a-B_{\ast }^b{|}_{\mathrm{\infty }}$ can be less than any $ϵ>0$. Thus, there must exists an $N\in \mathbb{N}$, such that $|B_{\ast }^a-B_{\ast }^b{|}_{\mathrm{\infty }}<ϵ$, for all $a,b>N$ . The sequence is a Cauchy sequence, and the fixed point $V^{\ast }$ exists.

第二步，证明不动点 $V^{\ast }$ 存在。去证 $\{V,B_{\ast }V,B_{\ast }^{2}V,\cdots \}$ 是柯西序列（Cauchy sequence），如果是柯西序列，那么就存在收敛的不动点。柯西序列的定义是，对于序列里的 $x_a,x_b$，存在正整数 N 使得对于任意 $a,b>N$，度量 $d(x_a,x_b)<ϵ$，其中 $ϵ$ 是任意小的正实数。先使用三角不等式，将 $x_a-x_b$ 的形式拆成 $(x_a-x_{a-1})+(x_{a-1}-x_{a-2})+\cdots +(x_{b+1}-x_b)$，然后使用压缩映射的性质，放缩成 γ 的等比序列求和，即可得到，它小于任意的小实数。

Reference - 参考资料

• A blog from Zhihu: https://zhuanlan.zhihu.com/p/419208786
• 非常感谢！很清晰的博客 😊🌹