机器学习第3章: 泛化

合集 - 机器学习(6)

1.机器学习第1章: 概述2024-01-222.机器学习第2章: 优化2024-01-19

3.机器学习第3章: 泛化2024-01-22

4.机器学习第4章: 监督学习2024-01-225.机器学习第5章: 无监督学习2024-01-226.机器学习第6章: 其他2024-01-22

收起

Chapter 3: Generalization Theory

Acknowledgment: Most of the knowledge comes from Yuan Yang's course "Machine Learning".

泛化理论想解决一个什么样的问题呢？

已知$L_{train}=ϵ$, what can we say on $L_D$ (population loss)?

1. The traditional way is sampling from D again to get a test set and then get $L_{test}$.
2. We can use theory to get generalization bounds.

The latter is what generalization theory wants to do.

No free lunch theorem

It says that if we don’t have any prior knowledge of the data distribution, we can't design an algorithm that always works well. In other words, there is no universal learner. Well, based on this theorem, even human brains only work in certain scenarios. But this theorem considers a really weird setting(没有先验的学习label), which may not exist in practice.

Theorem 5.1 (no-free-lunch) Let $A$ be any learning algorithm for the task of binary classification with respect to a loss within $[0,1]$ over a domain $X$. Let $m$ be any number smaller than $|X|/2$, representing a training set size. Then, there exists a distribution $D$ over s.t.

1. There exists a function $f:X\to 0,1$ with $L_D(f)=0$
2. With probability of at least $\frac{1}{7}$ over the choice of $S\sim D^m$ (that is $S$ is training set size $m$) we have that $L_D(A(S))\ge \frac{1}{8}$

That is

$$P_{S\sim {\mathcal{D}}^m}(L_{\mathcal{D}}(A(S))\ge \frac{1}{8})\ge \frac{1}{7}$$

首先我们需要明确一下这些符号的意思:

$X$ 是原始数据的集合。 $X\times \{0,1\}$是一个二分类的数据集空间，是数据集和标签集的直积。$D$ 是对$X\times \{0,1\}$这个数据集空间进行有概率地采样。也就是一个带标签的数据集。只有一些数据集是合法的，由条件1来控制。也就是至少要有一个$f$在这个数据集上能做到满分。

证明前我们先考虑这个条件，假设全集$X$大小为$2m$，那么就有$T=2^{2m}$ 种合法的函数(没有限制)能够把$X$ 映射到 $\{0,1\}$ 这个二分类标签集。这些函数记作$\{f_1,...,f_T\}$, 由这些函数定义的$D$ 记作$\{D_1,...,D_T\}$：

$$D_i(x_j,f_i(x_j))=\frac{1}{2m},\mathrm{\forall }{x}_j\in X$$

也就是$D_i$ 里面有$2m$条$f_i$标注的数据，每条等概率。

这样的$T$ 个$D$上都满足条件(1)，也就是存在一个全对的函数。

接下来的证明该怎么想呢？根据我们改写的这个概率形式，我们就会很自然的想到markov's inequality:

$$Pr(x>a)\le \frac{E|x|}{a}$$

加上一个我们比较直观的intuition，就是一半的题目我见过我会做，一半的题目我只能蒙，二分类有一半概率蒙对，所以正确率是3/4

$$\begin{array}{rl}{E}_{S\sim D^m}[1-L_D(A(S))]& \le \frac{3}{4}\\ Pr(1-L_D(A(S))>\frac{7}{8})& \le \frac{E_{S\sim D^m}[1-L_D(A(S))]}{7/8}\le \frac{6}{7}\\ Pr(L_D(A(S))<\frac{1}{8})& \le \frac{6}{7}\\ Pr(L_D(A(S))>\frac{1}{8})& \ge \frac{1}{7}\end{array}$$

接下来的关键就在于证明$E_{S\sim D^m}[1-L_D(A(S))]\le \frac{3}{4}$:

对于任意一个$D_i$, 我都可以在其中取出$k=(2m{)}^m$次方个不同的训练集，记作$\{S_1,...,S_k\}$. 为了表示这个是从$D_i$中取出的，或者说是由$f_i$ 标注的，加上上标$i$, 记作$\{S_1^i,...,S_k^i\}$

$$\begin{array}{rl}\underset{i\in [T]}{max}{\mathbb{E}}_{S\sim {\mathcal{D}}_i^m}(L_{{\mathcal{D}}_i}(A(S)))& =\underset{i\in [T]}{max}\frac{1}{k}\sum _{j=1}^k{L}_{{\mathcal{D}}_i}(A(S_j^i))\\ & \ge \frac{1}{T}\sum _{i=1}^T\frac{1}{k}\sum _{j=1}^k{L}_{{\mathcal{D}}_i}(A(S_j^i))\\ & \ge \underset{j\in [k]}{min}\frac{1}{T}\sum _{i=1}^T{L}_{{\mathcal{D}}_i}(A(S_j^i))\text{max>均值>min}\end{array}$$

原本是对于不同的训练集求均值，现在是对于不同的标注方式求均值，事情就变得了简单了起来，因为训练元素固定，我们现在可以轻松的去定义，没有出现在训练集中的元素。定义这些元素为 $\{v_1,...,v_p\}$ 是没有出现在训练集中的元素. 因为有可能训练集多次取到了相同的元素，所以总数小于等于m, $p\ge m$:

$$\begin{array}{rl}{L}_{{\mathcal{D}}_i}(h)& =\frac{1}{2m}\sum _{x\in C}[h(x)\ne f_i(x)]\\ & \ge \frac{1}{2m}\sum _{r=1}^p\mathbb{1}[h(v_r)\ne f_i(v_r)]\\ & \ge \frac{1}{2p}\sum _{r=1}^p\mathbb{1}[h(v_r)\ne f_i(v_r)]\end{array}$$

So对所有$D_i$ 求均值我们能得到:

$$\begin{array}{rl}\frac{1}{T}\sum _{i=1}^T{L}_{{\mathcal{D}}_i}(A(S_j^i))& \ge \frac{1}{T}\sum _{i=1}^T\frac{1}{2p}\sum _{r=1}^p\mathbb{1}[(A(S_j^i))(v_r)\ne f_i(v_r)]\\ & =\frac{1}{2}\frac{1}{p}\sum _{r=1}^p\frac{1}{T}\sum _{i=1}^T\mathbb{1}[(A(S_j^i))(v_r)\ne f_i(v_r)]\\ & \ge \frac{1}{2}\underset{r\in [p]}{min}\frac{1}{T}\sum _{i=1}^T\mathbb{1}[(A(S_j^i))(v_r)\ne f_i(v_r)]\end{array}$$

不管你学到的$(A(S_j^i))(v_r)$是什么, 遍历所有 $f_i(v_r)$, 肯定是一半概率错一半概率对，所以$\mathrm{\forall }r$ 我们有:

$$\frac{1}{T}\sum _{i=1}^T\mathbb{1}[(A(S_j^i))(v_r)\ne f_i(v_r)]=\frac{1}{2}$$

甚至都不用取$min$就可以得到:

$$\begin{array}{rl}\underset{i\in [T]}{max}{\mathbb{E}}_{S\sim {\mathcal{D}}_i^m}(L_{{\mathcal{D}}_i}(A(S)))& \ge \frac{1}{T}\sum _{i=1}^T{L}_{{\mathcal{D}}_i}(A(S_j^i))\\ & \ge \frac{1}{2}\frac{1}{T}\sum _{i=1}^T\mathbb{1}[(A(S_j^i))(v_r)\ne f_i(v_r)]=\frac{1}{4}\end{array}$$

Finite hypothesis class and ERM

Class of H, the class of functions can be finite/infinite.

$ER{M}_H$: pick the hypothesis with the smallest training loss

$$ER{M}_H(S)\in \arg \underset{h\in H}{min}{L}_S(h)$$

If H is a infinite class, $ER{M}_H$ may not give the best solution because of Overfitting (memorization).

Realizability assumption

To show this $ERM$ algorithm is powerful, we raise the Realizability assumption:

There exists $h^{\ast }\in H$ s.t. $L_{D,f}(h^{\ast })=0$.

Note that this assumption implies that with probability 1 over random samples, $S$, where the instances of $S$ are sampled according to $D$ and are labeled by $f$, we have $L_S(h^{\ast })=0$

Later we will see that it is not always possible to pick a function that perfectly solves the problem.

corollary 2.3.

It means, that if the problem is realizable, then the ERM hypothesis is good enough. 也就是达到的采样数量，ERM学到的函数就足够好。

Let $H$ be a finite hypothesis class. Let $\delta \in \{0,1\}$and $ϵ>0$ and let $m$ be an integer that satisfies $m\ge \log (\frac{|H|}{\delta })/ϵ$. Then, for any labeling function $f$ and for any distribution $D$, for which the realizability assumption holds (that is, for some $h\in H$, $L_{D,f}(h)=0$), with probability of at least $1-\delta$ over the choice of an iid sample $S$ of size $m$, we have them for every ERM hypothesis, $h_S$, it holds that:

$$L_{D,f}(h_S)\le ϵ$$

Proof:

We would like to bound the probability of the event $L_{D,f}(h_S)>ϵ$

Let $H_B$ be the bad hypothesis, that is:

$$H_B=\{h\in H:L_{D,f}(h)>ϵ\}$$

But, since the realizability assumption implies that $L_S(h_S)=0$, 所以ERM取到的bad hypothesis 也要在S上拿到满分。

Let $M$ be the training set that can train a bad hypothesis:

$$M=\{S:\text{exist}h\in H_B,L_S(h)=0\}$$

只有取到$M$ 中的$S$, 才有可能训出这样的bad hypothesis

$$\begin{array}{rl}{D}^m(\{S:L_{D,f}(h_S)& >ϵ\})\le D^m(M)\\ & =D^m({\cup }_{h\in H_B}\{S:L_S(h)=0\})\text{(定义)}\\ & \le \sum _{h\in H_B}{D}^m(\{S:L_S(h)=0\})\text{(union bound)}\end{array}$$

第一个不等式是因为，如果这个S上训出来的$h_S$不好，那起码这要是一个能训出来bad hypothesis的S，小于号因为是必要条件。

好现在对于一个fix的$h\in H_B$, 我们来选$S$ 中的元素。$D$中有大于$ϵ$的元素是$h$会预测错的，所以不能选。只能选剩下的的那$1-ϵ$

$$D(\{x_i:h(x_i)=y_i\})=1-L_{D,f}(h)\le 1-ϵ$$

$$\begin{array}{rl}{D}^m(\{S:h_S(h)=0\})& =D^m(\{\mathrm{\forall }{x}_i\in S:h(x_i)=y_i\})(\text{选中一些他会的例子})\\ & =\prod _{i=1}^{m}D(\{x_i:h(x_i)=y_i\})(\text{独立事件})\\ & \le (1-ϵ{)}^m\le e^{-m}\end{array}$$

最后:

$$\sum _{h\in H_B}{D}^m(\{S:h_S(h)=0\})\le |H_B|e^{-m}\le |H|e^{-m}(\text{这也就是为什么要finite})$$

要让$|H|e^{-m}\le \delta$, 解得：

$$m\ge \frac{\log (\frac{|H|}{\delta })}{ϵ}$$

PAC learnable

It’s one kind of learnability. Essentially it tells us whether it is possible to learn a hypothesis class $H$

The lower bound function $m_H$, tells you how many samples you need, and then if you have this many samples, you can find some algorithm to train on these samples, and the outcome, is good on the population distribution.

definition

A hypothesis class $H$ is PAC learnable if there exist a function $m_H(ϵ,\delta )$ and a learning algorithm(not essentially ERM) with the following property:

For every $ϵ,\delta \in (0,1)$, for every distribution $D$ over $X$, and for every labeling function $f:X\to \{0,1\}$, if the realizable assumption holds with respect to $H,D,f$, then when running the learning algorithm on $m\ge m_H(ϵ,\delta )$ iid examples generated by $D$ and labeled by $f$, the algorithm returns a hypothesis $h$ such that, with probability of at least $1-\delta$ (over the choice of the examples):

$$L_{D,f}(h)\le ϵ$$

Finite H is PAC learnable by ERM.

Agnostic PAC learnable

But don’t forget you need a realizable assumption to hold. Realizable is too strong!

The Bayes optimal predictor:

$$f(x)=\{\begin{array}{rl}& 1P(y=1|x)\ge \frac{1}{2}\\ & 0\text{otherwise }\end{array}$$

Given any probability distribution, Bayes optimal predictor is the best label predicting function. No other classifier has a lower error.

So realizable assumption fails to hold if the distribution is noisy or weird. If it does not hold, it might be hopeless to have $L_{D,f}(h)\le ϵ$. But that’s fine, we may have the adjusted version of PAC learning.

It’s more general than PAC learnable. 引入了一个数据集本身的的一个loss，就是最高分也达不到满分。

Definition of Agnostic PAC learnable:

A hypothesis class $H$ is Agnostic PAC learnable if there exist a function $m_H(ϵ,\delta )$ and a learning algorithm(not essentially ERM) with the following property:

For every $ϵ,\delta \in (0,1)$, for every distribution $D$ over $X$, then when running the learning algorithm on $m\ge m_H(ϵ,\delta )$ iid examples generated by $D$ and labeled by $f$, the algorithm returns a hypothesis $h$ such that, with probability of at least $1-\delta$ (over the choice of the examples):

$$L_D(h)\le \underset{h^{\star }\in H}{min}{L}_D(h^{\ast })+ϵ$$

It is more general than PAC learnable.

VC dimension

Is an infinite hypothesis class $H$ learnable? The answer can be yes. But how can we distinguish the infinite learnable $H$ and infinite unlearnable one? We need to introduce a VC dimension to help us.

Restriction of $H$ to $C$

Let $H$ be a class of functions from $X$ to $\{0,1\}$, and let $C=\{c_1,\dots ,c_m\}\subset X$. The restriction of $H$ to $C$ is the set of functions from $C$ to $\{0,1\}$ that can be derived from $H$. That is,

$$H_C=\{h_{c_1},\dots ,h_{c_m}:h\in H\}$$

where we represent each function from $C$ to $\{0,1\}$ as a vector in $\{0,1{\}}^{|C|}$.

Shattering
A hypothesis class $H$ shatters a finite set $C\subset X$ if the restriction of $H$ to $C$ is the set of all functions from $C$ to $\{0,1\}$. That is $|H_C|=2^{|C|}$

Corollary 6.4. like no free lunch theorem but about shattering

Corollary 6.4 tells us that if H shatters some set $C$ of size $2m$ then we cannot learn $H$ using m examples. Intuitively, if a set $C$ is shattered by $H$, we only contain half of the instances, the labels of these instances give us no information about the labels of the rest of the instances in $C$

VC-dimension

The VC dimension of a hypothesis class $H$ is the maximal size of a set C that can be shattered by $H$. If $H$ can shatter sets of arbitrarily large size we say that H has infinite VC-dimension.

Theorem 6.6. Let $H$ be a class of infinite VC-dimension. Then, $H$ is not PAC learnable.

Theorem 6.8 (the fundamental theorem of statistical learning – quantitative version).

omit here, give a coarse range of $m_H(ϵ,\delta )$

VC dimension is one way to measure sample complexity. But it also has a few drawbacks.

One main problem of the VC dimension is that It is mainly used for classification (due to the shattering notion).

what shall we do for regression? We may use Rademacher complexity

Rademacher complexity

$ϵ$-representative sample

A training set $S$ is called $ϵ$ - representative s.t.

$$\underset{h\in H}{sup}|L_D(h)-L_S(h)|\le ϵ$$

We define the representativeness of $S$ with respect to $F$ as the largest gap between the true error of a function $f$ and its empirical error:

$$Re{p}_D(F,S)=\underset{f\in F}{sup}(L_D(f)-L_S(f))$$

Now, suppose we would like to estimate the representativeness of $S$ using the sample $S$ only. One simple idea is to split $S$ into two disjoint sets, $S=S_1\cup S_2$, refer to $S_1$ as a validation set and to $S_2$ as a training set. We can then estimate the representativeness of $S$ by:

$$\underset{f\in F}{sup}(L_{S_1}(f)-L_{S_2}(f))$$

为了使这个式子变得更简单，我们于是引入的rademacher random variable:
$({\sigma }_1,{\sigma }_2,...,{\sigma }_m)\in \{-1,+1{\}}^m$ to be a vector s.t. $S_1=\{z_i=(x_i,y_i):{\sigma }_i=1\}$ and $S_2=\{z_i=(x_i,y_i):{\sigma }_i=-1\}$. If we assume $E[{\sigma }_i]=0$, that is $|S_1|=|S_2|$ then:

$$\underset{f\in F}{sup}(L_{S_1}(f)-L_{S_2}(f))=\frac{2}{m}\underset{f\in F}{sup}\sum _{i=1}^m{\sigma }_{i}f(z_i)$$

Rademacher complexity captures this idea by considering the expectation of the above with respect to a random choice of $\sigma$.
So Rademacher complexity of $𝐹$ with respect to $S$ is defined as follows:

$$R(F\circ S)=\frac{1}{m}{\mathbb{E}}_{\sigma }[\underset{f\in F}{sup}\sum _{i=1}^m{\sigma }_{i}f(z_i)]$$

We can see the the outcome of a hypithesis $f$ on the whole dataset as a vector $a$, each coordinate corresponding to a $f(z_i)$, then the equation becomes:

$$R(A)=\frac{1}{m}{\mathbb{E}}_{\sigma }[\underset{a\in A}{sup}\sum _{i=1}^m{\sigma }_i{a}_i]=\frac{1}{m}{\mathbb{E}}_{\sigma }[\underset{a\in A}{sup}⟨\sigma ,a⟩]$$

Lem 26.6 For any $A\subset R^m$ , scalar $c\in R$, and vector $a_0\in R^m$, we have.

$$R(\{ca+a_0:a\in A\})=|c|R(A)$$

就是将loss函数放大c倍的时候，R complexity也跟着放大这么多倍。

Lemma 26.7 Let $A$ be a subset of $R^m$ and $C=\{c_j\ge 0,\Vert c{\Vert }_1=1\}\subset R^N$ . And let $A^{\prime }=\{\sum _{j=1}^N{c}_j{a}^{(}j):c\in C,a^{(j)}\in A\subset R^m\}$. Then $R(A)=R(A^{\prime })$

(证明的关键是$\underset{c}{max}⟨c,v⟩=\underset{j}{max}{v}_j$)

Massart Lemma

Let $A=\{a_1,..,a_N\}$ be a finite set of vector in $R^m$. Define $\overline{a}=\frac{1}{N}\sum _{i=1}^N{a}_i$. Then:

$$R(A)\le \underset{a\in A}{max}\Vert a-\overline{a}\Vert \frac{\sqrt{2\log (N)}}{m}$$

Proof: 其实证明有点像Chernoff bound，先

Without loss of generality $\overline{a}=0$. Let $\lambda >0$. Let $A^{\prime }=\{\lambda a_1,...,\lambda a_n\}$. We upper bound the Rademacher complexity as follows:

$$\begin{array}{rl}mR(A^{\prime })& =E_{\sigma }[\underset{a^{\prime }}{max}⟨\sigma ,a^{\prime }⟩]\\ & =E_{\sigma }[\log (e^{\underset{a^{\prime }}{max}⟨\sigma ,a^{\prime }⟩})]\\ & =E_{\sigma }[\log (\underset{a^{\prime }}{max}{e}^{⟨\sigma ,a^{\prime }⟩})](\log \mathrm{和}e\mathrm{都}\mathrm{是}\mathrm{单}\mathrm{调}\mathrm{函}\mathrm{数})\\ & \le E_{\sigma }[\log (\sum _{a^{\prime }}{e}^{⟨\sigma ,a^{\prime }⟩})]\\ & \le \log (E_{\sigma }[\sum _{a^{\prime }}{e}^{⟨\sigma ,a^{\prime }⟩}])(\mathrm{琴}\mathrm{生}\mathrm{不}\mathrm{等}\mathrm{式})\\ & =\log (E_{\sigma }[\sum _{a^{\prime }}\prod _{i=1}^m{e}^{{\sigma }_i{a}_i^{\prime }}])\\ & =\log (\sum _{a^{\prime }}\prod _{i=1}^m{E}_{{\sigma }_i}[e^{{\sigma }_i{a}_i^{\prime }}])\end{array}$$

$$E_{{\sigma }_i}[e^{{\sigma }_i{a}_i^{\prime }}]=\frac{e^{a_i^{\prime }}+e^{-a_i^{\prime }}}{2}\le e^{a_i^{\prime 2}/2}(\mathrm{用}\mathrm{泰}\mathrm{勒}\mathrm{展}\mathrm{开}\mathrm{可}\mathrm{以}\mathrm{证})$$

$$\begin{array}{rl}mR(A^{\prime })& =\log (\sum _{a^{\prime }}\prod _{i=1}^m{e}^{a_i^{\prime 2}/2})\\ & =\log (\sum _{a^{\prime }}{e}^{\Vert a^{\prime }{\Vert }_2^2/2})\\ & \le \log (|A^{\prime }|\underset{a^{\prime }}{max}{e}^{\Vert a^{\prime }{\Vert }_2^2/2})\\ & =\log (|A|)+\underset{a^{\prime }}{max}(\frac{\Vert a^{\prime }{\Vert }_2^2}{2})\end{array}$$

$$R(A)\le \frac{\log (|A|)+\underset{a^{\prime }}{max}(\frac{\Vert a^{\prime }{\Vert }_2^2}{2})}{\lambda m}$$

通过设置$\lambda$来达到最后的那个值

Contraction Lemma

For each $i\in [m]$, let ${\varphi }_i:R\stackrel{}{\to }R$ be a $\rho$-smooth function. let $\varphi (a)$ denote $({\varphi }_1(a_1),{\varphi }_m(y_m))$. Let $\varphi \circ A=\{\varphi (a):a\in A\}$. Then $R(\varphi \circ A)\le \rho R(A)$.

proof: omit

也是跟loss函数有关，当loss函数再复合一个$\rho$-smooth function，R也跟着放大$\rho$倍

Lem26.2

$$E_{S\sim D^m}[Re{p}_D(F,S)]\le 2E_{S\sim D^m}R(F\circ S)$$

Proof: 关键就是要怎么加上R random variable

$$\begin{array}{rl}Re{p}_D(F,S)& =\underset{f\in F}{sup}(L_D(f)-L_S(f))\\ & =\underset{f\in F}{sup}(E_{S^{\prime }}[L_{S^{\prime }}(f)]-L_S(f))\\ & =\underset{f\in F}{sup}(E_{S^{\prime }}[L_{S^{\prime }}(f)-L_S(f)])\\ & \le E_{S^{\prime }}[\underset{f\in F}{sup}(L_{S^{\prime }}(f)-L_S(f))]\end{array}$$

So:

$$\begin{array}{rl}{E}_{S\sim D^m}[Re{p}_D(F,S)]& \le E_{S,S^{\prime }}[\underset{f\in F}{sup}(L_{S^{\prime }}(f)-L_S(f))]\\ & \le \frac{1}{m}{E}_{S,S^{\prime }}[\underset{f\in F}{sup}\sum _{i=1}^m(f(z_i^{\prime })-f(z_i))]\\ & =\frac{1}{m}{E}_{S,S^{\prime },\sigma }[\underset{f\in F}{sup}\sum _{i=1}^m{\sigma }_i(f(z_i^{\prime })-f(z_i))]\end{array}$$

We want to prove the last equality. Pick a specific index $j$ to do the operation:

$$\begin{array}{rl}{E}_{S,S^{\prime }\sim D^m}[\underset{f\in F}{sup}{\sigma }_j(f(z_j^{\prime })-f(z_j))]& =\frac{1}{2}{E}_{S,S^{\prime }\sim D^m}[\underset{f\in F}{sup}(f(z_j^{\prime })-f(z_j))]+\frac{1}{2}{E}_{S,S^{\prime }\sim D^m}[\underset{f\in F}{sup}(f(z_j)-f(z_j^{\prime }))]\\ & =E_{S,S^{\prime }\sim D^m}[\underset{f\in F}{sup}(f(z_j^{\prime })-f(z_j))](\mathrm{利}\mathrm{用}S\mathrm{和}{S}^{\prime }\mathrm{的}\mathrm{对}\mathrm{称}\mathrm{性})\end{array}$$

然后repeat for 所有的 $z_j$。这样我们就可以证明那个不等式。

$$\begin{array}{rl}\frac{1}{m}{E}_{S,S^{\prime },\sigma }[\underset{f\in F}{sup}\sum _{i=1}^m{\sigma }_i(f(z_i^{\prime })-f(z_i))]& \le \frac{1}{m}{E}_{S^{\prime },\sigma }[\underset{f\in F}{sup}\sum _{i=1}^m{\sigma }_{i}f(z_i^{\prime })]+\frac{1}{m}{E}_{S,\sigma }[\underset{f\in F}{sup}\sum _{i=1}^m{\sigma }_{i}f(z_i)]\\ & =E_{S^{\prime }\sim D^m}R(F\circ S^{\prime })+E_{S\sim D^m}R(F\circ S)\\ & =2E_{S\sim D^m}R(F\circ S)\end{array}$$

Thm26.3 相当于将ERM的结果代入前面那个lemma

再次证明了ERM的强大power

Thm 26.5 两个重要公式
Assume that for all $z$ and $h\in H$, we have that $|l(h,z)|\le c$, then:

1. with $1-\delta$ probability, $\mathrm{\forall }h$:
   $$L_D(h)-L_S(h)\le 2E_{S^{\prime }\sim D^m}R(l\circ H\circ S^{\prime })+c\sqrt{\frac{2\ln (\frac{2}{\delta })}{m}}$$

2. with $1-\delta$ probability, $\mathrm{\forall }h$:

$$L_D(h)-L_S(h)\le 2R(l\circ H\circ S^{\prime })+4c\sqrt{\frac{2\ln (\frac{4}{\delta })}{m}}$$

Rademacher complexity of linear classes as an example

Let $S=(x_1,x_2,...,x_m)$ be vectors in a Hilbert space. Define $H_2\circ S=\{w\circ S=(⟨w,x_1⟩,...,⟨w,x_1⟩):\Vert w{\Vert }_2\le C\}$. Then

$$R(H_2\circ S)\le \frac{\underset{i}{max}\Vert x_i{\Vert }_{2}C}{\sqrt{m}}$$

$$\begin{array}{rl}mR(H_2\circ S)& =E_{\sigma }[\underset{w}{sup}\sum _{i=1}^m{\sigma }_i⟨w,x_i⟩]\\ & =E_{\sigma }[\underset{w}{sup}⟨w,\sum _{i=1}^m{\sigma }_i{x}_i⟩]\\ & \le C{E}_{\sigma }[\Vert \sum _{i=1}^m{\sigma }_i{x}_i{\Vert }_2]\\ & \le C(E_{\sigma }[\Vert \sum _{i=1}^m{\sigma }_i{x}_i{\Vert }_2^2]{)}^{1/2}(\mathrm{均}\mathrm{值}\mathrm{不}\mathrm{等}\mathrm{式})\\ & =C(\sum _{i\ne j}⟨x_i,x_j⟩E_{\sigma }[{\sigma }_i{\sigma }_j]+\sum _{i=1}⟨x_i,x_i⟩E_{{\sigma }_i^2}{)}^{1/2}\\ & =C(\sum _{i=1}\Vert x_i{\Vert }_2^2{)}^{1/2}\\ & \le C\sqrt{m\underset{i}{max}\Vert x_i{\Vert }_2^2}\end{array}$$