机器学习第4章: 监督学习

合集 - 机器学习(6)

1.机器学习第1章: 概述2024-01-222.机器学习第2章: 优化2024-01-193.机器学习第3章: 泛化2024-01-22

4.机器学习第4章: 监督学习2024-01-22

5.机器学习第5章: 无监督学习2024-01-226.机器学习第6章: 其他2024-01-22

收起

Chapter 4: Supervised Learning

Acknowledgment: Most of the knowledge comes from Yuan Yang's course "Machine Learning".

Linear Regression

就是一个很传统的统计学任务。
用最小二乘法可知，$w^{\ast }=(X^{\mathrm{\top }}X{)}^{-1}{X}^{\mathrm{\top }}Y$. But the inverse is hard to compute.

We can use gradient descent. Define the square loss:

$$L(f,x_i,y_i)=\frac{1}{2}(f(x_i)-y_i{)}^2$$

What if the task is not regression but classification?

Consider binary classification at first. A naive way is to use positive and negative to represent 2 classes. However, the gradient can not give meaningful gradient information.

方法一：没有导数我就设计一种不需要导数信息的算法——感知机

方法二：把硬的变软，人为创造出导数——logistics regression

Perceptron

Intuition: adding $x$ to $w$ will make $w^x$ larger.

Limitation: it can only learn linear functions, so it does not converge if data is not linearly separable.

Update rule: Adjust $w$ only when make mistakes. If $y>0$ and $wx<0$:

$$w=w+x$$

If $y<0$ and $wx>0$:

$$w=w-x$$

combine these 2 into if $ywx<0$ $(y_i\in \{-1,+1\})$

$$w=w+y_i{x}_i$$

Convergence Proof:

Assume $\Vert w^{\ast }\Vert =1$ and $\text{exist}\gamma >0\text{s.t.}\mathrm{\forall }i{y}_i{w}^{\ast }{x}_i\ge \gamma$.
And $\Vert x_i\Vert \le R$. Then at most $\frac{R^2}{{\gamma }^2}$ mistakes.

$$\begin{array}{rl}⟨w_{t+1},w^{\ast }⟩& \ge ⟨w_t,w^{\ast }⟩+⟨y_t{x}_t,w^{\ast }⟩(\text{update rule})\\ & \ge ⟨w_t,w^{\ast }⟩+\gamma (\text{margin assumption})\end{array}$$

Start from $w_0=0$ Telescoping

$$⟨w_{t+1},w\ast ⟩\ge t\gamma$$

Then

$$\Vert w_{t+1}\Vert =\Vert w_{t+1}\Vert \Vert w^{\ast }\Vert \ge ⟨w_{t+1},w\ast ⟩\ge t\gamma$$

On the other hand:

$$\begin{array}{rl}\Vert w_{t+1}{\Vert }^2& =\Vert w_t+y_t{x}_t{\Vert }^2\\ & =\Vert w_t{\Vert }^2+\Vert y_t{x}_t{\Vert }^2+2⟨y_t{x}_t,w_t⟩\\ & \le \Vert w_t{\Vert }^2+\Vert y_t{x}_t{\Vert }^2\text{(update only when mistake)}\\ & \le \Vert w_t{\Vert }^2+R^2\text{(R-condition)}\end{array}$$

Telescoping:

$$\Vert w_{t+1}{\Vert }^2\le t{R}^2$$

So

$$\begin{array}{rl}{t}^2{\gamma }^2& \le \Vert w_{t+1}{\Vert }^2\le t{R}^2\\ t& \le \frac{R^2}{{\gamma }^2}\end{array}$$

Logistic Regression(逻辑回归)

将分类问题转化为概率的回归问题。

Instead of using a sign function, we can output a probability. Here comes the important thought we have used in matrix completion. Relaxation!!!

Make the hard function $\text{sign}(z)$ soft:

$$\frac{1}{1+e^{-z}}$$

It remains to define a loss function. L1 and L2 are both not good enough, let's come to cross entropy:

$$L(y,p)=-\sum _i{y}_i\log p_i$$

Explanation:

• We already know the actual probability distribution $y$.

• We estimate the difference of $p_i$ and $y_i$.

feature learning

Linear regression/classification can learn everything!

• If the feature is correct

In general, linear regression is good enough, but feature learning is hard

Deep learning is also called “representation learning” because it learns features automatically.

• The last step of deep learning is always linear regression/classification!

Regularization

This is a trick to avoid overfitting.

Ridge regression

$$\begin{array}{rl}minL& =\frac{1}{2N}\sum _i(w^T{x}_i-y_i{)}^2+\frac{\lambda }{2}\Vert w{\Vert }_2^2\\ {\mathrm{\nabla }}_{w}L& =\frac{1}{N}\sum _i(w^T{x}_i-y_i)x_i+\lambda w\\ H& =\frac{1}{N}\sum _i{x}_i{x}_i^T+\lambda I\ge \lambda \end{array}$$

This is $\lambda$-strongly convex.

An intuitive illustration of how this works. For each gradient descent step, split it into two parts:

• Part 1: Same as linear regression
• Part 2: “Shrink” every coordinate by $(1-\eta \lambda )$ (weight decay is a very important trick today)

Until two parts “cancel” out, and achieves equilibrium.

However, ridge regression can not find important features. It is essentially linear regression + weight decay.

Although the $w$ vector will have a smaller norm, every feature may get some(possibly very small) weight

If we need to find important features, we need to optimize:

$$minL=\frac{1}{2N}\sum _i(w^T{x}_i-y_i{)}^2\text{and at same time }\Vert 𝑤{\Vert }_0\le c$$

$c$ means we want to ensure that there are at most $c$ non-zero entries, i.e., those are themost important $c$ features. Other features are not important.

Then relax "hard" to "soft" as we always do. Then we do LASSO regression:

LASSO regression

$$\begin{array}{rl}minL& =\frac{1}{2N}\sum _i(w^T{x}_i-y_i{)}^2+\lambda \Vert w{\Vert }_1\\ {\mathrm{\nabla }}_{w}L& =\frac{1}{N}\sum _i(w^T{x}_i-y_i)x_i+\lambda \text{ sign}(w)\end{array}$$

An intuitive illustration of how this works. For each gradient descent step, split it into two parts:

• Part 1: Same as linear regression
• Part 2: for every coordinate $i$:
  $$\begin{array}{}\text{(1)}& w_{t+1}^{(i)}=\{\begin{array}{ll}{\hat{w}}_t^{(i)}-\eta \lambda ,& \text{for }{\hat{w}}_t^{(i)}>\eta \lambda \\ {\hat{w}}_t^{(i)}+\eta \lambda ,& \text{for }{\hat{w}}_t^{(i)}<-\eta \lambda \\ 0,& \text{for }{\hat{w}}_t^{(i)}\in [-\eta \lambda ,\eta \lambda ]\end{array}\end{array}$$

Until two parts “cancel” out, and achieves equilibrium.

Compressed Sensing

Definition of RIP condition:

Let $W\in {\mathbb{R}}^{n\times d}$ . $W$ is $(ϵ,s)$-RIP if for all $\Vert x{\Vert }_0\le s$, we have

$$(1-ϵ)\Vert x{\Vert }_2^2\le \Vert Wx{\Vert }_2^2\le (1+ϵ)\Vert x{\Vert }_2^2$$

This is called the Restricted Isometry Property (RIP). It means $W$ is (almost) not expanding $x$ towards any direction, as long as $x$ is sparse. It is close to the identity matrix in the "sparse space of $x$," whatever it means.

Without the sparsity condition, this is impossible (think about $W$'s eigenspace; there exists an eigenvector with a $0$ eigenvalue).

Naïve application for RIP (Theorem 1)

Theorem: Let $ϵ<1$ and let $W$ be a $(ϵ,2s)$-RIP matrix. Let $x$ be a vector such that $\Vert x{\Vert }_0\le s$, let $y=Wx$ be the compression of $x$, and let

$$\tilde{x}\in \arg \underset{v:Wv=y}{min}\Vert v{\Vert }_0$$

be a reconstructed vector. Then $\tilde{x}=x$.

x is the only vector that gives y as the result after applying W under sparse conditions.

proof: 反证法 注意到是 $(ϵ,2s)$-RIP

If not, i.e., $\tilde{x}\ne x$. Since $Wx=y$, we know $\Vert \tilde{x}{\Vert }_0\le \Vert x{\Vert }_0\le s$, so $\Vert x-\tilde{x}{\Vert }_0\le 2s$. We apply RIP condition on $x-\tilde{x}$ and get

$$(1-ϵ)\Vert x-\tilde{x}{\Vert }_2^2\le \Vert W(x-\tilde{x}){\Vert }_2^2\le (1+ϵ)\Vert x-\tilde{x}{\Vert }_2^2$$

Notice that $\Vert W(x-\tilde{x}){\Vert }_2^2=0$, and $\Vert x-\tilde{x}{\Vert }_2^2\ne 0$, so we have $0\le (1-ϵ)\le 0\le (1+ϵ)$, which is a contradiction.

这个定理还是有一样的问题，0阶不好优化，需要relax到一阶

Theorem2: 0-norm $\to$ 1-norm

Let $ϵ<1$ and let $W$ be a $(ϵ,2s)$-RIP matrix. Let $x$ be a vector such that $\Vert x{\Vert }_0\le s$, let $y=Wx$ be the compression of $x$, and $ϵ<\frac{1}{1+\sqrt{2}}$. Then

$$x=\arg \underset{v:Wv=y}{min}\Vert v{\Vert }_0=\arg \underset{v:Wv=y}{min}\Vert v{\Vert }_1$$

Theorem 2 + x contains noise (Theorem 3)

Let $ϵ<\frac{1}{1+\sqrt{2}}$ and let $W$ be a $(ϵ,2s)$-RIP matrix.

Let $x$ be an arbitrary vector and denote $x_s\in \arg \underset{v:\Vert v{\Vert }_0\le s}{min}\Vert x-v{\Vert }_1$.

That is, $x_s$ is the vector which equals $x$ on the $s$ largest elements of $x$ and equals $0$ elsewhere.

Let $y=Wx$ be the compression of $x$ and let $x^{\ast }\in \arg \underset{v:Wv=y}{min}\Vert v{\Vert }_1$ be the reconstructed vector. Then:

$$\Vert x^{\ast }-x{\Vert }_2\le \frac{2(1+\rho )s^{-1/2}}{1-\rho }\Vert x-x_s{\Vert }_1$$

where $\rho =\frac{\sqrt{2}ϵ}{1-ϵ}$. When $x=x_s$ ($s$-sparse), we get exact recovery $x^{\ast }=x$.

Proof:

Before the proof, we need to clarify some notations.

Given a vector $v$, a set of indices $I$, denote $v_I$ as the vector whose $i$th element is $v_i$ if $i\in I$, otherwise, the $i$th element is 0.

We partition indices as: $[d]=T_0\cup T_1\cup \dots \cup T_{\frac{d}{s}-1}$. Each $|T_i|=s$, and assume $\frac{d}{s}$ is an integer for simplicity.

$T_0$ contains the $s$ largest elements in absolute value in $x$. Then $T_0^c=[d]\setminus T_0$.

$T_1$ contains the $s$ largest elements in $h_{T_0^c}$. (not in $x$!). $T_{0,1}=T_0\cup T_1$ and $T_{0,1}^c=[d]\setminus T_{0,1}$.

$T_2$ contains the $s$ largest elements in $h_{T_{0,1}^c}$. $T_3,T_4,\dots$ are constructed in the same way.

Then based on our notation, $\Vert x-x_s{\Vert }_1=\Vert x_{T_0}^c{\Vert }_1$, 后面的证明只要都转到$\Vert x_{T_0}^c{\Vert }_1$ 即可

Then begin our proof:

Let $h=x^{\ast }-x$. We want to show that $\Vert h{\Vert }_2$ is small.

We split $h$ into two parts: $h=h_{T_{0,1}}+h_{T_{0,1}^c}$.

$h_{T_{0,1}}$ is $2s$-sparse, we move large elements inside, then use RIP to bound it.

$h_{T_{0,1}^c}$ is the rest small entries, we use intuition (袁洋老师经典的棍子问题) to bound it.

Step1: bound $\Vert h_{T_{0,1}^c}{\Vert }_2\le \Vert h_{T_0}{\Vert }_2+2s^{-1/2}\Vert x-x_s{\Vert }_1$

The first inequality is by definition of infinity norm. Sum over $j=2,3,\dots$ by triangle inequality, we have

We want to show $\Vert h_{T_0^c}{\Vert }_1$ will not be too big.

$$\begin{array}{rl}\Vert h_{T_{0,1}^c}{\Vert }_2& \le \sum _{j\ge 2}\Vert h_{T_j}{\Vert }_2(\text{均值不等式})\\ & \le \sum _{j\ge 2}\sqrt{s}\Vert h_{T_j}{\Vert }_{\mathrm{\infty }}(\text{根据无穷norm的定义})\\ & \le \sum _{j\ge 2}\frac{\Vert h_{T_{j-1}}{\Vert }_1}{\sqrt{s}}(\ast )\\ & =\frac{\Vert h_{T_0^c}{\Vert }_1}{\sqrt{s}}(\text{根据1-norm的定义})\end{array}$$

解释一下这个式子(*): For any $j>1$, $\mathrm{\forall }i\in T_j$, $i^{\prime }\in T_{j-1}$, $|h_i|\le |h_{i^{\prime }}|$ (by definition of $T$). So

$$\Vert h_{T_j}{\Vert }_{\mathrm{\infty }}\le \frac{\Vert h_{T_{j-1}}{\Vert }_1}{s}$$

Then we want to bound $\Vert h_{T_0}^c{\Vert }_1$。

$x^{\ast }=x+h$ has minimal ${\ell }_1$ norm in all vector satisfy $Wx=y$, and $x$ satisfies $Wx=y$.

So:

$$\begin{array}{rl}\Vert x{\Vert }_1& \ge \Vert x+h{\Vert }_1\\ & =\sum _{i\in T_0}|x_i+h_i|+\sum _{i\in T_0^c}|x_i+h_i|\\ & \ge \Vert x_{T_0}{\Vert }_1-\Vert h_{T_0}{\Vert }_1+\Vert h_{T_0^c}{\Vert }_1-\Vert x_{T_0^c}{\Vert }_1(\text{绝对值不等式})\\ \Vert h_{T_0^c}{\Vert }_1& \le \Vert h_{T_0}{\Vert }_1+\Vert x_{T_0^c}{\Vert }_1+\Vert x{\Vert }_1-\Vert x_{T_0}{\Vert }_1\\ & =\Vert h_{T_0}{\Vert }_1+2\Vert x_{T_0^c}{\Vert }_1\text{(1-norm可以这样直接加减运算，真好)}\\ & \le s^{1/2}\Vert h_{T_0}{\Vert }_2+2\Vert x_{T_0^c}{\Vert }_1\text{(均值不等式)}\\ & \le s^{1/2}\Vert h_{T_{0,1}}{\Vert }_2+2\Vert x_{T_0^c}{\Vert }_1\text{(柯西不等式)}\end{array}$$

因为我们等一下也要bound$|h_{T_{0,1}}{|}_2$ 所以这边放缩到这里，也是没有问题的。

总结一下，经过step1之后，我们得到了什么

$$\begin{array}{rl}\Vert h{\Vert }_2& \le \Vert h_{T_{0,1}}{\Vert }_2+\Vert h_{T_{0,1}^c}{\Vert }_2\text{(均值不等式)}\\ & \le s^{-1/2}\Vert h_{T_0^c}{\Vert }_1+\Vert h_{T_{0,1}^c}{\Vert }_2\\ & \le s^{-1/2}(s^{1/2}\Vert h_{T_{0,1}}{\Vert }_2+2\Vert x_{T_0^c}{\Vert }_1)+\Vert h_{T_{0,1}^c}{\Vert }_2\\ & \le 2\Vert h_{T_{0,1}}{\Vert }_2+2s^{-1/2}\Vert x_{T_0^c}{\Vert }_1\end{array}$$

Step2: Then we want to bound $\Vert h_{T_{0,1}}{\Vert }_2$

Use RIP condition bound these big terms:

$$\begin{array}{rl}(1-ϵ)\Vert h_{T_{0,1}}{\Vert }_2^2& \le \Vert W{h}_{T_{0,1}}{\Vert }_2^2\\ & =⟨W{h}_{T_{0,1}},W{h}_{T_{0,1}}⟩\\ & =-\sum _{j\ge 2}⟨W{h}_{T_0}+W{h}_{T_1},W{h}_{T_j}⟩\end{array}$$

The last equality is because $W{h}_{T_{0,1}}=Wh-\sum _{j\ge 2}W{h}_{T_j}=-\sum _{j\ge 2}W{h}_{T_j}$,

The final result is the inner products between disjoint index sets. We use a very useful lemma:

Theorem: Let $W$ be $(ϵ,2s)$-RIP, $\mathrm{\forall }I,J$ disjoint sets of size $\le s$, for any vector $\mu$ we have $⟨W{u}_I,W{u}_J⟩\le ϵ\Vert u_I\Vert \Vert u_J\Vert$

proof of this small thm:

WLOG assume $\Vert u_I\Vert =\Vert u_J\Vert =1$.

$$\begin{array}{r}⟨W{u}_I,W{u}_J⟩=\frac{\Vert W{u}_I+W{u}_J{\Vert }^2-\Vert W{u}_I-W{u}_J{\Vert }^2}{4}(\mathrm{极}\mathrm{化}\mathrm{恒}\mathrm{等}\mathrm{式})\end{array}$$

We can bound these two terms by RIP condition:

Since $|J\cup I|\le 2s$, we get from RIP condition that (notice $⟨u_I,u_J⟩=0$)

$$\begin{array}{rl}\Vert W{u}_I+W{u}_J{\Vert }^2& =\Vert W(u_I+u_J){\Vert }^2\\ & \le (1+ϵ)(\Vert u_I{\Vert }^2+\Vert u_J{\Vert }^2)=2(1+ϵ)\\ -\Vert W{u}_I-W{u}_J{\Vert }^2& =-\Vert W(u_I-u_J){\Vert }^2\\ & \le -(1-ϵ)(\Vert u_I{\Vert }^2+\Vert u_J{\Vert }^2)=-2(1-ϵ)\end{array}$$

Now come back to the original proof.

Therefore, we have for $i\in \{0,1\}$, $j\ge 2$: $|⟨W{h}_{T_i},W{h}_{T_j}⟩|\le ϵ\Vert h_{T_i}{\Vert }_2\Vert h_{T_j}{\Vert }_2$.

Continue:

$$\begin{array}{rl}(1-ϵ)\Vert h_{T_{0,1}}{\Vert }_2^2& \le -\sum _{j\ge 2}⟨W{h}_{T_0}+W{h}_{T_1},W{h}_{T_j}⟩\\ & \le ϵ\sum _{j\ge 2}(\Vert h_{T_0}{\Vert }_2+\Vert h_{T_1}{\Vert }_2)\Vert h_{T_j}{\Vert }_2\text{(前面的那个定理)}\\ & \le \sqrt{2}ϵ\Vert h_{T_{0,1}}{\Vert }_2\sum _{j\ge 2}\Vert h_{T_j}{\Vert }_2\text{(均值不等式)}\\ & \le \sqrt{2}ϵ\Vert h_{T_{0,1}}{\Vert }_2\frac{\Vert h_{T_0^c}{\Vert }_1}{\sqrt{s}}\text{(前面证过)}\\ \Vert h_{T_{0,1}}{\Vert }_2& \le \frac{\sqrt{2}ϵ}{1-ϵ}{s}^{-1/2}\Vert h_{T_0^c}{\Vert }_1\end{array}$$

$\Vert h_{T_0}^c\Vert$ 我们是bound过的呀，但是这里又转到 $\Vert h_{T_{0,1}}{\Vert }_2$，互相bound？这么奇怪。

Continue: 设$\frac{\sqrt{2}ϵ}{1-ϵ}=\rho$

$$\begin{array}{rl}\Vert h_{T_{0,1}}{\Vert }_2& \le \rho s^{-1/2}\Vert h_{T_0^c}{\Vert }_1\\ & \le \rho \Vert h_{T_{0,1}}{\Vert }_2+2\rho s^{-1/2}\Vert x_{T_0^c}{\Vert }_1\\ \Vert h_{T_{0,1}}{\Vert }_2& \le \frac{2s^{-1/2}\rho }{1-\rho }\Vert x_{T_0^c}{\Vert }_1\end{array}$$

然后我们把这个再代入前面的式子:

$$\begin{array}{rl}\Vert h{\Vert }_2& \le 2\Vert h_{T_{0,1}}{\Vert }_2+2s^{-1/2}\Vert x_{T_0^c}{\Vert }_1\\ & \le 2s^{-1/2}(\frac{2\rho }{1-\rho }+1)\Vert x_{T_0^c}{\Vert }_1\\ & =\frac{2s^{-1/2}(1+\rho )}{1-\rho }\Vert x_{T_0^c}{\Vert }_1\end{array}$$

这样就证完了。

How to construct a RIP matrix?

Random matrix

Theorem 4: Let $U$ be an arbitrary fixed $d\times d$ orthonormal matrix, let $ϵ,\delta$ be scalars in $(0,1)$. Let $s$ be an integer in $[d]$, let $n$ be an integer that satisfies:

$$n\ge \frac{100s\log (\frac{40d}{\delta ϵ})}{{ϵ}^2}$$

Let $W\in {\mathbb{R}}^{n\times d}$ be a matrix such that each element of $W$ is distributed normally with zero mean and variance of $1/n$. Then with a probability of at least $1-\delta$ over the choice of $W$, the matrix $WU$ is $(ϵ,s)$-RIP.

解释：

• Matrix $U$ can be identity, so $W$ is also $(ϵ,s)$-RIP.

• Before we delve deeper into this proof, we discuss a question: Why is $WU$ useful?

  • $y=Wx$, $W$ is RIP, but $x$ can not always be sparse.
  • But also $y=(WU)a$, $WU$ is still RIP, $U$ is orthonormal, $a$ is sparse. 也就是可以加一个任意的正交的矩阵，将不sparse的输入变成sparse。

The rough idea of this proof is:

RIP condition holds for all possible (infinitely many) sparse vectors, but we want to apply union bound. So:

• Continuous space $\to$ finite number of points

• Consider a specific index set $I$ of size $s$(get this RIP condition on this specific index set)

• Use this index set to enter the sparse space

• Apply union bound over all possible $I$

Let's begin our proof:

Lemma 2:

Let $ϵ\in (0,1)$. There exists a finite set $Q\subseteq {\mathbb{R}}^d$ of size $|Q|\le (\frac{5}{ϵ}{)}^d$ such that:

$$\underset{x:\Vert x\Vert \le 1}{sup}\underset{v\in Q}{min}\Vert x-v\Vert \le ϵ$$

意义:

我们能够用有限的点，覆盖一个高维空间(此处覆盖的意思是，高维空间中的任意一点，到我们取的有限的点集的最小距离，都很小)。也就是对应我们idea中的第一步——"Continuous space $\to$ finite number of points"

Proof:

假设$ϵ=1/k$. 把k上取整成整数，let

$$Q^{\prime }=\{x\in {\mathbb{R}}^d:\mathrm{\forall }j,\mathrm{\exists }i\in \{-k,-k+1,\dots ,k\}\text{ s.t. }{x}_j=i/k\}$$

也就是$Q^{\prime }$中的每个点，的所有坐标，都是$[-1,1]$之间的的2k+1分点上。那这样，空间中的任意一点都一定处在一个方格中，到方格顶点的距离，最大是$\frac{1}{k}\le ϵ$.

Clearly, $|Q^{\prime }|=(2k+1{)}^d$(每个维度2k+1个点嘛).

We shall set $Q=Q^{\prime }\cap B_2(1)$, where $B_2(1)$ is the unit $L_2$ ball of ${\mathbb{R}}^d$. 然后应该还有一些计算，会得到$(\frac{5}{ϵ}{)}^d$这个答案。

JL lemma:

Let $Q$ be a finite set of vectors in ${\mathbb{R}}^d$. Let $\delta \in (0,1)$ and $n$ be an integer such that:

$$ϵ=\sqrt{\frac{6\ln (\frac{2|Q|}{\delta })}{n}}\le 3$$

$$n=\frac{6\ln (\frac{2|Q|}{\delta })}{{ϵ}^2}$$

Then, with probability of at least $1-\delta$ over a choice of a random matrix $W\in {\mathbb{R}}^{n\times d}$ such that each element of $W$ is independently distributed according to $\mathcal{N}(0,1/n)$, we have:

$$\underset{x\in Q}{sup}|\frac{\Vert Wx{\Vert }^2}{\Vert x{\Vert }^2}-1|<ϵ$$

意义:

• 这个定理已经就和我们最后要证的RIP的条件几乎一模一样了。只不过这里x不是sparse，而是在finite set中取。也就是对应我们idea中的第二步——"Consider a specific index set $I$ of size $s$"

• n的取值大小，和维数d没有关系了，只和finite set的size d有关，那么我们就可以用一个很小的Q和一个维数很低的n*d的矩阵W来做操作。

proof: omit here.

Lemma 3:

Let $U$ be an orthonormal $d\times d$ matrix and let $I\subset [d]$ be a set of indices of size $|I|=s$. Let $S$ be the span of $\{U_i:i\in I\}$, where $U_i$ is the $i$-th column of $U$. Let $\delta \in (0,1)$, $ϵ\in (0,1)$, and $n$ be an integer such that:

$$n\ge 24\frac{\ln \left(\frac{2}{\delta }\right)+s\ln \left(\frac{20}{ϵ}\right)}{{ϵ}^2}$$

Then, with probability of at least $1-\delta$ over a choice of a random matrix $W\in {\mathbb{R}}^{n\times d}$ such that each element of $W$ is independently distributed according to $\mathcal{N}(0,1/n)$, we have:

$$\underset{x\in S}{sup}|\frac{\Vert Wx\Vert }{\Vert x\Vert }-1|<ϵ$$

意义:

lemma3已经非常接近我们最后要证的东西了，也就是对应idea中的第三步——"Use this index set to enter the sparse space"

首先解释一下: $S$是index的子集对应的多个列向量张成的向量空间，那其中的元素x，就是他们的线性组合，所以We can write $x=U_{I}a$ where $a\in {\mathbb{R}}^s$

If Lemma 3 is true, we know for any $x\in S$,

$$(1-ϵ)\le \frac{\Vert Wx\Vert }{\Vert x\Vert }\le (1+ϵ)$$

Which implies that

$$\begin{array}{rl}(1-3ϵ)& \le \frac{\Vert Wx{\Vert }^2}{\Vert x{\Vert }^2}\le (1+3ϵ)(\text{就是同时平方})\\ (1-3ϵ)& \le \frac{\Vert W{U}_{I}a{\Vert }^2}{\Vert U_{I}a{\Vert }^2}\le (1+3ϵ)(x=U_{I}a\mathrm{代}\mathrm{入})\\ (1-3ϵ)& \le \frac{\Vert W{U}_{I}a{\Vert }^2}{\Vert a{\Vert }^2}\le (1+3ϵ)\end{array}$$

因为a是unit length，所以我们只需要bound $\Vert W{U}_{I}a\Vert$或者说$\Vert Wx\Vert$

只要lemma3正确，然后做第四步——"apply union bound over all possible $I$" (from $(\genfrac{}{}{0ex}{}{d}{s})$) i.e., ${\delta }^{\prime }=\delta \cdot d^s$，我们就证完了。

proof:

大致思路已经很清晰了，首先用JL lemma bound住离散的点$v$上的这个模长$\Vert W{U}_{I}v\Vert$，再用lemma2 bound从离散集合到全空间所有点的一个差距$\Vert a-v\Vert$也就是$\Vert W{U}_I(a-v)\Vert$，那我们也就bound住了$\Vert W{U}_{I}a\Vert$.

It suffices to prove the lemma for all $x\in S$ of the unit norm.

We can write $x=U_{I}a$ where $a\in {\mathbb{R}}^s$, $\Vert a{\Vert }_2=1$, and $U_I$ is the matrix whose columns are $\{U_i:i\in I\}$. 这里简单解释一下: 我前面说过，$x$是列向量的线性组合，所以写成了$x=U_{I}a$，又因为$x$和$U$的列向量都是unit length，$U$的列向量还相互之间正交，所以$a$也是unit length

Using Lemma 2 we know that there exists a set $Q$ of size $|Q|\le (\frac{20}{ϵ}{)}^s$ such that：

$$\underset{a:\Vert a\Vert =1}{sup}\underset{v\in Q}{min}\Vert a-v\Vert \le ϵ/4$$

Since $U$ is orthonormal, we also have

$$\underset{a:\Vert a\Vert =1}{sup}\underset{v\in Q}{min}\Vert U_{I}a-U_{I}v\Vert \le ϵ/4$$

也就是:

$$\underset{x\in S}{sup}\underset{v\in Q}{min}\Vert x-U_{I}v\Vert \le ϵ/4$$

Apply JL lemma on $\{U_{I}v:v\in Q\}$, we know that for:

$$n\ge 4\cdot 6\frac{\ln (\frac{2|Q|}{\delta })}{{ϵ}^2}=24\frac{\ln (\frac{2}{\delta })+s\ln (\frac{20}{ϵ})}{{ϵ}^2}$$

We can have:

$$\underset{v\in Q}{sup}|\frac{\Vert W{U}_{I}v{\Vert }^2}{\Vert U_{I}v{\Vert }^2}-1|\le ϵ/2$$

This also implies

$$\underset{v\in Q}{sup}|\frac{\Vert W{U}_{I}v\Vert }{\Vert U_{I}v\Vert }-1|\le ϵ/2$$

Let $a$ be the smallest value such that for all $x\in S$, $\frac{\Vert Wx\Vert }{\Vert x\Vert }\le 1+a$, clearly $a<\mathrm{\infty }$.(显然有上确界，然后设出来，然后求他) We want to show $a<ϵ$. This is the right half of the RIP condition.

$\mathrm{\forall }x\in S$ of unit norm, there exists $v\in Q$ such that $\Vert x-U_{I}v\Vert \le ϵ/4$, and therefore

$$\begin{array}{rl}\Vert Wx\Vert & \le \Vert W{U}_{I}v\Vert +\Vert W(x-U_{I}v)\Vert \\ & \le (1+ϵ/2)\Vert U_{I}v\Vert +\Vert W(x-U_{I}v)\Vert \text{(JL lemma的条件作用在某个index的离散点集上)}\\ & \le (1+ϵ/2)\Vert U_{I}v\Vert +(1+a)ϵ/4\text{(lemma2和我们自己加的上确界假设)}\end{array}$$

Notice that $\Vert U_{I}v\Vert \le 1$, because $U_I$ does not change length, and $v$ is in $Q$(Q 肯定在单位球内). So

$$\begin{array}{rl}\Vert Wx\Vert & \le 1+\frac{ϵ}{2}+(1+a)ϵ/4\\ \frac{\Vert Wx\Vert }{\Vert x\Vert }& \le 1+\frac{ϵ}{2}+\frac{(1+a)ϵ}{4}\end{array}$$

By definition of $a$, we know

$$\begin{array}{rl}a& \le \frac{ϵ}{2}+\frac{(1+a)ϵ}{4}\\ a& \le \frac{\frac{3ϵ}{4}}{1-\frac{ϵ}{4}}\le ϵ\end{array}$$

There may be some problem with the left half in PPT. TODO

Between this lemma 3 and the thm may be a gap, but 老师上课没讲。

作业中还有提到用random orthonormal basis 来构造RIP matrix的，这个在后面decision tree中会出现。

Support Vector Machine

概念解释

支持向量机(SVM)是一种二分类模型，它的基本模型是定义在特征空间上的间隔(margin)最大的线性分类器。

Margin: distance from the separator to the closest point

Samples on the margin are "support vectors".

数学表达式:

找到一个超平面$wx+b=0$, s.t.

$$\begin{array}{rl}\text{for}{y}_i& =1,w{x}_i+b>1\\ \text{for}{y}_i& =-1,w{x}_i+b<1\\ & min\frac{1}{\Vert w{\Vert }_2}\end{array}$$

If the space is perfectly linear separable, this can be solved using quadratic programming in polytime.

那如果不是线性可分的呢？

Naïve answer:
$min\Vert w{\Vert }_2+\lambda \cdot$ mistakes

i.e.

$$min\Vert w{\Vert }_2+\lambda \sum _{i}1\{y_i(w^T{x}_i)<1\}$$

This is NP-Hard to minimize.

The indicator function is hard to optimize. Make it soft.

Relax the hard constraint:

$$\begin{array}{rl}& min\Vert w{\Vert }_2+\lambda \sum _i{\xi }_i\\ & \mathrm{\forall }i,y_i{w}^T{x}_i\ge 1-{\xi }_i,{\xi }_i\ge 0\end{array}$$

${\xi }_i$ is a "slack variable".

"Violating the constraint" a little bit is OK. SVM with "Soft" margin.

This is called SVM with a "Soft" margin.

计算

Then we use "Dual" to solve SVM:

线性可分的情况:

Primal:

$$\begin{array}{rl}& min\frac{\Vert w{\Vert }_2^2}{2}\\ & \mathrm{\forall }i,y_i{w}^T{x}_i\ge 1\end{array}$$

Dual:

$$\begin{gathered} L(w,\alpha )=\frac{\Vert w{\Vert }_2^2}{2}-\sum _i{\alpha }_i(y_i{w}^T{x}_i-1) \\ \mathrm{\forall }i,{\alpha }_i\ge 0 \end{gathered}$$

Take derivative:

$$\frac{\partial L}{\partial w}=0\Rightarrow w=\sum _i{\alpha }_i{y}_i{x}_i$$

Plug it into $L(w,\alpha )$, we get:

$$L(w,\alpha )=\sum _i{\alpha }_i-\frac{1}{2}\sum _i\sum _j{y}_i{y}_j{\alpha }_i{\alpha }_j⟨x_i,x_j⟩$$

The relaxed case

Primal:

$$\begin{array}{rl}& min\frac{1}{2}\Vert w{\Vert }_2^2+\lambda \sum _i{\xi }_i\\ & \mathrm{\forall }i,y_i(w^T{x}_i+b)\ge 1-{\xi }_i,{\xi }_i\ge 0\end{array}$$

The Lagrangian:

$$L(w,b,\xi ,a,\kappa )=\frac{1}{2}\Vert w{\Vert }_2^2+\lambda \sum _i{\xi }_i-\sum _i{a}_i(y_i(w^T{x}_i+b)-1+{\xi }_i)-\sum _i{\kappa }_i{\xi }_i$$

where ${\alpha }_i\ge 0$, ${\kappa }_i\ge 0$

Take derivative, we get the optimality conditions:

$$\begin{array}{rl}\frac{\partial L}{\partial w}& =w-\sum _i{a}_i{y}_i{x}_i=0\\ \frac{\partial L}{\partial b}& =\sum _i{y}_i{a}_i=0\\ \frac{\partial L}{\partial {\xi }_i}& =\lambda -a_i-{\kappa }_i=0\end{array}$$

So, $a_i=\lambda -{\kappa }_i\le \lambda$. Here is one more constraint than the linearly separable case.

Then we put all the solution into $L(w,b,\xi ,a,\kappa )$:

$$\begin{array}{rl}L(w,b,\xi ,a,\kappa )& =\frac{1}{2}\sum _i\sum _j{y}_i{y}_j{a}_i{a}_j⟨x_i,x_j⟩+\sum _i({\lambda }_i-{\kappa }_i){\xi }_i-\sum _i{a}_i{y}_i{w}^T{x}_i+\sum _i{a}_i{y}_i{b}_i(=0)+\sum _i{a}_i-\sum _i{a}_i{\xi }_i\\ & =\frac{1}{2}\sum _i\sum _j{y}_i{y}_j{a}_i{a}_j⟨x_i,x_j⟩+\sum _i{a}_i{\xi }_i-\sum _i\sum _j{y}_i{y}_j{a}_i{a}_j⟨x_i,x_j⟩+\sum _i{a}_i-\sum _i{a}_i{\xi }_i\\ & =\sum _i{a}_i-\frac{1}{2}\sum _i\sum _j{y}_i{y}_j{a}_i{a}_j⟨x_i,x_j⟩\end{array}$$

The last formulation is the same as the linearly separable case.

kernel trick

We can use a kernel function to transform non-linearly separable into high dimensional space, in which the data is linearly separable.

Two problems:

• kernal function $\varphi (x)$ dim too high, hard to compute.

• The separator has a higher dim, hard to compute.

For $⟨\varphi (x_1),\varphi (x_2)⟩$, we can get it from $⟨x_1,x_2⟩$. e.g. Quadratic kernel in PPT and Gauss kernel we always use.

$max\sum _i{a}_i-\frac{1}{2}\sum _i\sum _j{y}_i{y}_j{a}_i{a}_j⟨\varphi (x_i),\varphi (x_j)⟩$

• Only need to do $n^2$ kernel computations $⟨\varphi (x_i),\varphi (x_j)⟩$

When a new data point $x$ comes, what do we do? Not computing $\varphi (x)$ -- can be expensive!

• $w^T\varphi (x)=\sum _i{a}_i{y}_i⟨\varphi (x_i),\varphi (x)⟩$
  Assume $r$ support vectors, then we only need to do at most $r$ kernel computations

Therefore, no need for computing $\varphi (x)$!

Move a step further, do we need to define $\varphi (x)$?

Mercer’s Theorem: If the kernel matrix is positive semidefinite for any $\{x_i\}$, then we know there exists $\varphi (\cdot )$ such that K is the kernel for $\varphi (\cdot )$

That means we may define a $K$ function for SVM, without knowing the exact format of $\varphi (\cdot )$!!!

Decision Tree

pro: good explanations

con: big and deep, hard to switch, overfit easily.

前置知识

boolean function analysis: $f:\{-1,1{\}}^n\stackrel{}{\to }[0,1]$

Fourier basis of boolean function: ${\chi }_S(x)=\prod _{i\in S}{x}_i,S\subset [n]$

$f(x)=\sum _S{\hat{f}}_S{\chi }_S(x),{\hat{f}}_S=⟨f,{\chi }_S(x)⟩=E_{x\sim D}[f(x){\chi }_S(x)]$

$L_1(f)=\sum _S|{\hat{f}}_S|$

$L_0(f)=|S:{\hat{f}}_S\ne 0|$ sparsity 就是$L_0(f)$

A low degree means the degree of all its terms is bounded by this number. 也就是$S\subset [n],|S|\le degree$

Convert decision tree to low-degree sparse function

Thm: for any decision tree $T$ with $s$ leaf nodes, there exists a degree-$\log \frac{s}{ϵ}$ and sparsity-$\frac{s^2}{ϵ}$ function $h$ that $4ϵ$-approximates $T$.

Proof:

Step 1: bounded its depth. Truncating $T$ at depth $\log \frac{s}{ϵ}$

This means there are at most $\frac{s}{ϵ}$ nodes remaining only when it is a full binary tree.

It differs by at most $\frac{ϵ}{s}\times s=ϵ$. 这是为什么呢？因为走到每个最后一层的node的概率是$\frac{ϵ}{s}$，然后最多有s个node底下连到最终的leaves，也就是会被切掉。

(注意：这里已经造成了$ϵ$的误差了)

So below we assume $T$ has depth at most $\log \frac{s}{ϵ}$

Step 2: bounded its degree and L1 norm

A tree with $𝑠$ leaf nodes can be represented by union of $𝑠$ “AND” terms

这是为什么呢？以下面这个图为例，假如我想要走到其中一个node，比如$x_2$右侧的那个1，必须$x_1=1,x_2=1$， 那么这个node就可以被写成$x_1\wedge x_2$

每个node都可以这样表示：${\wedge }_{i\in S}(x_i={\text{flag}}_i),{\text{flag}}_i\in \{-1,1\}$

那么这个函数的各个系数分别是多少，可以用这个公式 ${\hat{f}}_S=⟨f,{\chi }_S(x)⟩=E_{x\sim D}[f(x){\chi }_S(x)]$进行计算。算出来一定刚好是

$x_1\wedge x_2=\frac{1}{4}+\frac{1}{4}{x}_1+\frac{1}{4}{x}_2+\frac{1}{4}{x}_1\cdot x_2$

Because every AND term has $L_1=1$ and at most $\log \frac{s}{ϵ}$ variables(depth is at most $\log \frac{s}{ϵ}$), so $L_1(f)\le s$ and degree of $f$ bounded by $\log \frac{s}{ϵ}$

Step 3

We need to prove:

For $f$ s.t. $L_1(f)\le s$ and degree at most $\log \frac{s}{ϵ}$, there is a function $h$ with $L_0\le \frac{s^2}{ϵ}$ s.t. $E[(f-h{)}^2]\le ϵ$.

Proof:

Let $h$ include all terms in $f$ s.t. $|{\hat{f}}_S|\ge \frac{ϵ}{L_1(f)}$. degree的条件肯定满足了.

这样的term最多只有$L_1(f)/\frac{ϵ}{L_1(f)}=\frac{L_1(f{)}^2}{ϵ}\le \frac{s^2}{ϵ}$. Sparsity的条件也满足了.

By Parseval identity, the missing term has a contribution at most：

$$E[(f-h{)}^2]=\sum _{S\notin h}({\hat{f}}_S{)}^2\le \underset{S\notin h}{max}{\hat{f}}_S\sum _{S\notin h}{\hat{f}}_S\le \frac{ϵ}{L_1(f)}\times L_1(f)=ϵ$$

这样误差的要求也满足。

上下两个误差相结合即可。

好了，现在我们已经把学一个决策树的问题转化为学一个函数的问题，而且这个函数在bool function basis上是low degree and sparse. 我们要怎么求这个函数在这组基分解上的系数呢？

Theoretically analysis

KM algorithm: (Not required)

Key point: recursively prune the less promising set of basis and explore a promising set of basis.

$f_{\alpha }$ means the function of all the Fourier basis starting with $\alpha$ summed together.

以一个由$x_1,x_2,x_3$三个变量的函数为例

$$\begin{array}{rl}{f}_1& ={\hat{f}}_{x_1{x}_2{x}_3}{x}_2{x}_3+{\hat{f}}_{x_1{x}_2}{x}_2+{\hat{f}}_{x_1{x}_3}{x}_3+{\hat{f}}_{x_1}\text{有x1的项，然后x1=1}\\ f_0& ={\hat{f}}_{x_2{x}_3}{x}_2{x}_3+{\hat{f}}_{x_2}{x}_2+{\hat{f}}_{x_3}{x}_3+{\hat{f}}_1\text{没有x1的项}\end{array}$$

All these functions are well defined and also satisfy parseval identity.

e.g.

$$E[f_{11}]={\hat{f}}_{x_1{x}_2{x}_3}^2+{\hat{f}}_{x_1{x}_2}^2$$

Algorithm process:

def Coef(a):

​ If $E[f_a^2]\ge {\theta }^2$:

​ If $|a|=n$, then output $a$ the degree can not be larger than n

​ Else Coef(a0); Coef(a1) // 感觉这也太暴力了吧qwq

大概就是只留下，系数比therohold大的节点，但是尽可能延展，然后又不会太长的节点。

当然这里大家肯定有疑问，我们要怎么样计算$E[f_a^2]$呢？

Lem3.2:

For any function $f$, any $1\le k<n$, any $\alpha \in [-1,1{]}^k$, and any $x\in [-1,1{]}^{n-k}$, $f_{\alpha }(x)={\mathbb{E}}_{y\in [-1,1{]}^k}[f(yx){\chi }_{\alpha }(y)]$.

大概就是说，除了$\alpha$确定的那些bool变量，剩下的没有确定的bool变量，uniformly取所有可能，然后求均值。

This formulation implies that even though we do not know how to compute the value of ${𝑓}_{\alpha }(𝑥)$ we can approximate it, by approximating the above expectation.

这个算法有two problem:

• Pretty slow
• Sequential algorithm, cannot be done in parallel

LMN Algorithm：

For every $S\subset [n]$ with low degree, we estimate ${\hat{f}}_S$ using $m$ samples:

$${\hat{f}}_S=\frac{1}{m}\sum _{i}f(x_i){\chi }_S(x_i)$$

Do it for all $S$ to get the function.

sample complexity is small and parallelizable.

Two problems:

• does not work well in practice
• does not have guarantees in the noisy setting.

From the point of view compressed sensing, we can control the error, because Fourier basis $\{{\chi }_{𝑆}\}$ is a random orthonormal family.

Harmonica: compressed sensing

$y=Ax$

$x$ is coefficient for $X_s$, sparse in our case.

$y$ 是整个函数的在不同取值下的结果，$y_i=f(x_i)$

$A$: $a_{ij}$ is $X_{Sj}(x_i)$ 第i行代表的是第i个输入，第j列代表是第j个函数基的结果。

这个boolean function analysis是一个orthogonoal random matrix. 可以做compressed sensing 来复原$x$ .

In practice, how to build a decision tree? Gini index

For a specific node, we have to distinguish a set of variables. Then we need to determine the next node.

We use the Gini index to measure the uncertainty of a variable relative to the decision outcome.

We pick the variable with the smallest Gini index/uncertainty.

$$\begin{array}{rl}\text{Gini}(v)& =Pr[v=0]\text{Gini}(v=0)+Pr[v=1]\text{Gini}(v=1)\\ \text{Gini}(v=0)& =1-Pr[\text{result}=0|v=0{]}^2-Pr[\text{result}=1|v=0{]}^2\\ \text{Gini}(v=0)& =1-Pr[\text{result}=0|v=1{]}^2-Pr[\text{result}=1|v=1{]}^2\end{array}$$

Theoretically, if the decision tree is super deep, it can fit anything. But overfitting. We should avoid it, how? random forest.

Random forest (one of the best tools in practice)

Bagging

• Bagging是并行式集成学习方法最著名的代表。它直接基于自助采样法(bootstrap sampling)

随机森林是Bagging的一个扩展变体。

课件上面是说，我有放回的从原本的训练集中sample n次，构造一个大小一样的，但是(可能)有重复元素的集合。就相当于一个带权的子集。然后在这上面跑构造decision tree的算法。

重复B次，得到B个tree，然后取平均。

这样子的话，得到的tree会更稳定一些。

这还只是data bagging. 我们还可以做feature bagging，就是排除某些特征，来构造tree。这样也会更好。

• Each tree can only use a random subset of features

• So the decision tree is forced to use all possible information to predict,

instead of relying on a few powerful features

Boosting

We may have lots of weak learners. We want to combine them together to get a strong learner?

That is we want to learn a fixed combination of $h_i$.

Adaboost

key idea: we can set the weight of each sample to control.

Constructing:

​ init: $D_1(i)=\frac{1}{m},\mathrm{\forall }i$

​ Given $D_T$ and $h_T$:

​ $D_{t+1}(i)=\frac{D_t(i)}{Z_t}{e}^{-{\alpha }_t}$, if $y_i=h_t(x_i)$ 答对了就减少权重

​ $D_{t+1}(i)=\frac{D_t(i)}{Z_t}{e}^{{\alpha }_t}$, if $y_i\ne h_t(x_i)$ 答错了就增大权重

​ $Z_t=\sum _i{D}_t(i)e^{-{\alpha }_i{y}_i{h}_t(x_i)}$. Normalization factor.

​ ${\alpha }_t=\frac{1}{2}\ln (\frac{1-{ϵ}_t}{{ϵ}_t})\ge 0$.

​ ${ϵ}_t$ 是错误率。

​ ${\alpha }_t$后面会讲为什么这么设定，也就是$h_t$的参数。错误率越小，权重越高，也很合理。

​ $H_t=\text{sign}(\sum _t{\alpha }_t{h}_t(x))$

Convergence Thm

$$\begin{gathered} {\gamma }_t=\frac{1}{2}-{ϵ}_t,\gamma =\underset{t}{max}{\gamma }_t \\ \text{Error}(H_{final})\le e^{-2{\gamma }^{2}T} \end{gathered}$$

Adaboost is adaptive: does not need to know $\gamma$ or $T$ as prior knowledge.

Convergence Proof

Step 1: unwrapping recurrence

$$D_T(i)=\frac{1}{m\prod _t{Z}_t}{e}^{-y_i\sum _t{\alpha }_t{h}_t(x_i)}$$

Let $f(x_i)=\sum _t{\alpha }_t{h}_t(x_i)$, it becomes:

$$D_T(i)=\frac{1}{m\prod _t{Z}_t}{e}^{-y_{i}f(x_i)}$$

Step 2: bound error of H

$$\begin{array}{}\text{(2)}& \text{Error}(H)& =\frac{1}{m}\sum _i\mathbb{1}[y_i\ne H(x_i)]\text{(3)}& & =\frac{1}{m}\sum _i\mathbb{1}[y_{i}f(x_i)\le 0](\mathrm{因}\mathrm{为}H(x)=\text{sign}(f(x)))\text{(4)}& & \le \frac{1}{m}\sum _i{e}^{-y_{i}f(x_i)}(\text{a soft version of the indicator function})\text{(5)}& & =(\prod _t{Z}_t)(\sum _i{D}_T(i))=\prod _t{Z}_t(\text{每一步都是归一化的，这个D的求和自然是1})\end{array}$$

Step 3: bound $Z_T$

$$\begin{array}{rl}{Z}_t& =\sum _i{D}_t(i)e^{-{\alpha }_i{y}_i{h}_t(x_i)}\\ & =\sum _{i:y_i\ne h_t(x_i)}{D}_t(i)e^{{\alpha }_i}+\sum _{i:y_i=h_t(x_i)}{D}_t(i)e^{-{\alpha }_i}\\ & ={ϵ}_t{e}^{{\alpha }_i}+(1-{ϵ}_t)e^{-{\alpha }_i}({ϵ}_t\mathrm{是}\mathrm{错}\mathrm{误}\mathrm{率})\end{array}$$

对$\alpha$ 求导，找到最小值${\alpha }_t=\frac{1}{2}\ln (\frac{1-{ϵ}_t}{{ϵ}_t})$, 所以$Z_t=2\sqrt{{ϵ}_t(1-{ϵ}_t)}$

因为${ϵ}_t=\frac{1}{2}-{\gamma }_t$

$$Z_t=2\sqrt{(\frac{1}{2}-{\gamma }_t)(\frac{1}{2}+{\gamma }_t)}=\sqrt{1-4{\gamma }_t^2}$$

$$\prod _t{Z}_t=\prod _t\sqrt{1-4{\gamma }_t^2}\le e^{-2\sum _t{\gamma }_t^2}(\mathrm{泰}\mathrm{勒}\mathrm{展}\mathrm{开})$$

From this Thm, we can know when $T\stackrel{}{\to }+\mathrm{\infty },Error(H)\stackrel{}{\to }0$. 但是这只是训练集的loss减小了，但是结合过多的weak learners容易过拟合。这就要讲接下来的这个部分 Margin-based analysis。

Margin-based analysis

Thm1: Let $S$ be a sample of $m$ samples chosen independently at random according to $D$. Assume that the base hypothesis space $H$ is finite, and let $\delta >0$. Then with probability at least $1-\delta$ over the random choice of the training set $S$, every weighted average function $f\in C$ satisfies the following bound for all $\theta >0$:

$$P_S[yf(x)\le \theta ]\le P_D[yf(x)\le \theta ]+O(\frac{1}{\sqrt{m}}(\frac{\log m\log |H|}{{\theta }^2}+\log (\frac{1}{\delta }){)}^{1/2})$$

Define $C_N$ to be the set of unweighted averages over $N$ elements from $H$

$$C_N=\{g:x\stackrel{}{\to }\sum _{i=1}^N{h}_i(x)|h_i\in H\}$$

We allow the same $h\in H$ to appear multiple times in the sum. This set will play the role of the approximating set in the proof.

Any majority vote hypothesis $f\in C$ can be associated with a distribution over $H$ as defined by the coefficient $a_h$. By choosing $N$ elements of $H$ independently at random according to this distribution we can generate an element of $C_N$. Using such a construction we map each $f\in C$ to a distribution $Q$ over $C_N$.

这边的思想还是很妙的。就是对于我的每一个函数f，我都构造一个函数集合的概率分布，这个函数集合的多个函数求均值等于f的值。

那么如何构造这个概率分布呢？$f=\sum _i{c}_i{h}_i$, 那么$h_i$被采样到的概率，就是$c_i$。 这样去采样可以得到一个概率分布，而且$C_N$中的函数还是有限的，这是我们最关键。

Our goal is to upper bound the generalization error of $f\in C$. For a fixed $g\in C_N$ and $𝜃>0$ we can separate this probability into two terms:

$$\begin{array}{rl}{P}_{x\sim D}[yf(x)\le 0]& \le P_{x\sim D}[yg(x)\le \frac{\theta }{2},yf(x)\le 0]+P_{x\sim D}[yg(x)>\frac{\theta }{2},yf(x)\le 0]\\ & \le P_{x\sim D}[yg(x)\le \frac{\theta }{2}]+P_{x\sim D}[yg(x)>\frac{\theta }{2}|yf(x)\le 0]\end{array}$$

As this inequality holds for any $g\in C_N$:

$$\begin{array}{rl}{P}_D[yf(x)\le 0]& \le P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}]+P_{x\sim D,g\sim Q}[yg(x)>\frac{\theta }{2}|yf(x)\le 0]\end{array}$$

We first look at the second term:

Since $f(x)=E_{g\sim Q}[g(x)]$, for a fixed sample $x\sim D$, chernoff hoeffding bound can get：

$$\begin{array}{rl}{P}_{g\sim Q}[yg(x)>\frac{\theta }{2}|yf(x)\le 0]& \le P_{g\sim Q}[yg(x)-yf(x)>\frac{\theta }{2}]\\ & =P_{g\sim Q}[yg(x)-E[yg(x)]>\frac{\theta }{2}]\\ & \le e^{-N{\theta }^2/8}g\text{是N个东西的平均}\end{array}$$

Then we come to the first term:

Change a little bit of the statement, with a probability $1-{\delta }_N$, we have:

$$P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}]\le P_{x\sim S,g\sim Q}[yg(x)\le \frac{\theta }{2}]+{ϵ}_N$$

现在我们其实想bound的是，在已知$P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}]$的情况下，求$P_{x\sim S,g\sim Q}[yg(x)>\frac{\theta }{2}]$的概率下界。这就有点像已知realizable assumption，求取样复杂度。

To upper bound the first term, we use the union bound.

For any signle term, $g$ and $\theta$, 假设:

$$P_{x\sim S,g\sim Q}[yg(x)>\frac{\theta }{2}]>P_{x\sim D,g\sim Q}[yg(x)>\frac{\theta }{2}]+{ϵ}_N$$

那么这个不等式成立的概率该怎么算呢？chernoff hoeffding bound

$$\begin{array}{rl}P[|\sum _{x\sim S}{P}_{x\sim S,g\sim Q}[yg(x)>\frac{\theta }{2}]-mE|>m{ϵ}_N]& \le e^{-2m{ϵ}_N^2}\end{array}$$

上面这个式子中的$E$是指$E=E[P_{x\sim D,g\sim Q}[yg(x)>\frac{\theta }{2}]]$
然后遍历所有的$|C_N|\le |H{|}^N$还有所有的$\theta$ 选择（$\theta$ 只能是$2i/N$ for $i=0,1,...,N$, 因为$C_N$ 的构造方式）这个概率就变成了$(N+1)|C_N|e^{-2m{ϵ}_N^2}$

接着再对这个$P[\cdot ]$ 的概率不等式事件做等效变换：

$$\begin{array}{r}{P}_{x\sim S,g\sim Q}[yg(x)>\frac{\theta }{2}]>P_{x\sim D,g\sim Q}[yg(x)>\frac{\theta }{2}]+{ϵ}_N\\ 1-P_{x\sim S,g\sim Q}[yg(x)\le \frac{\theta }{2}]>1-P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}]+{ϵ}_N\\ P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}]>P_{x\sim S,g\sim Q}[yg(x)\le \frac{\theta }{2}]+{ϵ}_N\end{array}$$

最后这个式子的概率，应该小于${\delta }_N$。 所以:

$$\begin{array}{r}{\delta }_N\ge (N+1)|C_N|e^{-2m{ϵ}_N^2}\\ \ln (\frac{{\delta }_N}{(N+1)|C_N|})\ge -2m{ϵ}_N^2\\ {ϵ}_N\ge \sqrt{-\frac{1}{2m}\ln (\frac{{\delta }_N}{(N+1)|C_N|})}\\ {ϵ}_N\ge \sqrt{\frac{1}{2m}\ln (\frac{(N+1)|C_N|}{{\delta }_N})}\end{array}$$

要保证最后这个不等是的话，由于$|C_N|\le |H{|}^N$， 我们只需要保证

$${ϵ}_N\ge \sqrt{\frac{1}{2m}\ln (\frac{(N+1)|H{|}^N}{{\delta }_N})}({ϵ}_N\mathrm{越}\mathrm{大}\mathrm{越}\mathrm{好})$$

类似前面这个式子$P_D[yf(x)\le 0]\le P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}]+P_{x\sim D,g\sim Q}[yg(x)>\frac{\theta }{2}|yf(x)\le 0]$

我们有在$S$ 上的一个式子

$$P_S[yg(x)\le \frac{\theta }{2}]\le P_{x\sim S,g\sim Q}[yf(x)\le \theta ]+P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}|yf(x)>\theta ]$$

By chernoff hoeffding bound:

$$P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}|yf(x)>\theta ]\le e^{-N{\theta }^2/8}$$

Combine them together, we can get for every $\theta >0$ and $N\ge 1$:

$$\begin{array}{rl}{P}_D[yf(x)\le 0]& \le P_{x\sim D,g\sim Q}[yg(x)\le \frac{\theta }{2}]+e^{-N{\theta }^2/8}\\ & \le P_{x\sim S,g\sim Q}[yg(x)\le \frac{\theta }{2}]+{ϵ}_N+e^{-N{\theta }^2/8}(\mathrm{这}\mathrm{一}\mathrm{步}\mathrm{是}\mathrm{有}\mathrm{一}\mathrm{个}1-\delta \mathrm{的}\mathrm{概}\mathrm{率}\mathrm{的})\\ & \le P_{x\sim S,g\sim Q}[yf(x)\le \theta ]+e^{-N{\theta }^2/8}+{ϵ}_N+e^{-N{\theta }^2/8}\end{array}$$

将${ϵ}_N$代入，但是因为我们求的是任意的N，所以失败概率是被所有N取值的失败概率bound住的，$\sum _N{\delta }_N=\delta ,{\delta }_N=\frac{\delta }{N(N+1)}$

最后再设置一下N的值，和$\theta$ 有关。证毕，好长的定理啊。

Modification on Adaboost

Thm2:

Suppose the base learning algorithm, when called by AdaBoost, generates hypotheses with weighted training errors ${ϵ}_1,\cdots ,{ϵ}_{𝑇}$. Then for any $𝜃$, we have that:

$$\begin{gathered} P_{x\sim S}[yf(x)\le \theta ]\le 2^T\prod _{t=1}^T\sqrt{{ϵ}_t^{1-\theta }(1-{ϵ}_t{)}^{1-\theta }} \\ (\text{Error}(H)\le \prod _{t}2\sqrt{{ϵ}_t(1-{ϵ}_t)}) \end{gathered}$$

Proof:

Note that if $yf(x)\le \theta$, then （$f(x)=\frac{\sum _t{\alpha }_t{h}_t(x_i)}{\sum _t{\alpha }_t}$ here）

$$y_i\sum _t{\alpha }_t{h}_t(x_i)\le \theta \sum _t{\alpha }_t$$

So:

$$\begin{array}{rl}{P}_{x\sim S}[yf(x)\le \theta ]& =E_{x\sim S}[\mathbb{1}[\theta \sum _t{\alpha }_t-y_i\sum _t{\alpha }_t{h}_t(x_i)\ge 0]]\\ & \le E_{x\sim S}[e^{\theta \sum _t{\alpha }_t-y_i\sum _t{\alpha }_t{h}_t(x_i)}]\text{soft version of indecator function}\\ & =e^{\theta \sum _t{\alpha }_t}\frac{1}{m}\sum _i{e}^{-y_i\sum _t{\alpha }_t{h}_t(x_i)}\\ & =e^{\theta \sum _t{\alpha }_t}\prod _t{Z}_t\sum _i{D}_t(i)\\ & =e^{\theta \sum _t{\alpha }_t}\prod _t{Z}_t\text{(其实就是因为不是0是}\theta \text{多了一项)}\end{array}$$

因为${\alpha }_t=\frac{1}{2}\ln (\frac{1-{ϵ}_t}{{ϵ}_t})$， 所以：

$$\begin{array}{rl}{e}^{{\alpha }_t}& =\sqrt{{ϵ}_t(1-{ϵ}_t)}\\ e^{\theta \sum _t{\alpha }_t}& =\prod _t\sqrt{{ϵ}_t^{\theta }(1-{ϵ}_t{)}^{\theta }}\\ e^{\theta \sum _t{\alpha }_t}\prod _t{Z}_t& =\prod _{t}2\sqrt{{ϵ}_t^{1-\theta }(1-{ϵ}_t{)}^{1+\theta }}\end{array}$$

这也是随着T指数级减小的。

Gradient Boosting

One drawback of adaboost: only cares about binary classification tasks.

What about regression tasks? We want to extend adaboost to regression tasks. Then we get gradient boosting.

Recall Adaboost is optimizing the following function:

• $\prod _t{Z}_t=\frac{1}{m}\sum _i\exp (-y_{i}f(x_i))=\frac{1}{m}\sum _i\exp (-y_i\sum _t{\alpha }_t{h}_t(x_i))$
• This will go to 0 as T goes up
• How does it optimize? It uses coordinate descent

我们的需求是，给定了一些weak learners，我们要学一个他们的权重。

求一个全导数做gradient descent太贵了，因为weak learner数量可能是非常多的。

那就每次先固定所有其他的$\alpha$ ，只学一个。

process:

• initially all ${\alpha }_i=0$
• In each iteration of adaboost, we pick a coordinate ${\alpha }_i$ and set it to make loss decrease most.
• It is like each time, we add a weak learning $h$ into $f$

因为我们每次是往$f$中加一个weak learner，所以，所以我们加入的新函数可以在$(x_i,y_i-f(x_i))$ 做优化。

How is it related to gradient descent?

假如我们定义要优化的loss是 $\mathcal{l}2$ loss:

$$\begin{array}{r}L=\sum _i\frac{(y_i-f(x_i){)}^2}{2}\\ \frac{\partial L}{\partial f(x_i)}=f(x_i)-y_i\end{array}$$

如果做gradient descent，$f_{t+1}(x_i)=f_t(x_i)-\eta \frac{\partial L}{\partial f(x_i)}=f_t(x_i)-\eta (f(x_i)-y_i)\to y_i$那么更新之后就是更接近$y$

但是实际上，我每次不是更新$\frac{\partial L}{\partial f(x_i)}$ 而是加一个${\alpha }_i{h}_i$. 效果也就只能勉强接近gradient descent

More generally, other losses are the same.