机器学习第2章: 优化

合集 - 机器学习(6)

1.机器学习第1章: 概述2024-01-22

2.机器学习第2章: 优化2024-01-19

3.机器学习第3章: 泛化2024-01-224.机器学习第4章: 监督学习2024-01-225.机器学习第5章: 无监督学习2024-01-226.机器学习第6章: 其他2024-01-22

收起

Chapter 2: Optimization

Acknowledgment: Most of the knowledge comes from Yuan Yang's course "Machine Learning".

What is optimization? Use algorithms to minimize a function. In machine learning, minimizing a loss function is very important.

GD: Gradient descent

To minimize a function $f(x)$, we use the following formula to update. (𝜂 is step size/learning rate)

$$x_{t+1}=x_t-\eta \mathrm{\nabla }f(x_t)$$

Smoothness assumption

(a) $\Vert {\mathrm{\nabla }}^{2}f(x)\Vert \le L$

(b)$\Vert \mathrm{\nabla }f(x)-\mathrm{\nabla }f(y)\Vert \le L\Vert x-y\Vert \stackrel{}{\to }⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),x-y⟩\le L\Vert x-y{\Vert }^2$

(c)$|f(y)-f(x)-⟨f(x),y-x⟩|\le \frac{L}{2}\Vert y-x{\Vert }^2$

这三者等价。证明如下：

(a) $\stackrel{}{\to }$ (b): 也是泰勒展开成积分的形式，因为我们是要利用二阶导的性质得到关于一阶导的性质。

$$\begin{array}{rl}& \Vert \mathrm{\nabla }f(w^{\prime })-\mathrm{\nabla }f(w)\Vert \\ =& \Vert {\int }_0^1{\mathrm{\nabla }}^{2}f(w+t(w^{\prime }-w))dt\ast (w^{\prime }-w)\Vert (\mathrm{泰}\mathrm{勒}\mathrm{展}\mathrm{开})\\ \le & \Vert {\int }_0^1{\mathrm{\nabla }}^{2}f(w+t(w^{\prime }-w))dt\Vert \Vert (w^{\prime }-w)\Vert (\mathrm{内}\mathrm{积}\mathrm{不}\mathrm{等}\mathrm{式})\\ \le & L\Vert w^{\prime }-w\Vert (\mathrm{条}\mathrm{件}a)\end{array}$$

(b) $\stackrel{}{\to }$ (c): 套路类似，泰勒展开，利用积分的形式实现降维

$$\begin{array}{rl}& |f(w^{\prime })-f(w)-⟨\mathrm{\nabla }f(w),w^{\prime }-w⟩|\\ =& |⟨{\int }_0^1\mathrm{\nabla }f(w+t(w^{\prime }-w))dt,w^{\prime }-w⟩-⟨\mathrm{\nabla }f(w),w^{\prime }-w⟩|(\mathrm{泰}\mathrm{勒}\mathrm{展}\mathrm{开})\\ =& |{\int }_0^1⟨\mathrm{\nabla }f(w+t(w^{\prime }-w))-\mathrm{\nabla }f(w),w^{\prime }-w⟩dt|\\ \le & {\int }_0^1\Vert \mathrm{\nabla }f(w+t(w^{\prime }-w))-\mathrm{\nabla }f(w)\Vert \Vert w^{\prime }-w\Vert dt(\mathrm{内}\mathrm{积}\mathrm{不}\mathrm{等}\mathrm{式})\\ \le & {\int }_0^{1}L\Vert t(w^{\prime }-w)\Vert \Vert w^{\prime }-w\Vert dt(\mathrm{条}\mathrm{件}b)\\ =& L\Vert w^{\prime }-w{\Vert }^2{\int }_0^{1}tdt=\frac{L}{2}\Vert w^{\prime }-w{\Vert }^2\end{array}$$

(c) $\stackrel{}{\to }$ (a):

(a)泰勒展开

$$\frac{1}{2}(w^{\prime }-w{)}^T{\mathrm{\nabla }}^{2}f(w+\theta (w^{\prime }-w))(w^{\prime }-w)\le \frac{L}{2}\Vert w^{\prime }-w{\Vert }^2$$

Assume $w^{\prime }=w+tv,\Vert v\Vert =1$

$$v^T{\mathrm{\nabla }}^{2}f(w+\theta tv)v\le L$$

Let $t\stackrel{}{\to }0$ （$w^{\prime }\stackrel{}{\to }w$), we can get $ v^T\nabla2 f(w)v \le L$.

$$\Vert {\mathrm{\nabla }}^{2}f(w)\Vert ={\lambda }_{max}({\mathrm{\nabla }}^{2}f(w))=\underset{\Vert v\Vert =1}{max}{v}^T{\mathrm{\nabla }}^{2}f(w)v\le L$$

Convexity assumption

(a) ${\mathrm{\nabla }}^{2}f(w)\ge 0$

(b) $f(w^{\prime })\ge f(w)+⟨\mathrm{\nabla }f(w),w^{\prime }-w⟩$

(c) $f(tx+((1-t)y)\le tf(x)+(1-t)f(y)$

GD convergence guarantee

Using these 2 assumptions and the update rule, we can get a GD guarantee.

Lemma: Combine L smooth and update rule, we can get

$$f(w_{i+1})-f(w_i)\le -\frac{\eta }{2}\Vert \mathrm{\nabla }f(w_i)\Vert$$

Proof: Use L smooth condition (c) and the update rule

$$\begin{array}{rl}f(w_{i+1})& \le f(w_i)+⟨\mathrm{\nabla }f(w_i),-\eta \mathrm{\nabla }f(w_i)⟩+\frac{L{\eta }^2}{2}\Vert \mathrm{\nabla }f(w_i){\Vert }^2\\ & =f(w_i)+\eta (1-\frac{L\eta }{2})\Vert \mathrm{\nabla }f(w_i){\Vert }^2\end{array}$$

set learning rate $\eta =\frac{1}{L}$, then we can get:

$$\frac{1}{2L}\Vert \mathrm{\nabla }f(w_i){\Vert }^2=\frac{\eta }{2}\Vert \mathrm{\nabla }f(w_i){\Vert }^2\le f(w_i)-f(w_{i+1})\le f(w_i)-f(w^{\ast })$$

begin with the lemma

$$\begin{array}{rl}& f(w_{i+1})\le f(w_i)-\frac{\eta }{2}\Vert \mathrm{\nabla }f(w_i)\Vert (\mathrm{\nabla }f(w_i)\text{没法telescoping})\\ =& f(w_i)-\frac{1}{2\eta }\Vert w_i-w_{i+1}{\Vert }^2(\text{update rule})\\ \le & f(w^{\ast })-⟨\mathrm{\nabla }f(w_i),w^{\ast }-w_i⟩-\frac{1}{2\eta }\Vert w_i-w_{i+1}{\Vert }^2(\text{convex})\\ =& f(w^{\ast })-\frac{1}{\eta }⟨w_i-w_{i+1},w^{\ast }-w_i⟩-\frac{1}{2\eta }\Vert w_i-w_{i+1}{\Vert }^2(\text{update rule})\\ =& f(w^{\ast })-\frac{1}{2\eta }(\Vert w_i-w_{i+1}{\Vert }^2-2⟨w_i-w_{i+1},w_i-w^{\ast }⟩)(\mathrm{这}\mathrm{里}\mathrm{就}\mathrm{比}\mathrm{较}\mathrm{明}\mathrm{显}\mathrm{了})\\ =& f(w^{\ast })-\frac{1}{2\eta }(\Vert w_i-w_{i+1}{\Vert }^2-2⟨w_i-w_{i+1},w_i-w^{\ast }⟩+\Vert w_i-w^{\ast }{\Vert }^2)+\frac{1}{2\eta }\Vert w_i-w^{\ast }{\Vert }^2\\ =& f(w^{\ast })+\frac{1}{2\eta }\Vert w_i-w^{\ast }{\Vert }^2-\frac{1}{2\eta }\Vert w_{i+1}-w^{\ast }{\Vert }^2\end{array}$$

然后开始telescoping。第一个不等号是因为non increasing

$$T(f(w_T)-f(w^{\ast }))\le \sum _{i=0}^{T-1}f(w_{i+1})-Tf(w^{\ast })\le \frac{\Vert w_0-w^{\ast }{\Vert }^2-\Vert w_T-w^{\ast }{\Vert }^2}{2\eta }\le \frac{\Vert w_0-w^{\ast }{\Vert }^2}{2\eta }$$

$\mu$-strongly convex assumption

(a) $\Vert {\mathrm{\nabla }}^{2}f(x)\Vert \ge \mu$

(b) $⟨\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y),x-y⟩\ge \mu \Vert x-y{\Vert }^2\stackrel{}{\to }\Vert \mathrm{\nabla }f(x)-\mathrm{\nabla }f(y)\Vert \ge \mu \Vert x-y\Vert$

(c) $f(y)-f(x)-⟨f(x),y-x⟩\ge \frac{\mu }{2}\Vert y-x{\Vert }^2$

和前面L - smooth类似，二阶导，导数，原函数的性质各一条。证明也类似，此处略过。

GD can get a linear convergence rate on an L-smooth $\mu$-strongly convex function.

Then let's move on to the proof of the question. Set learning rate $\eta =\frac{1}{L}$.

$$\begin{array}{rl}\Vert w_{T+1}-w^{\ast }{\Vert }^2& =\Vert w_T-\eta \mathrm{\nabla }f(w_T)-w^{\ast }{\Vert }^2\\ & =\Vert w_T-w^{\ast }{\Vert }^2-2\eta ⟨\mathrm{\nabla }f(w_T),w_T-w^{\ast }⟩+{\eta }^2\Vert \mathrm{\nabla }f(w_T)\Vert \\ & \le \Vert w_T-w^{\ast }{\Vert }^2+2\eta ⟨\mathrm{\nabla }f(w_T),w^{\ast }-w_T⟩+2\eta (f(w_T)-f(w^{\ast }))(\text{lemma})\\ & \le \Vert w_T-w^{\ast }{\Vert }^2+2\eta (f(w^{\ast })-f(w_T)-\frac{\mu }{2}\Vert w_T-w^{\ast }{\Vert }^2)+2\eta (f(w_T)-f(w^{\ast }))(\text{strongly convex})\\ & =(1-\frac{\mu }{L})\Vert w_T-w^{\ast }{\Vert }^2\end{array}$$

$$\begin{array}{rl}\Vert w_T-w^{\ast }{\Vert }^2& \le (1-\frac{\mu }{L}{)}^T\Vert w_0-w^{\ast }{\Vert }^2\text{(telescoping)}\\ f(w_T)-f(w^{\ast })& \le ⟨\mathrm{\nabla }f(w^{\ast }),w_T-w\ast ⟩+\frac{L}{2}\Vert w_T-w^{\ast }{\Vert }^2\\ & =\frac{L}{2}\Vert w_T-w^{\ast }{\Vert }^2(\mathrm{因}\mathrm{为}\mathrm{\nabla }f(w^{\ast })=0)\\ & \le \frac{L}{2}(1-\frac{\mu }{L}{)}^T\Vert w_0-w^{\ast }{\Vert }^2\end{array}$$

SGD: Stochastic gradient descent

but gradient descent has 2 limitations:

• Computing full gradient is slow for big data
• Could get stuck at stationary points (stationary point & local minimum)

So we add some randomness to the GD algorithm

$$\begin{array}{rl}{x}_{t+1}& =x_t-\eta \mathrm{\nabla }{G}_t\\ \frac{1}{\eta }E[x_t-x_{t+1}]=E[G_t]& =\mathrm{\nabla }L(w_t)=\frac{1}{N}\sum _i\mathrm{\nabla }L(w,x_i,y_i)\\ V(G_t)& =E[\Vert G_t{\Vert }^2]-\Vert \mathrm{\nabla }L(w_t){\Vert }^2(\text{方差的定义})\\ G_t& =\frac{1}{|S|}\sum _{i\in S}\mathrm{\nabla }L(w,x_i,y_i)\text{for a randomly selected subset S}\end{array}$$

$|S|$ is called the mini-batch size, or simply batch size.

• If $|S|$ is too small, say, $|S|$=1, $G_t$ still has the right expectation, but with very large variance
• If $|S|$ is too big, say, $|S|$=n, $G_t$ is slow to compute
• Small but not too small $|S|$ has at least two benefits
  • Has some noise to help escape saddle points or bad local minimum
  • Helps to get the right mini-batch statistics for batch normalization (不懂)

If we consider update rule of SGD as $x_{t+1}=x_t-\eta (\mathrm{\nabla }L(w_t)+{\xi }_t)$. Full batch + white noise is also SGD.

We run SGD with bounded variance $Var(G_t)\le {\sigma }^2$

convergence analysis

because the update rule is different, so we can not begin with the lemma before. We need to propose a new lemma.

Lemma: combine L smooth and the update rule, then we can get

$$E[f(w_{i+1})]\le f(w_i)-\frac{\eta }{2}\Vert \mathrm{\nabla }f(w_i){\Vert }^2+\frac{\eta {\sigma }^2}{2}$$

Proof: Use L smooth condition (c) and the update rule

$$\begin{array}{rl}& E[f(w_{i+1})]\le E[f(w_i)+⟨\mathrm{\nabla }f(w_i),w_{i+1}-w_i⟩+\frac{L}{2}\Vert w_{i+1}-w_i{\Vert }^2]\\ =& f(w_i)+⟨\mathrm{\nabla }f(w_i),E[(w_{i+1}-w_i)]⟩+\frac{L}{2}E[\Vert w_{i+1}-w_i{\Vert }^2](\mathrm{期}\mathrm{望}\mathrm{的}\mathrm{线}\mathrm{性}\mathrm{性}\mathrm{质})\\ =& f(w_i)-\eta \Vert \mathrm{\nabla }f(w_i){\Vert }^2+\frac{\eta }{2}E[\Vert G_i{\Vert }^2](\text{update rule})\\ =& f(w_i)-\eta \Vert \mathrm{\nabla }f(w_i){\Vert }^2+\frac{\eta }{2}(V(G_i)+\Vert \mathrm{\nabla }f(w_i){\Vert }^2)(\text{方差的定义})\\ \le & f(w_i)-\frac{\eta }{2}\Vert \mathrm{\nabla }f(w_i){\Vert }^2+\frac{\eta {\sigma }^2}{2}(\text{bounded方差的条件})\end{array}$$

SGD convergence guarantee

begin with $E[f(w_{i+1})]\le f(w_i)-\eta \Vert \mathrm{\nabla }f(w_i){\Vert }^2+\frac{\eta }{2}E[\Vert G_i{\Vert }^2]$ in the proof of lemma

$$\begin{array}{rl}E[f(w_{i+1})]\le & f(w_i)-\eta \Vert \mathrm{\nabla }f(w_i){\Vert }^2+\frac{\eta }{2}E[\Vert G_i{\Vert }^2]\\ \le & f(w_i)-\frac{\eta }{2}E[\Vert G_i{\Vert }^2]+\eta {\sigma }^2(\text{方差的定义})\\ \le & f(w^{\ast })-⟨\mathrm{\nabla }f(w_i),w^{\ast }-w_i⟩-\frac{\eta }{2}E[\Vert G_i{\Vert }^2]+\eta {\sigma }^2(\text{convex})\\ =& f(w^{\ast })-⟨\frac{1}{\eta }E[(w_i-w_{i+1})],w^{\ast }-w_i⟩-\frac{1}{2\eta }E[\Vert w_i-w_{i+1}{\Vert }^2]+\eta {\sigma }^2(\text{update rule})\\ =& f(w^{\ast })-\frac{1}{2\eta }E(\Vert w_i-w_{i+1}{\Vert }^2-2⟨w_i-w_{i+1},w_i-w^{\ast }⟩)+\eta {\sigma }^2(\mathrm{这}\mathrm{里}\mathrm{就}\mathrm{比}\mathrm{较}\mathrm{明}\mathrm{显}\mathrm{了})\\ =& f(w^{\ast })-\frac{1}{2\eta }E(\Vert w_i-w_{i+1}{\Vert }^2-2⟨w_i-w_{i+1},w_i-w^{\ast }⟩+\Vert w_i-w^{\ast }{\Vert }^2-\Vert w_i-w^{\ast }{\Vert }^2)+\eta {\sigma }^2\\ =& f(w^{\ast })+\frac{1}{2\eta }E(\Vert w_i-w^{\ast }{\Vert }^2-\Vert w_{i+1}-w^{\ast }{\Vert }^2)+\eta {\sigma }^2\end{array}$$

然后开始telescoping

$$\begin{array}{rl}\sum _{i=0}^{T-1}E[f(w_{i+1})]-Tf(w^{\ast })& \le \frac{1}{2\eta }E(\Vert w_0-w^{\ast }{\Vert }^2-\Vert w_T-w^{\ast }{\Vert }^2)+T\eta {\sigma }^2\le \frac{1}{2\eta }\Vert w_0-w^{\ast }{\Vert }^2+T\eta {\sigma }^2\\ E[f(\overline{w})]-f(w^{\ast })\le E[\frac{1}{T}\sum _{i=0}^{T-1}f(w_{i+1})]-f(w^{\ast })& \le \frac{\Vert w_0-w^{\ast }{\Vert }^2}{2\eta T}+\eta {\sigma }^2(\text{琴生不等式})\end{array}$$

convergence rate $\frac{1}{\sqrt{T}}$

proof: set $\eta =\frac{ϵ}{2{\sigma }^2}$

$$E[f(\overline{w})]-f(w^{\ast })\le \frac{\Vert w_0-w^{\ast }{\Vert }^2{\sigma }^2}{ϵT}+\frac{ϵ}{2}(=ϵ)$$

So $T=\frac{2\Vert w_0-w^{\ast }{\Vert }^2{\sigma }^2}{{ϵ}^2}$. so convergence rate is $\frac{1}{\sqrt{T}}$

Thinking: It seems SGD is much slower than GD, why do we still use it?

• Noise 可能没有那么大
• Sometimes noise helps, especially in non-convex settings (we will see it later)
• $ϵ$ 不需要到那么小: We do not care about training loss, but care about population loss. Even with 𝜖 = 0.1, it does not guarantee the population loss will be better compared with 𝜖 =0.1
• The variance of the noise can be reduced gradually along the process. (therefore as fast as GD)

If the variance $\sigma$ is the problem, can we make it smaller to have a faster convergence rate?

The answer is yes, we could reduce the variance and get $\frac{1}{𝑇}$ rate for convex function, and linear convergence for strongly convex function.

Unfortunately, it seems to only work for convex function, not useful for deep learning (yet).

There are many such algorithms, i.e., SAG, SVRG, SAGA, etc.

SVRG

assumption: L-smooth and $\mu$-strongly convex

Update rule

For $s=1,2,...,T,...$
$\overline{w}={\overline{w}}_{s-1}$ init the start point of this step

​ $\mathrm{\nabla }f(\overline{w})=\frac{1}{N}\sum _{i=1}^N\mathrm{\nabla }{l}_i(\overline{w})$ (calucalte full gradient of the start point)

​ $w_0=\overline{w}$ init the start point of the micro for loop below

​ For t = 1, 2, ..., m

​ Random pick $i\in [N]$ 随机选取一条数据的$l_i$计算gradient

​ $w_t=w_{t-1}-\eta (\mathrm{\nabla }{l}_i(w_{t-1})-\mathrm{\nabla }{l}_i(\overline{w})+\mathrm{\nabla }f(\overline{w}))$ 用single gradient来更新

​ ${\overline{w}}_s=w_m$ 小循环结束之后，更新终点作为下一step的起点

备注，在下面证明的时候，$\overline{w_s}=w_{rand(t)}$。但是in practical 是上面那种做法

thinking

We compute the full gradient for every step. So although a full gradient is expensive, the amortized cost is small.

We assume $\mathrm{\nabla }f(w_t)-\mathrm{\nabla }f(\overline{w})\approx \mathrm{\nabla }{l}_i(w_t)-\mathrm{\nabla }{l}_i(\overline{w})$

Convergence analysis

先研究小循环的一步，会造成什么影响

$$\begin{array}{rl}E\Vert w_t-w^{\ast }{\Vert }^2& =E\Vert w_{t-1}-\eta u_t-w^{\ast }{\Vert }^2(\text{update rule})\\ & =\Vert w_{t-1}-w^{\ast }{\Vert }^2-2⟨w_{t-1}-w^{\ast },\eta E[u_t]⟩+E[\Vert u_t{\Vert }^2]\end{array}$$

小循环中的 $i$ 是随机的，那么要去掉这个随机，那就是用$E$。加上$E$之后我们操作空间就大了

$E[u_t]=E[(\mathrm{\nabla }{l}_i(w_{t-1})-\mathrm{\nabla }{l}_i(\overline{w})+\mathrm{\nabla }f(\overline{w}))]=E[\mathrm{\nabla }{l}_i(w_{t-1})]-E[\mathrm{\nabla }{l}_i(\overline{w})]+\mathrm{\nabla }f(\overline{w})=\mathrm{\nabla }f(w_{t-1})$

至于$E[\Vert u_t{\Vert }^2]$ 单独算一下，目标: 找到一个不含E的上界

$$\begin{array}{rl}E[\Vert u_t{\Vert }^2]& =E[\Vert \mathrm{\nabla }{l}_i(w_{t-1})-\mathrm{\nabla }{l}_i(\overline{w})+\mathrm{\nabla }f(\overline{w}){\Vert }^2]\\ & \le 2E[\Vert \mathrm{\nabla }{l}_i(w_{t-1})-\mathrm{\nabla }{l}_i(w^{\ast }){\Vert }^2]+2E[\Vert \mathrm{\nabla }{l}_i(\overline{w})-\mathrm{\nabla }{l}_i(w^{\ast })-\mathrm{\nabla }f(\overline{w}){\Vert }^2](\mathrm{引}\mathrm{入}\mathrm{了}{w}^{\ast }，\mathrm{均}\mathrm{值}\mathrm{不}\mathrm{等}\mathrm{式})\\ & =2E[\Vert \mathrm{\nabla }{l}_i(w_{t-1})-\mathrm{\nabla }{l}_i(w^{\ast }){\Vert }^2]+2E[\Vert \mathrm{\nabla }{l}_i(\overline{w})-\mathrm{\nabla }{l}_i(w^{\ast })-E[\mathrm{\nabla }{l}_i(\overline{w})]{\Vert }^2]\\ & (\mathrm{因}\mathrm{为}\mathrm{其}\mathrm{他}\mathrm{都}\mathrm{是}\text{single gradient},\mathrm{所}\mathrm{以}\mathrm{将}\mathrm{这}\mathrm{个}\text{full gradient}\mathrm{转}\mathrm{化}\mathrm{为}\text{single gradient}\mathrm{或}\mathrm{者}\mathrm{想}\mathrm{办}\mathrm{法}\mathrm{去}\mathrm{掉})\\ & =2E[\Vert \mathrm{\nabla }{l}_i(w_{t-1})-\mathrm{\nabla }{l}_i(w^{\ast }){\Vert }^2]+2E[\Vert \mathrm{\nabla }{l}_i(\overline{w})-\mathrm{\nabla }{l}_i(w^{\ast })-E[\mathrm{\nabla }{l}_i(\overline{w})-\mathrm{\nabla }{l}_i(w^{\ast })]{\Vert }^2]\\ & (E[\mathrm{\nabla }{l}_i(w^{\ast })]=\mathrm{\nabla }f(w^{\ast })=0\mathrm{所}\mathrm{以}\mathrm{可}\mathrm{以}\mathrm{随}\mathrm{意}\mathrm{添}\mathrm{项})\\ & \le 2E[\Vert \mathrm{\nabla }{l}_i(w_{t-1})-\mathrm{\nabla }{l}_i(w^{\ast }){\Vert }^2]+2E[\Vert \mathrm{\nabla }{l}_i(\overline{w})-\mathrm{\nabla }{l}_i(w^{\ast }){\Vert }^2]\\ & (V(X)=E[\Vert X-EX{\Vert }^2]=E[X^2]-(EX{)}^2\le E[X^2],\mathrm{到}\mathrm{此}\mathrm{为}\mathrm{止}，\mathrm{总}\mathrm{算}\mathrm{去}\mathrm{掉}\mathrm{了}\text{full gradient}\mathrm{的}\mathrm{那}\mathrm{一}\mathrm{项})\\ & \le 4L(f(w_{t-1})-f(w^{\ast })+f(\overline{w})-f(w^{\ast }))\end{array}$$

(最后一步使用的结论$E[\Vert \mathrm{\nabla }{l}_i(w)-\mathrm{\nabla }{l}_i(w^{\ast })\Vert ]\le 2L(f(w)-f(w^{\ast }))$，见sys博客[机器学习] 2. 随机方差缩减梯度下降 SVRG - shiys22 - 博客园 (cnblogs.com)，证这个结论需要L-smooth)

以上的放缩都没有用到smooth和convex的条件。

现在回头代入之前的式子

$$\begin{array}{rl}E\Vert w_t-w^{\ast }{\Vert }^2& =\Vert w_{t-1}-w^{\ast }{\Vert }^2-2⟨w_{t-1}-w^{\ast },\eta E[u_t]⟩+{\eta }^{2}E[\Vert u_t{\Vert }^2]\\ & \le \Vert w_{t-1}-w^{\ast }{\Vert }^2-2\eta ⟨w_{t-1}-w^{\ast },\mathrm{\nabla }f(w_{t-1})⟩+4{\eta }^{2}L(f(w_{t-1})-f(w^{\ast })+f(\overline{w})-f(w^{\ast }))\\ & \le \Vert w_{t-1}-w^{\ast }{\Vert }^2-2\eta (f(w_{t-1})-f(w^{\ast }))+4{\eta }^{2}L(f(w_{t-1})-f(w^{\ast })+f(\overline{w})-f(w^{\ast }))\\ & (\mathrm{没}\mathrm{法}\mathrm{配}\mathrm{方}\mathrm{的}\mathrm{内}\mathrm{积}\mathrm{都}\mathrm{得}\mathrm{想}\mathrm{办}\mathrm{法}\mathrm{消}\mathrm{灭},\text{convex})\\ & =\Vert w_{t-1}-w^{\ast }{\Vert }^2+2\eta (2\eta L-1)(f(w_{t-1})-f(w^{\ast }))+4{\eta }^{2}L(f(\overline{w})-f(w^{\ast }))\\ 2\eta (1-2\eta L)(f(w_{t-1})-f(w^{\ast }))& \le 4{\eta }^{2}L(f(\overline{w})-f(w^{\ast }))+\Vert w_{t-1}-w^{\ast }{\Vert }^2-E\Vert w_t-w^{\ast }{\Vert }^2\end{array}$$

That is $E\Vert w_t-w^{\ast }{\Vert }^2-E[\Vert w_{t-1}-w^{\ast }{\Vert }^2]\le \text{xxx}$，显然可以telescoping，当然是对于这一整个小循环来telescoping

在这里会有一些添加$E$ 的操作，我简单说一下。

前面的证明考虑的是，小循环中的一步$w_{t-1}\stackrel{}{\to }{w}_t$，所以前面$\Vert w_{t-1}-w^{\ast }{\Vert }^2$没有加$E$。但是现在telescoping考虑整个小循环，就要加上$E$

但是对于一整个小循环来说$f(\overline{w})$是一开始init的时候就固定好了的。

左侧累加

$$\sum _{t=1}^{m}f(w_{t-1})-f(w^{\ast })=m(E[f({\overline{w}}_s)]-f(w^{\ast }))=mE[f({\overline{w}}_s)-f(w^{\ast })]$$

右侧累加

$$\begin{array}{rl}& \sum _{t=1}^{m}4{\eta }^{2}L(f(\overline{w})-f(w^{\ast }))+E\Vert w_{t-1}-w^{\ast }{\Vert }^2-E\Vert w_t-w^{\ast }{\Vert }^2\\ =& 4{\eta }^{2}Lm(f(\overline{w})-f(w^{\ast }))+E\Vert w_0-w^{\ast }{\Vert }^2-E\Vert w_m-w^{\ast }{\Vert }^2\\ \le & 4{\eta }^{2}Lm(f(\overline{w})-f(w^{\ast }))+\Vert w_0-w^{\ast }{\Vert }^2\\ =& 4{\eta }^{2}Lm(f(\overline{w})-f(w^{\ast }))+\Vert \overline{w}-w^{\ast }{\Vert }^2(w_0=\overline{w})\\ \le & 4{\eta }^{2}Lm(f(\overline{w})-f(w^{\ast }))+\frac{2}{\mu }(f(\overline{w})-f(w^{\ast }))(\text{strongly convex})\\ =& (4{\eta }^{2}Lm+\frac{2}{\mu })(f(\overline{w})-f(w^{\ast }))\end{array}$$

接下来要对大循环的每一次从$\overline{w}\stackrel{}{\to }\overline{w_s}$来telescoping， 所以$f(\overline{w})$要加上E:

$$\begin{array}{r}2\eta (1-2\eta L)mE[f({\overline{w}}_s)-f(w^{\ast })]\le (4{\eta }^{2}Lm+\frac{2}{\mu })E[f(\overline{w})-f(w^{\ast })]\end{array}$$

This is linear convergence. faster than GD if $\frac{L}{\mu }$(condition number) is large.

Linear coupling of GD & MD

What do we find in GD? $f(w_t)-f(w_{t+1})\ge \frac{1}{2L}\Vert \mathrm{\nabla }f(w_t){\Vert }^2$. Therefore, if the gradient is big, GD makes progress. What if the gradient is small?

We want to have another optimization method to make use of a small gradient and still converge to the optimal point.

$$\mathrm{\forall }x,f(x)\ge f(x_0)+⟨\mathrm{\nabla }f(x_0),x-x_0⟩(\text{convex})$$

If we have a sequence of queried points, then:

$$\begin{array}{rl}\mathrm{\forall }x,f(x)& \ge \frac{1}{k}\sum _{i=0}^{t-1}(f(x_i)+⟨\mathrm{\nabla }f(x_i),x-x_i⟩)\\ & \ge f(\overline{x})+\frac{1}{k}\sum _{i=0}^{t-1}⟨\mathrm{\nabla }f(x_i),x-x_i⟩(\text{convex})\\ f(\overline{x})-f(x^{\ast })& \le \frac{1}{k}\sum _{i=0}^{t-1}⟨\mathrm{\nabla }f(x_i),x_i-x^{\ast }⟩\end{array}$$

不等号右边的这个东西叫做regret。我们可以观察到，gradient越小，regret越小，这个bound就越紧，达成了我们的目的: make use of small gradient and still convergent to the optimal point.

那这种优化方法也很显然了，就是每次选的点，都得让regret变小。

MD: mirror descent

作为一阶优化问题，我们还是只能沿着导数的方向去找点，但是gradient的超平面会延伸到负无穷，这肯定是我们不希望的，我们希望的是走一小步。GD中我们是如何解决这个问题呢？

$$\begin{array}{rl}{x}_{t+1}& =\underset{x}{\arg min}(f(x_t)+⟨\mathrm{\nabla }f(x_t),x-x_t⟩+\frac{1}{2\eta }\Vert x-x_t{\Vert }^2)(\mathrm{超}\mathrm{平}\mathrm{面}+\mathrm{正}\mathrm{则}\mathrm{项})\\ & =\underset{x}{\arg min}(\eta ⟨\mathrm{\nabla }f(x_t),x-x_t⟩+\frac{1}{2}\Vert x-x_t{\Vert }^2)\end{array}$$

引入正则项$\frac{1}{2}\Vert x-x_t{\Vert }^2$, 解刚好就是$x_{t+1}=x_t-\eta \mathrm{\nabla }f(x_t)$

现在考虑对这个正则项一般化，于是就提出了Bregman divergence:

$$V_x(y)=w(y)-⟨\mathrm{\nabla }w(x),y-x⟩-w(x)$$

$w(x)$ is a 1-strongly convex function. So $V_x(y)\ge \frac{1}{2}\Vert y-x{\Vert }^2$. (Assume $w(x)$ to be the simplest 1-strongly convex function $\frac{x^2}{2}$， 正则项刚好就是GD的$\frac{1}{2}\Vert x-x_t{\Vert }^2$)

The update rule of MD is:

$$x_{k+1}=\underset{x}{\arg min}(\alpha ⟨\mathrm{\nabla }f(x_k),x-x_k⟩+V_{x_k}(x))$$

convergence

Lemma: triangle equality

$$\begin{array}{rl}⟨-\mathrm{\nabla }{V}_{x_1}(x_2),x_2-x_3⟩& =⟨\mathrm{\nabla }w(x_1)-\mathrm{\nabla }(x_2),x_2-x_3⟩\\ & =-⟨\mathrm{\nabla }w(x_1),x_3-x_1⟩+⟨\mathrm{\nabla }w(x_1),x_2-x_1⟩+⟨\mathrm{\nabla }w(x_2),x_3-x_2⟩\\ & (\mathrm{这}\mathrm{一}\mathrm{步}\mathrm{是}\mathrm{比}\mathrm{较}\mathrm{明}\mathrm{显}\mathrm{的}，\mathrm{因}\mathrm{为}\mathrm{要}\mathrm{凑}\mathrm{出}{V}_x(x^{\prime })\mathrm{的}\mathrm{这}\mathrm{种}\mathrm{形}\mathrm{式}，\mathrm{那}\mathrm{就}\mathrm{是}\mathrm{要}\mathrm{先}\mathrm{把}\mathrm{内}\mathrm{积}\mathrm{变}\mathrm{成}⟨\mathrm{\nabla }w(x),x^{\prime }-x⟩\mathrm{的}\mathrm{形}\mathrm{式})\\ & =(w(x_3)-⟨\mathrm{\nabla }w(x_1),x_3-x_1⟩-w(x_1))\\ & -(w(x_2)-⟨\mathrm{\nabla }w(x_1),x_2-x_1⟩-w(x_1))\\ & -(w(x_3)+⟨\mathrm{\nabla }w(x_2),x_3-x_2⟩-w(x_2))(w(x)\mathrm{添}\mathrm{加}\mathrm{的}\mathrm{项}\mathrm{刚}\mathrm{好}\mathrm{全}\mathrm{部}\mathrm{抵}\mathrm{消})\\ & =V_{x_1}(x_3)-V_{x_1}(x_2)-V_{x_2}(x_3)\end{array}$$

$$\begin{array}{rl}\alpha (f(x_k)-f(x^{\ast }))& \le \alpha ⟨\mathrm{\nabla }f(x_k),x_k-x^{\ast }⟩(\text{convex})\\ & =⟨\alpha \mathrm{\nabla }f(x_k),x_k-x_{k+1}⟩+⟨\alpha \mathrm{\nabla }f(x_k),x_{k+1}-x^{\ast }⟩(\mathrm{添}\mathrm{项}，\mathrm{因}\mathrm{为}\mathrm{特}\mathrm{殊}\mathrm{点}\mathrm{就}\mathrm{这}\mathrm{三}\mathrm{个})\\ & =⟨\alpha \mathrm{\nabla }f(x_k),x_k-x_{k+1}⟩+⟨-\mathrm{\nabla }{V}_{x_k}(x_{k+1}),x_{k+1}-x^{\ast }⟩\\ & (\mathrm{因}\mathrm{为}{x}_{k+1}\mathrm{是}(\alpha ⟨\mathrm{\nabla }f(x_k),x-x_k⟩+V_{x_k}(x))\mathrm{的}\mathrm{极}\mathrm{小}\mathrm{值}\mathrm{点}，\mathrm{所}\mathrm{以}\mathrm{求}\mathrm{导}\mathrm{为}0)\\ & =⟨\alpha \mathrm{\nabla }f(x_k),x_k-x_{k+1}⟩+V_{x_k}(x^{\ast })-V_{x_{k+1}}(x^{\ast })-V_{x_k}(x_{k+1})(\text{Lemma: triangle equality})\\ & =⟨\alpha \mathrm{\nabla }f(x_k),x_k-x_{k+1}⟩-V_{x_k}(x_{k+1})+V_{x_k}(x^{\ast })-V_{x_{k+1}}(x^{\ast })(\mathrm{把}\mathrm{相}\mathrm{同}\mathrm{变}\mathrm{量}\mathrm{的}\mathrm{放}\mathrm{一}\mathrm{起})\\ & \le ⟨\alpha \mathrm{\nabla }f(x_k),x_k-x_{k+1}⟩-\frac{1}{2}\Vert x_k-x_{k+1}{\Vert }^2+V_{x_k}(x^{\ast })-V_{x_{k+1}}(x^{\ast })(\text{1-strongly convex})\\ & \le \frac{{\alpha }^2}{2}\Vert \mathrm{\nabla }f(x_k){\Vert }^2+V_{x_k}(x^{\ast })-V_{x_{k+1}}(x^{\ast })\text{(配方，内积优先考虑配方，其次考虑convex)}\end{array}$$

后面这两项可以telescoping

$$\begin{array}{rl}\alpha T(f(\overline{x})-f(x^{\ast }))\le \alpha (\sum _{k=0}^{T-1}f(x_k)-Tf(x^{\ast }))& \le \frac{{\alpha }^2}{2}\sum _{k=0}^{T-1}\Vert \mathrm{\nabla }f(x_k){\Vert }^2+V_{x_0}(x^{\ast })-V_{x_T}(x^{\ast })\le \frac{{\alpha }^2}{2}\sum _{k=0}^{T-1}\Vert \mathrm{\nabla }f(x_k){\Vert }^2+V_{x_0}(x^{\ast })\\ f(\overline{x})-f(x^{\ast })& \le \frac{\alpha {\rho }^2}{2}+\frac{V_{x_0}(x^{\ast })}{\alpha T}(\rho \text{-smooth})\\ f(\overline{x})-f(x^{\ast })& \le \frac{\alpha {\rho }^2}{2}+\frac{\theta }{\alpha T}(\text{bound})\end{array}$$

Let $\alpha =\frac{\sqrt{2\theta }}{\rho \sqrt{T}}$

$$\begin{array}{rl}f(\overline{x})-f(x^{\ast })& \le \frac{\rho \sqrt{2\theta }}{\sqrt{T}}=ϵ\\ T& =\mathrm{\Omega }(\frac{{\rho }^2}{{ϵ}^2})\end{array}$$

所以MD是$\frac{1}{\sqrt{T}}$的收敛速度

Linear coupling

Can we combine GD and MD?

Wishful thinking, assume $\Vert \mathrm{\nabla }f(x)\Vert$ is either always larger than a threshold $\rho$ or smaller than this $\rho$.

Consider we need to update form $f(x_0)-f(x^{\ast })\le 2ϵ$ to $f(x_T)-f(x^{\ast })\le ϵ$.

• Run MD, it needs $\mathrm{\Omega }({\rho }^2/{ϵ}^2)$
• Run GD, it decreases by $\Vert \mathrm{\nabla }f{\Vert }^2/2L\ge {\rho }^2/2L$ in each step. So it needs $2Lϵ/{\rho }^2$

Combine these two, we know $T\ge \mathrm{\Omega }(max\{{\rho }^2/{ϵ}^2,2Lϵ/{\rho }^2\})$. Tunning ${\rho }^2=ϵ\sqrt{Lϵ}$ . we get $T\ge \mathrm{\Omega }(\sqrt{L/ϵ})$. Then this gets convergence rate $\frac{1}{T^2}$.

But this is an ideal situation, we rarely handle this kind of function. For an arbitrary function, if GD and MD go to 2 different points, how shall we choose?

This is why we use linear coupling: In every iteration, we run GD and MD simultaneously to get $y_k$ and $z_k$ respectively.

$$\begin{array}{rl}{x}_0& =y_0=z_0\\ y_k& =GD(x_k)\\ z_k& =Mir{r}_{z_{k-1}}(\alpha \mathrm{\nabla }f(x_k))=\arg \underset{y}{min}{V}_{z_k}(y)+⟨\alpha \mathrm{\nabla }f(x_k),y-z_k⟩\\ x_{k+1}& =\tau z_k+(1-\tau )y_k\end{array}$$

$\alpha$ and $\tau$ are hyperparameters to be decided later.

$$\begin{array}{rl}\alpha (f(x_{k+1})-f(x^{\ast }))\le & \alpha ⟨\mathrm{\nabla }f(x_{k+1}),x_{k+1}-x^{\ast }⟩(\text{convex})\\ =& \alpha ⟨\mathrm{\nabla }f(x_{k+1}),x_{k+1}-z_k⟩+\alpha ⟨\mathrm{\nabla }f(x_{k+1}),z_k-x^{\ast }⟩\\ & \text{(拆添项，极值点导数为0，然后三角不等式，再用一下strongly convex，配方一下)}\\ \le & \text{...}+(\frac{{\alpha }^2}{2}\Vert \mathrm{\nabla }f(x_{k+1}){\Vert }^2+V_{z_k}(x^{\ast })-V_{z_{k+1}}(x^{\ast }))(\text{MD})\\ \le & \text{...}+({\alpha }^{2}L(f(x_{k+1})-f(y_{k+1}))+V_{z_k}(x^{\ast })-V_{z_{k+1}}(x^{\ast }))(\text{GD})\\ =& \frac{(1-\tau )\alpha }{\tau }⟨\mathrm{\nabla }f(x_{k+1}),y_k-x_{k+1}⟩+(\text{...})(\text{linear coupling})\\ \le & \frac{(1-\tau )\alpha }{\tau }(f(y_k)-f(x_{k+1}))+(\text{...})(\text{convex})\end{array}$$

We can choose $\tau$ s.t. $\frac{1-\tau }{\tau }=\alpha L$, then: ($\tau$ has been fixed, then we only has one freedom $\alpha$)

$$\begin{array}{rl}\alpha (f(x_{k+1})-f(x^{\ast }))\le & {\alpha }^{2}L(f(y_k)-f(x_{k+1}))+({\alpha }^{2}L(f(x_{k+1})-f(y_{k+1}))+V_{z_k}(x^{\ast })-V_{z_{k+1}}(x^{\ast }))\\ \le & {\alpha }^{2}L(f(y_k)-f(y_{k+1}))+V_{z_k}(x^{\ast })-V_{z_{k+1}}(x^{\ast })\end{array}$$

Then this can do telescoping!

$$\begin{array}{rl}\alpha T(f(\overline{x})-f(x^{\ast }))\le & \alpha \sum _{k=0}^{T-1}(f(x_k)-f(x^{\ast }))\text{(琴生不等式)}\\ \le & {\alpha }^{2}L(f(y_0)-f(y_T))+V_{z_0}(x^{\ast })-V_{z_T}(x^{\ast })\\ \le & {\alpha }^{2}Ld+\theta \\ f(\overline{x})-f(x^{\ast })\le & \frac{{\alpha }^{2}Ld+\theta }{\alpha T}=\frac{1}{T}(Ld\alpha +\frac{\theta }{\alpha })\end{array}$$

We can choose $\alpha =\sqrt{\theta /Ld}$, then:

$$f(\overline{x})-f(x^{\ast })\le \frac{2\sqrt{L\theta d}}{T}$$

It seems its converge rate is $\frac{1}{T}$, but this converge rate is related to $d$. By restarting from $d\stackrel{}{\to }\frac{d}{2}\stackrel{}{\to }...\stackrel{}{\to }2ϵ\stackrel{}{\to }ϵ$.

$$O(\sum _{i=0}^{log(d/ϵ)}4\sqrt{\frac{L\theta }{2^iϵ}})=O(4\sqrt{\frac{L\theta }{ϵ}}\sum _{i=0}^{log(d/ϵ)}(\frac{1}{\sqrt{2}}{)}^i)=O(\sqrt{\frac{L\theta }{ϵ}})$$

Matrix completion

For a matrix $A$, some entries are known, while some are unknown. We want to predict the unknown entries. Namely, we want to find a find low rank $A^{\prime }$ so that $A^{\prime }$ matches A best on known entries.

Assumption

• low rank

• The known entries are uniformly distributed in their positions.

• Incoherence (like smooth assumption in unsupervised learning)

  • We hope all coordinates are comparable, rather than only a few of them being too large or too small.
  • Maybe the "gradient" between the entries can not be too sharp.
  • $\frac{1}{\sqrt{n}}(1,1,...,1)$ is better than $(1,0,...,0)$
  • Mathematically:

  • $$\begin{array}{rl}A=U\mathrm{\Sigma }{V}^T& ,U\in {\mathbb{R}}^{n\times r},V\in {\mathbb{R}}^{m\times r}\\ \Vert e_i^{T}U\Vert & \le \sqrt{\frac{ur}{n}},\mathrm{\forall }i\in [n]\\ \Vert e_j^{T}V\Vert & \le \sqrt{\frac{ur}{m}},\mathrm{\forall }j\in [m]\\ 1\le u& \le min\{n,m\}/r\end{array}$$

  • Here $U\in {\mathbb{R}}^{n\times r}$, $V\in {\mathbb{R}}^{m\times r}$, (Attention here, we do not need to complete the $U$ and $V$ to be a square matrix )

algorithm

Convex relaxation:

• How shall we calculate rank(A) naively? $A=U\mathrm{\Sigma }{V}^T$ $rank(A)=\Vert \mathrm{\Sigma }{\Vert }_0$.
• But in this way, minimizing rank(A) is hard. Because zero norm is not a good function (not convex).
• relax to L1-norm: Minimizing $\Vert A{\Vert }_1$. or relax to nuclear norm $\Vert \mathrm{\Sigma }{\Vert }_1$.

This is equivalent to a convex SDP problem.

However! SDP can not solve a large matrix. Say larger than 100*100.

交替最小二乘法 alternating least square minimization

Use $\mathrm{\Omega }$ to represent the known entries mask, $P_{\mathrm{\Omega }}(A)$ means the known entries with unknown positions filled with 0.
Assume the solution is $A=U{V}^T$

$$U,V=\underset{U,V\in R^{n\ast r}}{\arg min}\Vert P_{\mathrm{\Omega }}(U{V}^T)-P_{\mathrm{\Omega }}(A){\Vert }^2$$

The loss is nonconvex, one can easily find multiple local minimal, like $U^{\prime }=\frac{U}{2},V^{\prime }=2V$.

$$\begin{array}{rl}{V}_{t+1}& =\underset{V}{\arg min}\Vert P_{\mathrm{\Omega }}(U_t{V}^T)-P_{\mathrm{\Omega }}(A){\Vert }^2\\ U_{t+1}& =\underset{U}{\arg min}\Vert P_{\mathrm{\Omega }}(U{V}_{t+1}^T)-P_{\mathrm{\Omega }}(A){\Vert }^2\end{array}$$

Every subproblem is convex and can be solved efficiently.

Non-convex Optimization

Is convex landscape the only good case for GD? No.

Another important case: no-spurious-local-min.

• no-spurious-local-min: If all the local min are equally good, it suffices to find one of them! (Although nonconvex)

Escape saddle point

For stationary point: $\mathrm{\nabla }L(w)=0$:

• If ${\mathrm{\nabla }}^{2}L(w)\ge 0$, it's a local minimum or a flat saddle point. The latter one means we have no good direction to get out of this saddle point. That is all first-order algorithms will fail.
• If ${\mathrm{\nabla }}^{2}L(w)$ has both $+/-$ eigenvalues, it's a strict saddle point, which means at least one direction to escape.
• We call a function "strict saddle" if it does not contain any flat saddle points.

An informal Theorem

SGD will escape all saddle points and local maximum, converge to a
local minimum after a polynomial number of steps based on the following assumption.

• $L(w)$ is a smooth, bounded and strict saddle.
• ${\mathrm{\nabla }}^{2}L(w)$ is smooth
• SGD noise has non-negligible variance in every direction with constant probability

意义: 基于这个定理, 如果所有最小值一样好，且函数满足上面的条件，那么SGD就能收敛到全局最小值。

proof

For the current point ${𝑤}_0$, we discuss what happens with SGD.

Case1: $\Vert \mathrm{\nabla }f(w_0)\Vert$ is big

Since the gradient is big, Noise will not affect the decreasing trend of function values.

Case2: $\Vert \mathrm{\nabla }f(w_0)\Vert$ is small

subcase1: If we close to a local minimum

A local minimum is a trap. Although we have noise in gradient, we still have very little probability of getting out. 这保证我们在收敛到最小值之后不会离开

subcase2: If we are close to a saddle point

There must be a negative direction according to strict saddle property. Then a random perturbation will give a positive
projection in the negative direction. We will follow this direction and escape the saddle.

Escape local minimum

If the local minimum trap is too wide, then noise makes no difference.

But if the local minimum trap is sharp and narrow, the noise makes sense.

Informal Proof

Update rule:

$$\begin{array}{rl}{w}_{t+1}& =w_t-\eta (\mathrm{\nabla }L(w_t)-{\omega }_t)\\ y_t& =w_t-\eta \mathrm{\nabla }L(w_t)(y_t\mathrm{是}GD\mathrm{的}\mathrm{点})\end{array}$$

So we can get

$$\begin{array}{rl}{y}_{t+1}& =w_{t+1}-\eta \mathrm{\nabla }L(w_{t+1})=y_t-\eta {\omega }_t-\eta \mathrm{\nabla }L(y_t-\eta {\omega }_t)\\ E[y_{t+1}]& =E[y_t]-\eta \mathrm{\nabla }E[L(y_t-\eta {\omega }_t)]\end{array}$$

In expectation, it is like doing GD using an average gradient of the neighborhood, like doing a convolution on gradient.