RBM
1. 玻尔兹曼分布：
$$p(E) \thicksim e^{-E/kT} $$
2. RBM
- 两层：隐层和可视层， $\mathbf v$, $\mathbf h$
$$v_i \in \{0, 1\}, \ \ \ h_j \in \{0, 1\}$$
- 能量假设：
$$ E (\mathbf v, \mathbf h; \theta) = - \mathbf b \cdot \mathbf v - \mathbf c \cdot \mathbf h - \mathbf v^T W \mathbf h \\
\theta = \{\mathbf b,\ \mathbf c,\ W\}$$
- 概率分布：
$$p(v, h; \theta) = \frac 1 Z e ^{-E(v,h; \ \theta)} \\
Z(\theta) = \sum_{v,h} e^{-E(v,h; \theta)}$$
- 条件概率：
$$p(v | h;\ \theta) = \frac{e^{-E(v,h)}}{\sum_ve^{-E(v,h)}} \\
p(h | v;\ \theta) = \frac{e^{-E(v,h)}}{\sum_he^{-E(v,h)}} \\
p(v_i = 1 \ | \ h; \theta) = \sigma(b_i + \sum_j W_{ij} h_j) \\
p(h_j = 1 \ | \ v; \theta) = \sigma(c_j + \sum_i W_{ij} v_i)$$
- 全概率：
$$ p(v) = \sum_h p(v,h) = \frac{\sum_h e^{-E(v,h)}}{\sum_{v,h} e^{-E(v,h)}} $$
3. 优化
- 极大化似然函数：
$$\mathcal L(\theta\ |\ v) = \ln p(v;\ \theta) = \ln\sum_he^{-E(v,h)} - \ln\sum_{v,h} e^{-E(v,h)} $$
- 梯度：
$$ \frac{\partial L}{\partial \theta}
= \mathrm E_{p(h|v)}[-\frac{\partial E(v,h)}{\partial\theta}] - \mathrm E_{p(v,h)}[-\frac{\partial E(v,h)}{\partial \theta}] \\
\frac{\partial E(v,h)}{\partial W_{ij}} = - v_i h_j, \\
\frac{\partial E(v,h)}{\partial b_i} = - v_i, \\
\frac{\partial E(v,h)}{\partial c_j} = - h_j$$
4.其他能量模型
- Gaussian-Bernoulli RBM：
- 能量定义：
$$E(v,h; \theta) = \sum_i \frac{(v_i - b_i)^2}{2\sigma_i^2} - \sum_j c_j h_j - \sum_{ij} W_{ij}\frac{v_i}{\sigma_i}h_j \\ \theta = \{b,\ \sigma,\ c,\ W \} \qquad\qquad \qquad\qquad $$
- 条件概率：
$$p(v_i = x \ | \ h;\ \theta) = \mathcal N(b_i + \sigma_i \sum_j W_{ij}h_j,\ \sigma_i) \\
p(h_j = 1 \ | \ v;\ \theta) = \sigma(c_j + \sum_i W_{ij}\frac {v_i}{\sigma_i}) \ \ \ \ \ $$
- extended energy
- 能量定义
$$E(v,\ y,\ h) = -\sum b_i v_i - \sum c_j h_j - \sum W_{ij} v_i h_j - \sum d_k y_k - \sum U_{jk} h_j y_k \\
\theta = \{b,\ c,\ W,\ d,\ U\} \qquad\qquad\qquad$$
- 条件概率
$$p(v_i = 1 | h) = \sigma(b_i + \sum_j W_{ij}h_j) \qquad\qquad\\
p(h_j = 1| x,\ y) = \sigma(c_j + \sum_iW_{ij}x_i + \sum_kU_{jk}y_k) \\
p(y_k = 1 | h) = \frac{\exp(d_k + \sum_j U_{jk}h_j)}{\sum_k \exp(d_k + \sum_kU_jk h_j)} $$
5. 附录
1. 玻尔兹曼分布的最大熵推导
封闭系统能量守恒，总能量 $\mathcal E$。共有$N$个状态，每个状态$i$的能量 $E_i$，对应概率 $p_i$
则有约束条件：
$$\sum_ip_i = 1 \\ \sum_i p_i E_i = \mathcal E/N \equiv \bar E $$
最大化信息熵：
$$H[p] = -\sum_ip_i\ln p_i$$
等效于最大化下面的拉格朗日量：
$$\mathcal L[p] = H[p] + \alpha (1 - \sum_ip_i)+ \beta(\bar E - \sum_ip_iE_i)$$
即得能量的概率分布：
$$p(E_i) \propto e^{-\beta E_i}$$
2. RBM 条件概率推导
$$p(v_i=1 |h) = \frac{\sum_{v_k\ne i}p(v_i=1,v_k,h)}{\sum_vp(v,h)} \\
=\frac{\sum_{v_k\ne i}\exp[(b_iv_i+\sum_j W_{ij}v_ib_j)_{v_i=1}+\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]}
{\sum_{v_i,v_{k\ne i}} \exp[(b_iv_i+\sum_j W_{ij}v_ib_j)+\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]]} \\
= \frac{\exp[(b_iv_i+\sum_j W_{ij}v_ib_j)_{v_i=1}]\cdot\sum_{v_k\ne i}\exp[\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]}
{\sum_{v_i} \exp[(b_iv_i+\sum_j W_{ij}v_ib_j)] \cdot \sum_{v_{k\ne i}}\exp[\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]} \\
= \frac{\exp[(b_iv_i+\sum_j W_{ij}v_ib_j)_{v_i=1}]}
{\sum_{v_i} \exp[(b_iv_i+\sum_j W_{ij}v_ib_j)]} \\
= \frac{1}{1 + \exp[-b_i - \sum_jW_{ij}b_j]}. \qquad (v_i\in \{0,1\}) $$
