Exponential family of distributions

目录

• 定义
• 性质
  • $A(\theta)$
  • 极大似然估计
  • 最大熵
• 例子
  • Bernoulli
  • 指数分布
  • 正态分布

Choi H. I. Lecture 4: Exponential family of distributions and generalized linear model (GLM).

定义

定义: 一个分布具有如下形式的密度函数:

$$f_{\theta}(x) = \frac{1}{Z(\theta)} h(x) e^{\langle T(x), \theta \rangle},$$

则该分布属于指数族分布.
其中$x \in \mathbb{R}^m$, $T(x) = (T_1(x), T_2(x), \cdots, T_k(x)) \in \mathbb{R}^k$, $\theta = (\theta_1, \theta_2,\cdots, \theta_k)$为未知参数, $Z(\theta) = \int h(x)e^{\langle T(x), \theta \rangle} \mathrm{d}x$为配平常数.

若令$C(x) = \log h (x)$, $A(\theta) = \log Z(\theta)$, 则

$$f_{\theta}(x) = \exp (\langle T(x), \theta \rangle - A(\theta) + C(x)).$$

指数族分布还有一种更一般的形式:

$$f_{\theta}(x) = \exp (\frac{\langle T(x), \theta \rangle - A(\theta)}{\phi} + C(x, \phi)),$$

更甚者

$$f_{\theta}(x) = \exp (\frac{\langle T(x), \lambda(\theta) \rangle - A(\theta)}{\phi} + C(x, \phi)),$$

$\phi$控制分布的形状.

性质

$A(\theta)$

Proposition 1:

$$\nabla_{\theta}A(\theta) = \int f_{\theta}(x) T(x) \mathrm{d}x = \mathbb{E}[T(X)].$$

proof:

已知:

$$\int f_{\theta}(x) \mathrm{d}x = \int \exp (\frac{\langle T(x), \theta \rangle - A(\theta)}{\phi} + C(x, \phi)) \mathrm{d}x = 1.$$

两边关于$\theta$求梯度得:

$$\int f_{\theta}(x) \frac{T(x) - \nabla_{\theta} A(\theta)}{\phi} \mathrm{d}x = 0 \Rightarrow \nabla_{\theta} A(\theta) = \mathbb{E}[T(X)].$$

Proposition 2:

$$D^2_{\theta} A = (\frac{\partial^2 A}{\partial\theta_i \partial \theta_j}) = \frac{1}{\phi}\mathrm{Cov}(T(X), T(X)) = \frac{1}{\phi}Cov(T(X)).$$

proof:

$$\frac{\partial A}{\partial \theta_i} = \int \exp (\frac{\langle T(x), \theta \rangle - A(\theta)}{\phi} + C(x, \phi)) T_i(x) \mathrm{d}x.$$

$$\begin{array}{ll} \frac{\partial^2 A}{\partial \theta_i \partial \theta_j} &= \int f_{\theta}(x) \frac{T_j (x) - \frac{\partial A}{\partial \theta_j}}{\phi} T_i(x) \mathrm{d}x \\ &= \frac{1}{\phi}\int f_{\theta}(x) (T_j(x) - \frac{\partial A}{\partial \theta_j}) (T_i(x) - \frac{\partial A}{\partial \theta_i})\mathrm{d}x \\ &= \mathrm{Cov}(T_i(X), T_j(X)). \end{array}$$

Corollary 1: $A({\theta})$关于$\theta$是凸函数.

既然其黑塞矩阵半正定.

极大似然估计

设有$\{x^i\}_{i=1}^n$个样本, 则对数似然函数为

$$l(\theta) = \frac{1}{\theta}[\langle \theta, \sum_{i=1}^n T(x^i)-nA(\theta)] + \sum_{i=1}^n C(x^i, \phi),$$

因为$A(\theta)$是凸函数, 所以上述存在最小值点, 且

$$\nabla_{\theta} l(\theta) = \frac{1}{\phi}[\sum_{i=1}^n T(x^i) - n \nabla_{\theta}A(\theta)],$$

故该最小值点在

$$\nabla_{\theta}A(\theta) = \frac{1}{n} \sum_{i=1}^n T(x^i),$$

处达到.

最大熵

最大熵原理-科学空间

指数族分布实际上满足最大熵分布, 这是在没有任何偏爱的尺度下的分布.

即

$$\max_{f} \quad H(f) = -\int f(x)\log f(x) \mathrm{d} x.$$

等价于最小化

$$\min_f \int f(x)\log f(x) \mathrm{d}x.$$

往往, 我们会有一些已知的统计信息, 通常以期望的形式表示:

$$\int f(x) h_i(x) \mathrm{d}x = c_i, \quad i=1,2\cdots, s.$$

则我们的目标实际上是:

$$\min_f \quad \int f(x)\log f(x) \mathrm{d}x \\ \mathrm{s.t.} \quad \int f(x) h_i(x) \mathrm{d}x = c_i, \quad i=0,2\cdots, s.$$

其中$h_0 = 1, c_0 =1$, 即密度函数需满足$\int f(x) \mathrm{d} x= 1$.

利用拉格朗日乘数得:

$$J(f,\lambda) = \int f(x)\log f(x) \mathrm{d}x + \lambda_0 (1 - \int f(x) \mathrm{d}x) + \sum_{i=1}^s \lambda_i [c_i - \int f(x) h_i(x) \mathrm{d}x] .$$

最优条件, $J$关于$f$的变分为0, 即

$$1 + \log f(x) - \lambda_0 - \sum_{i=1}^s \lambda_i h_i(x) = 0.$$

即

$$f(x) = \frac{1}{Z} \exp(\sum_{i=1}^s \lambda_i h_i(x)).$$

属于指数分布族.

例子

Bernoulli

$$P(x) = p^x (1-p)^{1-x} = \exp[x\log\frac{p}{1-p} + \log (1 - p)].$$

$$\theta = \log \frac{p}{1-p}, \\ T(x) = x, \\ A(\theta) = \log (1 + e^{\theta}),\\ h(x) = 0.$$

指数分布

$$p(x) = \lambda \cdot e^{-\lambda x}=\exp[-\lambda x +\log \lambda ], \quad x \ge 0.$$

$$\theta = \lambda,\\ T(x) =-x, \\ A(\theta) = \log \frac{1}{\lambda}, \\ h(x) = \mathbb{I}(x\ge0).$$

正态分布

$$p(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp [-\frac{(x-\mu)^2}{2\sigma^2}].$$

$\sigma$视作已知参数:

$$p(x) = \exp [\frac{-\frac{1}{2}x^2 + x\mu - \frac{1}{2}\mu^2}{\sigma^2} - \frac{1}{2}\log (2\pi \sigma^2)].$$

$$\theta = (\mu, 1), \\ T(x) = (x, -\frac{1}{2}x^2), \\ \phi = \sigma^2, \\ A(\theta) = \frac{1}{2}\mu^2, \\ C(x, \phi) = \frac{1}{2} \log (2\pi \sigma^2).$$

$\sigma$视作未知参数:

$$p(x) = \exp [-\frac{1}{2\sigma^2}y^2 + \frac{\mu}{\sigma^2}x - \frac{1}{2\sigma^2}\mu^2 - \log \sigma - \frac{1}{2}\log 2\pi].$$

$$T(x) = (x, \frac{1}{2}x^2), \\ \theta = (\frac{\mu}{\sigma^2}, -\frac{1}{\sigma^2}), \\ A(\theta) = \frac{\mu^2}{2\sigma^2} + \log\sigma, \\ C(x) = -\frac{1}{2}\log(2\pi).$$