概率密度指数函数族与广义线性模型
Exponential Family
The pdf is as follows,
\[p(y;\eta)=b(y) e^{\eta^TT(y)-a(\eta)}
\]
Mathematical properties:
- MLE w.r.t. \(\eta\) is concave, but negative log likelihood is convex!
- \(E(y;\eta)=\frac{\partial}{\partial \eta}a(\eta)\)
- \(Var(y;\eta)=\frac{\partial^2}{\partial^2 \eta}a(\eta)\)
Generalized Linear Model
Assumptions:
- \(y|x;\theta \sim \text{Exponential Family}(\eta)\)
- \(\eta = \theta^Tx\) where \(\theta \in \R^{n+1}, x \in \R^{n+1}\)
- Test time, output is \(E(y|x;\theta)\)
No matter what distribution you choose, the learning update rule can be uniformly
\[\theta_j := \theta_j +\alpha(y^{(i)}-h_\theta(x^{(i)}))x^{(i)}_j
\]
