EM算法
EM Algorithm
Jensen's inequality
convex function: \(f''(x) \ge 0\) or \(H \ge 0\) (Hessian matrix when x is a vector)
\[E[f(x)] \ge f(EX)
\]
EM Algorithm
EM can be proved that it make the likelihood function increase monotonically.
maximize the lower-bound on the likelihood \(\ell\), \(\log\) is a concave function. the process is
\[\theta_{t-1} \rightarrow^{assign} Q_{z,t-1} \rightarrow^{\arg\max} \theta_{t} \rightarrow Q_{z,t}
\]
define:
\[J(Q,\theta)=\sum_{i} \sum_{z^{(i)}}Q_i(z^{(i)})\log \frac{p(x^{(i)},z^{(i)};\theta)}{Q_i(z^{(i)})}
\]
the EM algorithm can also be defined as coordinate ascent on \(J\).
