极大似然估计与方差

相关概念：极大似然估计，score function，Fisher information

Let f(X; θ) be the probability density function (or probability mass function) for X conditional on the value of θ. This is also the likelihood function for θ. It describes the probability that we observe a given sample X, given a known value of θ.

1、If f is sharply peaked with respect to changes in θ, it is easy to indicate the “correct” value of θ from the data, or equivalently, that the data X provides a lot of information about the parameter θ。

2、If the likelihood f is flat and spread-out, then it would take many, many samples like X to estimate the actual “true” value of θ that would be obtained using the entire population being sampled.

two if ： This suggests studying some kind of variance with respect to θ.

score function： 对数似然函数的一阶导

以上是证明一阶导的期望为0

fisher information：score function的二阶矩

Note that. A random variable carrying high Fisher information implies that the absolute value of the score is often high. The Fisher information is not a function of a particular observation, as the random variable X has been averaged out.

如果对数似然函数的二阶可导，则Fisher信息量可写成：

因为：

{

 对数似然的一阶导

  一阶导的方差就是信息量}

同时由于：

Thus, the Fisher information may be seen as the curvature （曲率）of the support curve (the graph of the log-likelihood). Near the maximum likelihood estimate, low Fisher information therefore indicates that the maximum appears "blunt", that is, the maximum is shallow and there are many nearby values with a similar log-likelihood. Conversely, high Fisher information indicates that the maximum is sharp.

 信息量可加：